Home | Sitemap | Contact | NRF Home

Monitoring & Evaluation

Enhancing the quality of research in South Africa
Programme Evaluation
 

 Protocol

Identification of the need for and purpose of the evaluation
Identification of the Assignment Principal (Reference Group) and definition of tasks
Appointment of a Project Manager/Management Agency
Compilation of terms of reference for the evaluation
Identification and appointment of the Evaluation Panel (review panel, panel of reviewers, technical team) to undertake the evaluation 
Compilation/Collection of source documentation on which the evaluation will be based (e.g. self-evaluation reports, questionnaires etc)
Identification of stakeholders with whom review panel should interact 
Development and approval of the work plan/programme for the evaluation

Invitation to stakeholders to participate in the evaluation

Finalisation of the evaluation report
Approval of the evaluation report

Dissemination of evaluation report


The following was developed by Monitoring & Evaluation for the evaluation of NRF programmes.

Identification of the need for and purpose of the evaluation
Both the need for and the purpose of the evaluation must be clearly articulated as these will determine the nature of the evaluation.  The need for the evaluation can be determined in a top down or bottom up process but in both instances it is imperative that the necessary buy-in to undertake such an evaluation is obtained from all relevant parties i.e. programme management, NRF Executive and other interested stakeholders.  The purpose of the evaluation will determine how the evaluation should be structured.

 Identification of the Assignment Principal (Reference Group) and definition of tasks
The Assignment Principal (Reference Group) is the principal person/group commissioning the evaluation (assignment) and is responsible for ensuring that the evaluation will be concluded in line with the terms of reference.  For an evaluation to be regarded as an independent evaluation it is important that the Assignment Principal/Reference Group is independent from the programme/activity that is being evaluated.  However, the Assignment Principal/Reference Group should interact closely with those in programme management and other interested stakeholders.

The deliberations of the Reference Group should be transparent at all times as buy-in into the evaluation is one of the most important ingredients to its successful conclusion.

The size of the Reference Group should be relatively small, ideally three to four members.  The members should attend all meetings.  However, the Reference Group, under normal conditions should not have to meet more than one or two times.  Meetings of the Reference Group should be attended by the programme management and by Monitoring & Evaluation.

The Assignment Principal must oversee the entire evaluation to its successful conclusion and has the following important tasks:
 Approval of the terms of reference
 Appointment of  a chairperson of the Reference Group (where applicable)
 Appointment of the Evaluation Panel (i.e. the Technical Team/Panel of Reviewers/Review Panel) and its convenor
 Commissioning the evaluation to Monitoring & Evaluation
 Approval of the programme for the evaluation (i.e. the work plan of the Evaluation Panel/Technical Team)
 Approval of the time frame for the evaluation
 Recommendation for approval (or approval if legally able to do so) of all contracts and budgets
 Receiving progress/final reports from the Evaluation Panel
 Soliciting a response from the programme management to the report
 Submission of the final report of the Evaluation Panel to the NRF Executive (where the Assignment Principal is external to the NRF)
 Approval of the dissemination process of the final report (outlined in the terms of reference)

Appointment of a Project Manager/Management Agency
Though Monitoring & Evaluation of the NRF fulfils this role within the NRF, outside agencies can also be contracted, in consultation with Monitoring & Evaluation, where the need arises. 

As the Reference Group members and members of the Evaluation Panel are generally distinguished persons with relatively limited time at their disposal it is important that they are supported by an effective secretariat to manage the entire evaluation process.  The Project Manager/Management Agency must be well versed in both the intellectual as well as the logistical requirements for evaluations.   

Compilation of terms of reference for the evaluation
The Assignment Principal/Reference Group is responsible for the development and approval of the terms of reference for the evaluation.  The compilation of precise and succinct terms of reference is crucial for the success of an evaluation.  The first draft of such terms of reference should be drawn up by Monitoring & Evaluation with the guidance of the Assignment Principal/Reference Group in consultation with the programme management.  

The terms of reference should address the following:
 The title of the evaluation (assignment).
 The Assignment Principal/Reference Group and its role. 
 The Project Manager/Management Agency, its role and how it will be appointed (e.g. tender process).  If the Project Manager/Management Agency has already been appointed it should be mentioned by name.
 The need and purpose of the evaluation.
 The scope of the evaluation.
 The aspects that the evaluation will cover, which could, inter alia, include 
  - Strategic Objectives
  - Implementation framework
  - Performance of the programme under evaluation with respect to the targets set the impact of the programme
  - Stakeholder interaction

 Details on the actual evaluation process in terms of
  - the appointment of the Evaluation Panel
  - the number of persons on the Evaluation Panel
  - the documentation required for the review
  - the programme (work plan) for the evaluation
  - the line of questioning that the Evaluation Panel will follow during interviews
  - deliverables such as verbal presentations, reports etc.
  - dissemination mechanisms of the report i.e. making the report available to the relevant stakeholders in hard copy or publishing it electronically on the NRF Website 

 Time frames i.e. when the evaluation should take place and how long it will be, when should the programme management’s response to evaluation report be provided, and how soon thereafter should the report be disseminated to all stakeholders
 Budgetary matters in terms of responsibility for costs, payments that will be made and extent thereof, etc.
 Contractual agreements with e.g. Evaluation Centre (where applicable), Evaluation Panel etc.

The terms of reference should be binding and should be signed by the Assignment Principal, programme management, interested stakeholders/parties and Monitoring & Evaluation.  In so doing the parties would be endorsing the contents of the terms of reference and further undertaking to deliver on their responsibilities within the agreed upon time frames.  

 The Evaluation Panel appointed could, if they see fit, also propose amendments to terms of reference, which upon acceptance by the Assignment Principal/Reference Group and the programme management, will be incorporated. 

Identification and appointment of the Evaluation Panel (review panel, panel of reviewers, technical team) to undertake the evaluation
The Evaluation Panel should comprise of a relatively small group.  Experience has shown that panels of four to five members work well.  Careful thought should be given to the “skill’s mix” that is required on such a panel depending on the nature of the programme and the evaluation.  It is also important that civil society, where possible, be represented in the Evaluation Panel.  The persons appointed should have the appropriate experience and skills to conduct the evaluation.  

The Assignment Principal/Reference Group should invite the programme management and any other relevant stakeholders to put forward names of possible panel members.  A brief curriculum and motivation for the nomination should also be provided.  The appointment of the right members to the Evaluation Panel is of cardinal importance to the success of the review. 

The need for contractual agreements between the contracting party and the Evaluation Panel member needs to be discussed.   

Compilation/Collection of source documentation on which the evaluation will be based (e.g. self-evaluation reports, questionnaires etc)
The terms of reference should already indicate the baseline documents that will be required for the review.  These documents serve as background information and are made available to the Evaluation Panel at least four weeks before the commencement of the evaluation so that members of the Evaluation Panel can familiarise themselves with the content and extent of the factual information they will be dealing with.  These documents include:
 Self-evaluation reports
 Business plans
 Annual reports including summaries of financial statements
 Any other reports by other stakeholders 

One of the key baseline documents required for a successful evaluation is the self-evaluation report compiled by the management of the programme being evaluated.  The format of the self-evaluation reports should be in line with the objectives of the programme and should address the various aspects of the terms of reference of the evaluation.  The format for the reports could be laid down in the terms of reference if this is seen as desirable.   

It is important that the documentation provided is succinct and that the Evaluation Panel is not inundated with unnecessary information.  Additional documentation could be made available for perusal by the Evaluation Panel.

Identification of stakeholders with whom review panel should interact 
An important facet of the evaluation is the careful identification of the stakeholders/stakeholder groups with whom the Evaluation Panel should interact.  Such stakeholders/stakeholder groups can be identified from the business plans or by the programme being evaluated.  Stakeholder lists should be compiled indicating a selection from various stakeholder groups.  

For a balanced review the stakeholder lists could indicate amongst the stakeholders those that are positively or negatively disposed to the programme being evaluated.  

The first stakeholder list produced by Monitoring & Evaluation in consultation with the programme management should be scrutinised by members of both the Reference Group and the Evaluation Panel who should feel free to propose additional stakeholders to the list.  Any interested party should essentially be allowed to make additions to the stakeholder lists and this should essentially be an open and transparent process.  

The actual selection of the stakeholders who will interact with the Evaluation Panel should, however, in the end be left to Monitoring & Evaluation and the Assignment Principal (Reference Group). 

Development and approval of the work plan/programme for the evaluation
The terms of reference should provide guidance on the programme/workplan according to which the Evaluation Panel will fulfill its task.  Hence some information should be provided on the length of the evaluation period, the nature of the interviews, the use and analysis of questionnaires, the need for on-site visits etc.  The draft programme should be tested with the Assignment Principal, programme management, the Evaluation Panel and other interested stakeholders. The Assignment Principal (Reference Group) should approve the final programme. 

Invitation to stakeholders to participate in the evaluation
Once the correct processes have been followed for the identification and selection of the stakeholders with whom the Evaluation Panel will interact and for the compilation of the programme the invitations to the participating stakeholders is a simple matter.  The letters to stakeholders should be signed by the Assignment Principal or Chairperson of the Reference Group. 

Finalisation of the evaluation report
Compilation of the evaluation report is the responsibility of the Evaluation Panel and acceptance of this report is the responsibility of the NRF Executive or Reference Group in cases of joint programmes.

Steps to be followed in the compilation of the report are 
1.  The Evaluation Panel will compile the draft evaluation report which must be completed within three days of the final stakeholder interview. Such a report should contain, inter alia, the following
  - An executive summary
  - Background to the evaluation
  - Evaluation questions that were addressed
  - Key findings
  - Recommendations
  - Appendices containing the e.g. terms of reference, the persons interviewed.

2.  The Evaluation Panel will present the key findings of the evaluation to the NRF or Reference Group immediately after the draft evaluation report is compiled. The presentation should be attended by
  - Management of the programme being evaluated
  - NRF Executives
  - Evaluation Centre
  - Interested stakeholders 

The parties in attendance may engage the Evaluation Panel on their findings.   

3.  The programme management and interested stakeholder should be invited to submit their comments to the Evaluation Panel on any factually inaccurate information that may be contained in the draft evaluation report within ten days after the presentation of the draft report. 

4.  The Evaluation Panel will finalise the evaluation report two weeks after receiving the comments of the programme management and interested stakeholders and submit it to Monitoring & Evaluation. 

Approval of the evaluation report
The programme management will be required to provide a written response to the evaluation report, which should be endorsed by the relevant line Executive Director, before being submitted to Monitoring & Evaluation.  This should be done within three weeks of receipt of the evaluation report from Monitoring & Evaluation.  The programme management’s response should address the issues raised in the evaluation report and should indicate proposed actions in view of the recommendations contained in the evaluation report.  The report and the response will be tabled by the line Director of Monitoring & Evaluation before an Executive Committee meeting for discussion and approval/acceptance. 

Dissemination of evaluation report
Upon acceptance of the evaluation report by the NRF Executive or Reference Group, it should be disseminated according to the criteria outlined in the terms of reference.  The NRF is committed to good corporate governance and transparency, hence all evaluation reports, including an NRF’s response, to be compiled by the relevant NRF Executive Director, should be made available to all stakeholders and publishable on the NRF website.  The NRF’s response and the evaluation report should be preceded by a foreword (not more than one page) prepared by the relevant Executive Director, contextualising the evaluation and broadly outlining how the NRF intends to respond to the issues emanating from the evaluation.  The foreword needs to be approved and signed by the Managing Director: Research Support Agency.  For joint programmes, the board of the relevant programme should approve the foreword.

 

Last updated on 6 May 2009