The following was developed by Monitoring & Evaluation for
the evaluation of NRF programmes.
of the need for and purpose of the evaluation
Both the need for and the purpose of the evaluation must
be clearly articulated as these will determine the nature of the evaluation.
The need for the evaluation can be determined in a top down or bottom up
process but in both instances it is imperative that the necessary buy-in to
undertake such an evaluation is obtained from all relevant parties
i.e. programme management, NRF Executive and other interested stakeholders. The
purpose of the evaluation will determine how the evaluation should be
the Assignment Principal
(Reference Group) and definition of tasks
The Assignment Principal (Reference Group) is the
principal person/group commissioning the evaluation (assignment) and is
responsible for ensuring that the evaluation will be concluded in
line with the terms of reference. For an evaluation to be
regarded as an independent evaluation it is important that the Assignment
Principal/Reference Group is independent from the programme/activity that is
being evaluated. However, the Assignment Principal/Reference Group should
interact closely with those in programme management and other interested
The deliberations of the Reference Group should be
transparent at all times as buy-in into the evaluation is one of
the most important ingredients to its successful conclusion.
The size of the Reference Group should be relatively
small, ideally three to four members. The members should
attend all meetings. However, the Reference Group, under normal
conditions should not have to meet more than one or two times. Meetings of the
Reference Group should be attended by the programme management and by Monitoring
The Assignment Principal must oversee the entire
evaluation to its successful conclusion and has the following important tasks:
Approval of the terms of reference
Appointment of a chairperson of the Reference Group
Appointment of the Evaluation Panel (i.e. the Technical
Team/Panel of Reviewers/Review Panel) and its convenor
the evaluation to Monitoring & Evaluation
Approval of the programme for the evaluation (i.e. the
work plan of the Evaluation Panel/Technical Team)
of the time frame for the evaluation
for approval (or approval if legally able to do so) of all contracts and budgets
progress/final reports from the Evaluation Panel
Soliciting a response from the programme management to the report
of the final report of the Evaluation Panel to the NRF Executive (where the
Assignment Principal is external to the NRF)
of the dissemination process of the final report (outlined in the terms of
Appointment of a Project
Though Monitoring &
Evaluation of the NRF fulfils this role
within the NRF, outside agencies can also be contracted, in consultation with
Monitoring & Evaluation, where the need arises.
As the Reference Group members and members of the
Evaluation Panel are generally distinguished persons with relatively limited
time at their disposal it is important that they are supported by an effective
secretariat to manage the entire evaluation process. The Project
Manager/Management Agency must be well versed in both the intellectual as well
as the logistical requirements for evaluations.
Compilation of terms of reference for
The Assignment Principal/Reference Group is responsible
for the development and approval of the terms of reference for the evaluation.
The compilation of precise and succinct terms of reference is crucial for the
success of an evaluation. The first draft of such terms of reference should be
drawn up by Monitoring & Evaluation with the guidance of the Assignment
Principal/Reference Group in consultation with the programme management.
The terms of reference should address the following:
The title of the evaluation (assignment).
The Assignment Principal/Reference Group and its role.
The Project Manager/Management Agency, its role and how
it will be appointed (e.g. tender process). If the Project Manager/Management
Agency has already been appointed it should be mentioned by name.
The need and purpose of the evaluation.
The scope of the evaluation.
The aspects that the evaluation will cover, which
could, inter alia, include
- Strategic Objectives
- Implementation framework
- Performance of the programme under evaluation with respect to the
targets set the impact of the programme
- Stakeholder interaction
Details on the actual evaluation process in terms of
- the appointment of the Evaluation Panel
- the number of persons on the Evaluation Panel
- the documentation required for the review
- the programme (work plan) for the evaluation
- the line of questioning that the Evaluation Panel will follow during
- deliverables such as verbal presentations, reports etc.
- dissemination mechanisms of the report i.e. making the report available
to the relevant stakeholders in hard copy or publishing it electronically on
the NRF Website
Time frames i.e. when the evaluation should take place
and how long it will be, when should the programme management’s response to
evaluation report be provided, and how soon thereafter should the report be
disseminated to all stakeholders
Budgetary matters in terms of responsibility for costs,
payments that will be made and extent thereof, etc.
Contractual agreements with e.g. Evaluation Centre
(where applicable), Evaluation Panel etc.
The terms of reference should be binding and should be
signed by the Assignment Principal, programme management, interested
stakeholders/parties and Monitoring & Evaluation. In so doing the parties would
be endorsing the contents of the terms of reference and further undertaking to
deliver on their responsibilities within the agreed upon time frames.
The Evaluation Panel appointed could, if they see fit,
also propose amendments to terms of reference, which upon acceptance by the
Assignment Principal/Reference Group and the programme management, will be
appointment of the Evaluation Panel (review panel, panel of reviewers, technical
team) to undertake the evaluation
The Evaluation Panel should comprise of a relatively
small group. Experience has shown that panels of four to five members work
well. Careful thought should be given to the “skill’s mix” that is required on
such a panel depending on the nature of the programme and the evaluation. It is
also important that civil society, where possible, be represented in the
Evaluation Panel. The persons appointed should have the appropriate experience
and skills to conduct the evaluation.
The Assignment Principal/Reference Group should invite
the programme management and any other relevant stakeholders to put forward
names of possible panel members. A brief curriculum and motivation for the
nomination should also be provided. The appointment of the right members to the
Evaluation Panel is of cardinal importance to the success of the review.
The need for contractual agreements between the
contracting party and the Evaluation Panel member needs to be discussed.
source documentation on which the evaluation will be based (e.g. self-evaluation
reports, questionnaires etc)
The terms of reference
should already indicate the baseline documents that will be required for the
review. These documents serve as background information and are made available
to the Evaluation Panel at least four weeks before the commencement of the
evaluation so that members of the Evaluation Panel can familiarise themselves
with the content and extent of the factual information they will be dealing
with. These documents include:
Annual reports including summaries of financial statements
Any other reports by other stakeholders
One of the key baseline documents required for a
successful evaluation is the self-evaluation report compiled by the management
of the programme being evaluated. The format of the self-evaluation reports
should be in line with the objectives of the programme and should address the
various aspects of the terms of reference of the evaluation. The format for the
reports could be laid down in the terms of reference if this is seen as
It is important that the documentation provided is
succinct and that the Evaluation Panel is not inundated with unnecessary
information. Additional documentation could be made available for perusal by
the Evaluation Panel.
stakeholders with whom review panel should interact
An important facet of the evaluation is the careful
identification of the stakeholders/stakeholder groups with whom the Evaluation
Panel should interact. Such stakeholders/stakeholder groups can be identified
from the business plans or by the programme being evaluated. Stakeholder lists
should be compiled indicating a selection from various stakeholder groups.
For a balanced review the stakeholder lists could
indicate amongst the stakeholders those that are positively or negatively
disposed to the programme being evaluated.
The first stakeholder list produced by Monitoring &
Evaluation in consultation with the programme management should be scrutinised by
members of both the Reference Group and the Evaluation Panel who should feel
free to propose additional stakeholders to the list. Any interested party
should essentially be allowed to make additions to the stakeholder lists and
this should essentially be an open and transparent process.
The actual selection of the stakeholders who will
interact with the Evaluation Panel should, however, in the end be left to
Monitoring & Evaluation and the Assignment Principal (Reference Group).
Development and approval of the
work plan/programme for the evaluation
The terms of reference should provide guidance on the
programme/workplan according to which the Evaluation Panel will fulfill its
task. Hence some information should be provided on the length of the evaluation
period, the nature of the interviews, the use and analysis of questionnaires,
the need for on-site visits etc. The draft programme should be tested with the
Assignment Principal, programme management, the Evaluation Panel and other
interested stakeholders. The Assignment Principal (Reference Group) should
approve the final programme.
Invitation to stakeholders to participate in the evaluation
Once the correct processes have been followed for the
identification and selection of the stakeholders with whom the Evaluation Panel
will interact and for the compilation of the programme the invitations to the
participating stakeholders is a simple matter. The letters to stakeholders
should be signed by the Assignment Principal or Chairperson of the Reference
Finalisation of the evaluation
Compilation of the evaluation report is the
responsibility of the Evaluation Panel and acceptance of this report is the
responsibility of the NRF Executive or Reference Group in cases of joint
Steps to be followed in the compilation of the report
1. The Evaluation Panel will compile the draft
evaluation report which must be completed within three days of the final
stakeholder interview. Such a report should contain, inter alia, the following
- An executive summary
- Background to the evaluation
- Evaluation questions that were addressed
- Key findings
- Appendices containing the e.g. terms of reference, the persons
2. The Evaluation Panel will present the key findings of
the evaluation to the NRF or Reference Group immediately after the draft
evaluation report is compiled. The presentation should be attended by
- Management of the programme being evaluated
- NRF Executives
- Evaluation Centre
- Interested stakeholders
parties in attendance may engage the Evaluation Panel on their findings.
3. The programme management and interested stakeholder
should be invited to submit their comments to the Evaluation Panel on any
factually inaccurate information that may be contained in the draft evaluation
report within ten days after the presentation of the draft report.
4. The Evaluation Panel will finalise the evaluation
report two weeks after receiving the comments of the programme management and
interested stakeholders and submit it to Monitoring & Evaluation.
Approval of the evaluation
The programme management will be required to provide a
written response to the evaluation report, which should be endorsed by the
relevant line Executive Director, before being submitted to Monitoring &
Evaluation. This should be done within three weeks of receipt of the evaluation
report from Monitoring & Evaluation. The programme management’s response should
address the issues raised in the evaluation report and should indicate proposed
actions in view of the recommendations contained in the evaluation report. The
report and the response will be tabled by the line Director of Monitoring &
Evaluation before an Executive Committee meeting for discussion and
Upon acceptance of the evaluation report by the NRF
Executive or Reference Group, it should be disseminated according to the
criteria outlined in the terms of reference. The NRF is committed to good
corporate governance and transparency, hence all evaluation reports, including
an NRF’s response, to be compiled by the relevant NRF Executive Director, should
be made available to all stakeholders and publishable on the NRF website. The
NRF’s response and the evaluation report should be preceded by a foreword (not
more than one page) prepared by the relevant Executive Director, contextualising
the evaluation and broadly outlining how the NRF intends to respond to the
issues emanating from the evaluation. The foreword needs to be approved and
signed by the Managing Director: Research Support Agency. For joint programmes,
the board of the relevant programme should approve the foreword.
Last updated on 6 May 2009