Monitoring & Evaluation
Enhancing the quality of research in South Africa
Evaluation and rating
 
Process to evaluate and rate individual researchers
Peer reports
Allocation of a rating
Process of appeal
 

Applications for evaluation and rating by the NRF will be submitted electronically at http:///nrfinterim.nrf.ac.za/nrfapp/. Details about the application procedure are found in the section "Apply to be rated" or can be obtained from the NRF Monitoring & Evaluation.

Peer reports

The application must be screened and approved by the applicant's institutional research administration, which in turn submits it electronically to Monitoring & Evaluation, where the application is acknowledged and screened for acceptance. The full documentation is then sent to members of subject-specific Specialist Committees who read the documents and, on the basis of their knowledge and the recommendations from the applicant, identify the names of at least six, but no more than ten, suitable peer reviewers. This is regarded as one of the crucial steps in the evaluation process and the committees are provided with detailed guidelines on the selection of peer reviewers. In some cases a Specialist Committee may identify a problem with the application. For example, in their judgement, it may be premature or too weak to submit to reviewers. In these cases the matter is referred to the relevant institution for a final decision on whether to proceed with/withdraw the application

At least six reviewers (peers) evaluate the research outputs of each applicant. Normally half of them are selected from a list of potential reviewers supplied by the applicant, while the rest are selected independently by the relevant Specialist Committee. Monitoring & Evaluation sends comprehensive documentation to peer reviewers and asks them to scrutinise this and to provide an appraisal/evaluation on the following:

  • The quality of the research-based outputs of the last eight years as well as the impact of the applicant's work in his/her field and how it has impacted on adjacent fields.
  • An estimation of the applicant's standing as a researcher in terms of both a South African and international perspective.

Because the identification of the best reviewers is at the heart of the entire process it is inevitable that a large number of the appointed reviewers are from prestigious institutions abroad. It is important to note that reviewers are neither informed about the previous evaluation or rating of applicants nor are they informed about the rating categories.

The most appropriate Specialist Committee to handle an application is selected, normally by the applicant, to drive the process. This Specialist Committee selects the reviewers but it may consult other Specialist Committees to select reviewers where this is necessary. The full suite of reviewers' reports will be handled by one Specialist Committees.

Allocation of a rating

The staff members of Monitoring & Evaluation collect the reports from reviewers and screen them in order to attend to any matters arising (such as reviewers requesting copies of research publications that they do not have ready access to). All reports are treated confidentially. They also take note of reviewers who are unable to respond to the NRF's request and liaise with conveners of Specialist Committees to ascertain whether more reviewers need to be approached. There are normally six reports on each applicant by the time that the Assessment Panels meet. The full set of reports by reviewers is sent to the Assessment Panels four weeks before the individual meetings of the Panels. Any reports received after this date are forwarded to the Panels until the day before they meet.

Specialist Committees assess the standing of applicants amongst their peers and recommend a rating to applicants on the basis of the statements contained in the reviewers' reports and on the objectivity of these reports in the light of the factual information contained in the submission documentation.

The reports by reviewers are received by all members of the Assessment Panels as described above. The documentation contains all the information that Panel members require to assist the NRF in establishing the ratings of applicants.

The formal meetings of the Assessment Panels, at which ratings are discussed and assigned, are preceded by an informal session during which the Chairperson of the Assessment Panels briefs the members of the Specialist Committees on the evaluation and rating process. This session also provides an opportunity for Specialist Committee members to seek clarity about certain issues. The briefing session is attended by the Chairperson, the Specialist Committee members, staff of Monitoring & Evaluation and sometimes the Assessor.

After the briefing has been completed the Specialist Committee members meet to undertake the following tasks:

  • Discussion of the research performance of each applicant and assessment of reports by reviewers to agree on the rating which the committee wishes to recommend for each applicant at the meeting of the Assessment Panel.
  • Identification of constructive feedback from reports by reviewers that may be usefully given to applicants for consideration in respect of their future research.
  • Discussion about the thoroughness and objectivity of the reports received from reviewers and the assessment of these reports according to a scale provided by the NRF.

When the Specialist Committees have completed the above tasks they are joined by the Chairperson and the Assessor, and the formal meetings of the Assessment Panels commence. Each applicant is discussed in turn to decide on a rating. The Convener of the Specialist Committee presents the rating recommended by this committee, the Assessor puts forward his/her independent rating and the Chairperson steers each case towards a decision, often also playing the role of a second independent assessor. There are three possible outcomes to the discussions about each individual application:

  • In the great majority of cases there is consensus between the ratings proposed by the Specialist Committee and the Assessor, with the Chairperson in agreement. Where the proposed rating is in the B, C or Y category, the final decision on the rating is taken at the Assessment Panel meeting. If consensus is reached about an A or P rating the application is referred to the Executive Evaluation Committee (EEC) for final approval.
  • Sometimes consensus cannot be reached between the Specialist Committee and the Assessor and/or the Chairperson on the rating of a particular applicant. All these cases are referred to the EEC. 
  • It can be decided at the meeting that there are insufficient reports by reviewers on which to base a decision. In this case additional new reports are solicited from reviewers and the evaluation is finalised after the meeting through telephonic and written contact between members of the Assessment Panel.

Once a decision has been taken about the rating of a particular applicant, be this by the Assessment Panel, the EEC, this is communicated to the applicant, together with any feedback identified by the Specialist Committee, via the appropriate authority at the employing institution.

Process of appeal

Written appeals against ratings assigned to applicants must be lodged by the appropriate authority at employing institutions on behalf of applicants within three months of the date on which the NRF informed the institution and the applicant about the rating. Appeals are screened by Monitoring & Evaluation and returned to the higher education institution for modification if necessary. Monitoring & Evaluation divides the appeals into two groups, those appeals that are lodged against decisions taken by the Assessment Panels and those that are lodged against decisions taken by the Executive Evaluation Committee (EEC).

Appeals against decisions taken by the Assessment Panels are heard by the EEC. Appeal documents are first referred by Monitoring & Evaluation to the Specialist Committee and Assessor that were involved in the rating of the applicant lodging the appeal. Their comments together with the appeal documents, which consist of the applicant's original submission, the reports by the reviewers, a record of the meeting of the Assessment Panel and the letter of appeal are then forwarded to the EEC for a final decision. The decision of the EEC is communicated to the higher education institution lodging the appeal.

Should there be an appeal against a decision of the EEC, it is heard by the Appeals Committee. First Monitoring & Evaluation will consult with the NRF President or Vice-President about the grounds for the appeal, who will make a pronouncement on whether a particular appeal will be heard or not. In cases where an appeal is deemed invalid, the President or Vice-President will communicate with the appropriate authority at the institution to give reasons why the appeal as submitted is not regarded as valid.

Valid appeals are then considered by the Appeals Committee, which takes the following into account: decisions taken by the relevant Assessment Panel and the EEC, the submission by the applicant, reports by reviewers and the letter of appeal by the employing institution as well as any further documentation for clarification of issues. The Appeals Committee judges on the fairness and correctness of the evaluation processes. It does not make a pronouncement on a rating. If the Appeals Committee is at variance with decisions of the EEC it refers these cases back to the EEC pointing out specific shortcomings in the process that warrant review. After reviewing the process the EEC provides feedback to the Chairperson of the Appeals Committee.

 
 

Page last updated on 216 October 2009