Home | Sitemap | Contact | NRF Home

Monitoring & Evaluation

Enhancing the quality of research in South Africa
Evaluation and rating
 
Apply to be rated
Eligibility issues
Application for evaluation and rating
Key Research Areas and Types of Research Outputs
Next closing date and submission of applications
Research outputs that can be assessed
Period of evaluation
Assessment panels
Reviewers
Definition of NRF rating categories
NRF definition of research
Expanded Thuthuka Rating Track
Re-evaluation
Outcome of the evaluation and communication of result
Feedback to researchers from reports by reviewers
Appeals
Alignment of the rating and proposal processes
 
The evaluation and rating of individuals is based primarily on the quality of the research outputs in the recent past and is undertaken by national and international peers/reviewers who are requested to critically scrutinise the completed research. "Recent", in the context of the NRF evaluation and rating system, means outputs of the past eight years, i.e. from 1 January 2002 to 31 December 2009 (for applications to be submitted on 28 February 2010). The onus is therefore on applicants to present the necessary evidence in a manner that will enable reviewers to make fair judgement.

Submissions for evaluation and rating are regarded as either new evaluations (first-time applications) or re-evaluations (applications preceded by a former application for evaluation and rating).
 

Eligibility issues

 The following persons are eligible to apply for evaluation and rating:

  • Researchers who are employed and remunerated on a full-time, part-time or contract basis at South African (SA) higher education institutions (HEIs) (including private HEIs in SA), museums or at NRF recognised research institutions*]. (For retired academics/researchers bullet 6 applies.)

  • Full-time temporary staff members at SA HEIs and SA museums who enjoy the conditions of service that normally also apply to their permanent full-time colleagues

  • Postdoctoral fellows who are conducting research in a full-time capacity at SA HEIs.

  • Persons who are being considered for full-time posts at SA HEIs and SA museums (i.e. applications from such persons can be submitted by the HEI considering their employment before they are actually employed).

  • Persons holding joint appointments between a SA institution and a foreign institution who are actively involved in research capacity building locally. Such persons should be employed by the SA institution in a full-time capacity for a period of at least six months per annum and should spend at least six months per annum in SA.

  • Persons holding joint appointments between two institutions within SA (of which at least one appointment should be a formal association with an NRF recognised research institution1) who are actively involved in research capacity building locally and who are still actively mentoring/training postgraduate students/young research staff. Such persons must indicate which institution is the primary institution, i.e. the institution to which the rating should be linked.

  • Retired academics/researchers must meet all criteria bulleted below

o    resident in SA, and

o     formally affiliated to a SA HEI (e.g. appointed as an emeritus professor, honorary research associate/professor, supernumerary/contract employee) and whose application for rating is formally supported and endorsed by a SA HEI to which he/she is or is anticipating to be affiliated, and

o     active researchers with a distinguished track record in research and postgraduate student supervision, and

o     still actively mentoring/training postgraduate students/young research staff.


*An NRF recognised research institution is one that meets all the following minimum requirements:
 any South African institution that conducts basic research or applied research,
 of a pre-competitive nature,
 for the benefit of the long-term knowledge base of the country,
 within the declared NRF focus areas,
 containing a research training component leading to master’s degrees and doctorates, while being committed to equity and redress.
 
 
Application for evaluation and rating
Applications are submitted electronically and can be accessed at: http://nrfinterim.nrf.ac.za/nrfapp/ The instructions on how to complete and submit the application are available at the same internet address.

Re-evaluation
Researchers at recognised research institutions who have been rated A, B, C, P, Y or L will be invited to submit documents for re-evaluation in approximately five-year cycles. Should a researcher choose not to respond to this invitation, his/her rating will lapse and he/she will not qualify for funding from the Focus Area Programmes and Institutional Capacity Development. Applicants who are not placed in one of the above categories should wait for three years before they apply for re-evaluation. In these cases, the onus to apply for re-evaluation rests with the applicant.

Special re-evaluation
An applicant may apply for special re-evaluation sooner than the cycles indicated above if, since a previous evaluation, an applicant has shown such progress that, in the opinion of the relevant authority at the employing institution, the existing rating is quite inconsistent with the applicant’s present standing. A newly prepared application must be sent to the NRF, together with a motivation from the appropriate authority stating the reasons why a special re-evaluation is requested.
 

Key Research Areas and Types of Research Outputs
Applications for evaluation and rating are considered by 22 Specialist Committees which are constituted according to discipline(s). Applicants may require greater clarity about the research areas covered by individual Specialist Committees so that they can select the most appropriate Assessment Panel to consider their application. The document “Key Research Areas and Types of Research Outputs” contains the following details for each Specialist Committee:
 Key research areas covered
 Research outputs taken into account in the rating process and their relative importance
 Boundaries and overlaps between Specialist Committees, allowing applicants whose field of research straddle more than one committee to ascertain where their application is best handled

Next closing date and submission of applications
 The electronic submissions system is accessible, all year round, to potential applicants who wish to update their CVs. The online application form for evaluation and rating to will be accessible from 15 October 2009.
 The closing date for 2010 will be 28 February 2010.
 Please note: Applications must be submitted to the NRF by the relevant authority at the institution where the applicant is employed. Applicants must determine the internal closing dates within their own institutions.
 No late applications will be accepted.
 Applications must be
submitted electronically via the NRF's electronic
submissions system to the applicant's institutional research administration, for screening, validation and electronic transmission to the NRF. Applicants are urged to determine the internal closing date for submission to the designated authority at their employing institution.
 Applications must be comprehensive and contain sufficient detail to allow for proper assessment, as the information supplied provides the basis on which judgements are based. If applicants do not adhere to this request, their applications will be returned, and they will only be able to submit at the next closing date.
 Applicants are advised to complete their submissions as soon as possible to prevent IT systems' overload at their institutions.

 

Research outputs that can be assessed
The evaluation and rating of researchers is based primarily on the quality of the research outputs during the past eight years.

Research outputs could include publications in peer-reviewed journals, books or chapters in books, peer-reviewed published conference proceedings, other significant conference proceedings including published abstracts, keynote or plenary addresses, patents, artefacts and products, technical reports and any other measurable outputs. These could include annotated bibliographies, CD-ROMS, development and production of software, electronic publications, plant breeding rights, research guides, vaccines, web sites etc. For all these research outputs appropriate and concise descriptions must be included in the application for evaluation and rating.

In all cases above, the actual research outputs must not be submitted, but should be made available upon request.

Period of evaluation
The assessment period for which research outputs will be considered for evaluation is seven years, which, for the closing date on 28  February 2010 is taken from 1 January 2002 to 31 December 2009. In addition to the research outputs of the last eight years, up to a maximum of the ten best research outputs of the period preceding the last eight years (i.e. before 1 January 2002) could be included.

Assessment Panels
Assessment Panels are constituted from members of the respective Specialist Committees, an independent Assessor and a Chairperson who is either an member of the NRF Executive or a researcher of international repute.

Specialist Committees
Three to six members, mostly from the South African research community, are appointed to the Specialist Committees, based on their research experience and reputation in the research community. The role of the Specialist Committees is to assess, based on the reviewers' comments,  the standing of applicants amongst their peers and to assign ratings to applicants on the basis of the statements contained in the reviewers' reports. It is also their role to assess the objectivity of these reports in the light of the factual information contained in the submission documents.
 

List of NRF Specialist Committees 2009/2010
Animal and Veterinary Sciences Information Technology
Anthropology, Development Studies, Geography, Sociology and Social Work Law
Biochemistry, Molecular and Cell Biology Literary Studies, Languages and Linguistics
Chemistry Mathematical Sciences
Communication, Media Studies, Library and Information Sciences Microbiology and Plant Pathology
Earth Sciences Performing and Creative Arts, and Design
Economics, Management, Administration and Accounting Physics
Education Plant Sciences
Engineering Political Sciences, Policy Studies and Philosophy
Health Sciences Psychology
Historical Studies Religious Studies
 
NRF Assessors
Three or four Assessors are appointed annually, normally for a period of three consecutive years. They are selected from the ranks of highly respected researchers who normally have served on a Specialist Committee in the past, who are therefore familiar with the process and who have earned a reputation during their tenure on these Committees for their wisdom and objective judgements. Assessors must ensure that the assessment process is fair and independent and that the same criteria are applied consistently by all the Committees.

Some general comments about Assessment Panels
Applicants are given the opportunity to select the Assessment Panel that should consider their application. Should more than one panel be appropriate, as in the case of multidisciplinary studies, applicants may select more than one panel to which their application should be submitted, although, generally applications are submitted to one panel only.

The NRF may refer applications to panels other than the one(s) selected by applicants for their consideration if deemed necessary. Ad hoc panels may also be constituted if the standing panels are unable to handle a particular application. It must be stressed that the role of these panels in the evaluation process is to make recommendations to the NRF based on the assessment of reviewers' reports and that the members of these panels do no act as peers of the applicants.

Reviewers
The selection of appropriate reviewers constitutes the very essence of the peer-review system that supports the evaluation and rating of individuals. The Specialist Committees and the applicants for evaluation and rating are thus expected to show great circumspection in nominating reviewers.

Applicants are requested to supply names of at least six but not more than ten research active researchers who are best able to assess the scope and impact of their recent research and other scholastic outputs, activities and contributions. They are also requested to indicate their relationship with the reviewer and to give reasons for each nomination in order to provide the Specialist Committee with additional information for the selection of reviewers. Applicants are also given the opportunity to indicate which reviewers should not be approached by the NRF. Although the NRF would normally not approach such reviewers, it does reserve the right to do so if this is regarded necessary.

The Specialist Committees are requested to nominate six reviewers of whom, generally speaking, three should be selected from those listed by the applicant and another three provided, independently, by the Specialist Committee. (NB: This is a general guideline and does not preclude scenarios where for example, all six reviewers could be chosen from the applicant’s list.)

Persons who serve on the Specialist Committees should be wise in the broader context of their fields with extensive national and international networks to assist them in the identification of suitable reviewers. These persons should make research contributions in their own right and be capable of fair evaluation. There is no substitute for the wisdom of members of the Specialist Committees who are ultimately responsible for the selection of reviewers and whose task it is to select a mix of reviewers from whose reports the impact of applicants’ research in their fields and in broader fields can be gleaned.

Such an assessment is based on:
 The quality of the research-based outputs of the last seven years as well as the impact of the applicant's work in his/her field and how it has impacted on adjacent fields.
 An estimation of the applicant's standing as a researcher in terms of both a South African and international perspective.
 The quality and appropriateness of the journals, books, conference proceedings, etc in which the applicant's work is published.
 Other research-based contributions.

Definition of NRF rating categories
The definitions of the rating categories are given below. Descriptions of sub-categories in the A, B C and Y categories have also been indicated. The definition of research at the end of the table should be consulted to clarify the interpretation of research as indicated in the various categories.

 It must be borne in mind that the peer evaluation process is intricate and not mechanistic. Ultimately the judgement of the members of the Assessment Committees and their wisdom which has some intangible components must be relied upon. Hence interpretation of words such as ‘broad field’, ‘narrow area’, ‘considerable’, etc form an important part of the Assessment Committees’ task in their role of assessment of reviewers’ reports.

Decisions on the ratings of applicants are taken by the relevant Assessment Panels or, in some instances, by an Executive Evaluation Committee (EEC). These decisions are based on the Panels' assessment of the reviewers' reports in the light of the applicants' submission. It must be borne in mind that the NRF peer evaluation process is intricate and not mechanistic. Ultimately, the judgement of the members of the Assessment Panels and their wisdom, which has some intangible components, must be relied upon. Hence, interpretation of words such as 'considerable' form an important part of the Assessment Panels' task in their role of assessment of reviewers' reports.

The following rating categories and accompanying descriptions are used for all applications received:

Category

Definition

Sub-category

Description

A

Researchers who are unequivocally recognised by their peers as leading international scholars in their field for the high quality and impact of their recent research outputs.

A1

A researcher in this group is recognised by all reviewers as a leading scholar in his/her field internationally for the high quality and wide impact (i.e. beyond a narrow field of specialisation) of his/her recent research outputs.

A2

A researcher in this group is recognised by the overriding majority of reviewers as a leading scholar in his/her field internationally for the high quality and impact (either wide or confined) of his/her recent research outputs.

B

Researchers who enjoy considerable international recognition by their peers for the high quality and impact of their recent research outputs.

B1

All reviewers concur that the applicant enjoys considerable international recognition for the high quality and impact of his/her recent research outputs, with some of them indicating that he/she is a leading international scholar in the field.

B2

All or the overriding majority of reviewers are firmly convinced that the applicant enjoys considerable international recognition for the high quality and impact of his/her recent research outputs.

B3

Most of the reviewers are convinced that the applicant enjoys considerable international recognition for the high quality and impact of his/her recent research outputs.

C

Established researchers with a sustained recent record of productivity in the field who are recognised by their peers as having:

·          produced a body of quality work, the core of which has coherence and attests to ongoing engagement with the field

·          demonstrated the ability to conceptualise problems and apply research methods to investigating them.

C1

While all reviewers concur that the applicant is an established researcher (as described), some of them indicate that he/she already enjoys considerable international recognition for his/her high quality recent research outputs.

C2

All or the overriding majority of reviewers are firmly convinced that the applicant is an established researcher (as described).

C3

Most of the reviewers concur that the applicant is an established researcher (as described).

P

Young researchers (normally younger than
35 years of age), who have held the doctorate or equivalent qualification for less than five years at the time of application and who, on the basis of exceptional potential demonstrated in their published doctoral work and/or their research outputs in their early post-doctoral careers are considered likely to become future leaders in their field.

 

Researchers in this group are recognised by all or the overriding majority of reviewers as having demonstrated the potential of becoming future leaders in their field, on the basis of exceptional research performance and output from their doctoral and/or early post-doctoral research careers.

Y

Young researchers (40 years or younger), who have held the doctorate or equivalent qualification for less than five years at the time of application, and who are recognised as having the potential to establish themselves as researchers within a five-year period after evaluation, based on their performance and productivity as researchers during their doctoral studies and/or early post-doctoral careers.

Y1

A young researcher (within 5 years from PhD) who is recognised by all reviewers as having the potential (demonstrated by research products) to establish him/herself as a researcher with some of them indicating that he/she has the potential to become a future leader in his/her field.

OR

A young researcher (within 5 years from PhD) who is recognised by all or the overriding majority of reviewers as having the potential to establish him/herself as a researcher of considerable international standing on the basis of the quality and impact of her/his recent research outputs.

 

Y2

A researcher in this group is recognised by all or the overriding majority of reviewers as having the potential to establish him/herself as a researcher (demonstrated by recent research products).

 

It is important to note that placement in the categories P, or Y can occur once only. As a result, the timing of submission of documents for evaluation can be very important (especially in the case of the P category). Bearing in mind the definition of the P category, it is imperative that the published research outputs of the doctoral and/or postdoctoral period have been subjected to international peer scrutiny before submitting documents to the NRF for evaluation. In the ideal scenario, outputs from both the doctoral and postdoctoral periods should be available for peer scrutiny.

Note: It is not necessary for an applicant to have a doctoral degree when applying for an A, B or C rating. However, an applicant applying for a Y or P rating must have a doctoral degree or equivalent. Applicants who are under 35 years of age and already established can also qualify for an A, B, or C rating.

Expanded Thuthuka Rating Track

University academic staff that can demonstrate that they were unable to establish themselves as researchers due to limited research opportunities e.g. working in a former historically disadvantaged institution, moving to and from industry or due to family responsibilities, may apply for research support in the Rating Track of the expanded Thuthuka Framework.  The intended purpose of the Thuthuka Rating Track is to support researchers to become established in their chosen research field.

 The Call for submissions of research proposals for funding will open in early 2010 (for funding from 2011 onwards). The Thuthuka Rating Track support will be offered, on a competitive basis, for a predetermined period only. Applicants must present a plan that will lead them to apply for NRF rating at the end of the funding period.

 Further information and details regarding the Rating Track in the expanded Thuthuka Framework will be released in 2010.

NRF definition of research
For purposes of the NRF, research is original investigation undertaken to gain knowledge and/or enhance understanding.

Research specifically includes:
  the creation and development of the intellectual infrastructure of subjects and disciplines
(e.g. through dictionaries, scholarly editions, catalogues and contributions to major research databases);
  the invention or generation of ideas, images, performances and artefacts where these manifestly embody new or substantially developed insights;
  building on existing knowledge to produce new or substantially improved materials, devices, products, policies or processes.

It specifically excludes:
  
routine testing and analysis of materials, components, instruments and processes, as distinct from the development of new analytical techniques.
  the development of teaching materials and teaching practices that do not embody substantial original enquiry.

 Download the NRF definition of research in MSWord.

Re-evaluation
Re-evaluation takes place at regular intervals either at the initiative of the NRF (for categories
A, B, C, P or Y or at the initiative of the applicant whose evaluation has lapsed or did not fall into one of the above categories at the previous assessment. It can also take place at the request of organisations requiring a special re-evaluation of specific applicants.

Re-evaluation by invitation of the NRF
Applicants with a valid rating will be invited by the NRF to submit documents for re-evaluation approximately five years after the previous evaluation. Details about the documents to be submitted for re-evaluation are issued together with the invitation for re-evaluation. If an individual elects not to respond to  the invitation for re-evaluation, his/her rating will lapse. Researchers who allow their ratings to lapse must note the this will result in the end of access to NRF funding for research.

Re-evaluation at the initiative of the applicant
Applicants who have not been rated in one of the defined categories may apply for re-evaluation through the relevant authority three years after their previous submission. In these cases, the onus to apply for re-evaluation rests with the applicant.

Special re-evaluation
If, since a previous evaluation, an applicant has shown such progress that according to the relevant authority the outcome of the previous evaluation is inconsistent with the applicant's present standing, a special re-evaluation may be requested, i.e. even before three years have elapsed in the case of applicants where no rating was assigned, or before the NRF has extended an invitation for re-evaluation in the case of applicants with a valid rating.

In the case of a special re-evaluation, a newly completed application for evaluation and rating must be submitted to the NRF together with a motivation from the appropriate authority justifying why a special re-evaluation is requested. In this case, the whole process is repeated, new reviewers approached and a complete re-evaluation is undertaken.

Outcome of the evaluation and communication of result
The outcome of the evaluation will be conveyed as soon as the rating has been finalised. Some evaluations will take longer than others depending on whether they have to be referred to the EEC or whether more reports by peers have to be solicited.

Normally
Assessment Panels meet during September to November of a particular year to assess applications. Results will normally be communicated to the applicant, together with feedback identified by the Specialist Committee, via the appropriate authority at the employing institution by the end of the same year. However, there may be cases that can only be finalised later (e.g. those cases where more reports by reviewers have to be solicited, or which have to be referred to the Executive Evaluation Committee (EEC), etc.) In these cases the applicants will be informed that there will be a delay and that the results will be made available immediately after the finalisation of the rating.

Feedback to researchers from reports by reviewers
Realising that feedback from the evaluation and rating process may be of great value to researchers, the NRF has the following policy in this regard:

Feedback from the process

  • Information is provided on how the Assessment Panel determined placement of the applicant in a rating category and sub-category based on the reviewers’ report.

Commendations and recommendations from the reports of reviewers

  • Specialist Committees also identify feedback from reports of reviewers that could potentially be helpful to applicants in their future careers.

N.B.:  Reviewers will remain anonymous.

The Monitoring and Evaluation unit will provide the above feedback as an attachment to the outcome letter if the applicant requested feedback in his application.

It is important to keep the following in mind:

·       The most important feedback from the evaluation process is the rating itself. It represents the overall opinion of reviewers regarding the standing of the applicant as a researcher, based on the research outputs over the last eight years.

·       Commendations and recommendations are selected because of its potential value to researchers in their future work, but the comments provided might be something applicants wish to disregard completely.

·       Feedback cannot be a basis for an appeal against the outcome of an evaluation.

While the NRF will engage in discussion about all aspects of the evaluation and rating process it cannot enter into any discussion on the contents of feedback supplied.

Appeals
The relevant authority at an institution can lodge a written appeal if it considers the result of a particular evaluation outcome to be seriously out of line with the institution’s own assessment of the applicant. Only those documents on which the original decision was based will be considered by the Executive Evaluation Committee (EEC) or the Appeals Committee. No new documentation on research outputs achieved subsequent to the submission of the application for evaluation and rating may be submitted. Any scholarly achievements of the applicant postdating the submission will not be taken into account in an appeal.

Appeals must be lodged within three months of the date of the NRF’s letter informing the relevant authority and applicant of the outcome of evaluation and rating. Letters of appeal should not exceed two pages.

The following guidelines will assist employing institutions and applicants when they consider lodging an appeal.
  Was the result of the evaluation seriously out of line with the applicant’s or institution’s assessment?
By serious is meant that the rating which was awarded differed by one rating category from that suggested by the institution. A case could possibly be made if the applicant’s institution had suggested a C1 rating and the outcome was a C3 rating. Appeals should generally not be lodged if an applicant received a C3 rating and it is felt he/she should have been placed one sub-category higher at C2.
  What are the main reasons for the appeal?
Institutions should carefully consider this question. An appeal is best aborted if it is, for example, based on any new research outputs, or a change of status of research outputs e.g. a paper in preparation at the time of submission has in the meantime been accepted for publication, or on the feedback which has been supplied.
Feedback from reviewers’ reports constitutes selected extracts. The supplied feedback is not the overriding determinant in the rating decision; it simply reflects the comments identified by the relevant Specialist Committee that could possibly be useful to the applicant. If the applicant or institution dislikes the feedback or disagrees with it, they may ignore it.
  Are the NRF rating categories well understood?
Institutional authorities and applicants should make quite sure that they understand the rating categories thoroughly before lodging an appeal.
  Do not submit other reports received from reviewers, e.g. those solicited by the employing institution for promotion purposes, should not be included with the appeal.

Incentive funding
Researchers who are rated by the NRF may apply for incentive funding which is available for the duration of the rating. The funding is not project-related and may be used for any research-related costs. The only 'string' attached to this funding is the submission of an Annual Progress Report at the end of the year to indicate how the funds were spent.
 

Res

 

Page last updated: 16 October 2009