Monitoring & Evaluation

Enhancing the quality of research in South Africa
Research niche area (RNA) appraisals
 

Process for the appraisal of research niche areas

1. Introduction

The introduction of new frameworks for the Technikon Research Development Programme (TRDP) and the University Research Development Programme (URDP) in 2001 resulted in the declaration of 2002 as an interim year for the appraisal of Research Niche Areas (RNAs) submitted to the NRF for possible support within these programmes. RNAs that were judged meritorious were provisionally approved, through a shortened approval process, pending the submission of five-year business plans/frameworks for evaluation in April 2002.

 
 

During 2002 a more extensive process was developed and implemented for the appraisal of five-year business plans submitted for RNAs at Historically Black Universities (HBUs) and Technikons. This process is being followed since and is being coordinated by the NRF Monitoring & Evaluation. The outcome of the appraisal determines the final status of the RNAs within the NRF Institutional Research Development Programmes within the hierarchy as outlined in the frameworks of the two programmes (see Appendix 1 and Appendix 2). RNAs that are approved are eligible for support for a period of up to five years provided their progress is considered satisfactory. RNAs that have received support for the past five years and that wish to continue are expected to be sufficiently developed to progress to the next level within the hierarchy of support.

In developing the process for the appraisal of RNAs within the TRDP and the URDP the following factors were taken into account:

  • The category into which the RNAs fall. Research Niche Areas fall into one of two categories i.e.
    • ongoing areas supported in the TRDP and URDP precursor programmes
    • totally new areas.
  • The outcome of the review of the TRDP and URDP precursor programmes during 2001, which included an extensive review of all ongoing activity areas/research thrusts continuing as RNAs.

With effect from July 2003 one NRF manager is responsible for the management of both the TRDP and the URDP and will be referred to as Manager: TRDP/URDP in this document.

2. The Review Process

The review process involves a combination of the following generic steps depending on the history of the RNA, changes to the RNA's focus as well as the composition of the RNA team:

2.1 Internal review

The TRDP/URDP management undertakes an initial screening and information gathering exercise before submitting the documents to Monitoring & Evaluation for screening and further processing. The TRDP/URDP and Monitoring & Evaluation management subsequently meet to determine the appropriate review process to be followed for each RNA. Submissions that do not address the requested information are referred back to the TRDP/URDP to be returned to the institution concerned for further development or improvement.

2.2 External Review

RNAs accepted by Monitoring & Evaluation for further processing are appraised by means of one of the following types of review involving suitable experts:

  • Postal review by at least three experts
  • On-site review during which at least two experts visit the institution concerned to discuss the proposed RNA with the relevant stakeholders (i.e. the RNA team members and students involved, the research management of the respective Higher Education Institution (HEI) and NRF staff).
  • Review by Panel is used in cases where an on-site visit is not considered necessary and where a cluster of similar RNAs can be considered by the same panel of experts (usually three) at a meeting held at the NRF in Pretoria.

Reviewers for the three types of review mentioned above are selected by the NRF Monitoring & Evaluation from recommendations by the institution concerned, the TRDP/URDP and own databases/advisers. The reviewers are required to provide Monitoring & Evaluation with their individual appraisal reports based on the documents provided by the RNAs/HEIs and the information received during the on-site visit, if applicable. Monitoring & Evaluation draws up a concise document based on the feedback from the reviewers. Monitoring & Evaluation, the RCD Subcommittee as well as the Executive Director: Research Development & Support also make recommendations regarding the future support of the RNA.

2.3 RCD Subcommittee

The RCD Subcommittee shall consist of:

  • TRDP/URDP Manager
  • Manager: Monitoring & Evaluation
  • Relevant Professional Officers (POs) of the TRDP/URDP and Monitoring & Evaluation
  • Additional members co-opted by the RCD Subcommittee, where appropriate.

The tasks of the RCD Subcommittee are to:

  • where necessary, make recommendations on the process to be followed and the status of the RNAs (i.e. Activity, Unit or Centre)
  • interpret the reviewers' reports emanating from the postal, on-site or postal review
  • consider the recommendations of the reviewers and Monitoring & Evaluation
  • make recommendations on each of the proposed RNAs to the Executive Director: Research Development & Support for his/her recommendations and submission to the Executive Subcommittee
  • address any other issues pertaining to the appraisal process.

2.4 Executive Director: Research Development & Support

The role of the Executive Director: Research Development & Support in the appraisal process is to:

  • ensure that the NRF's Research Capacity Development strategies have been adhered to
  • indicate in advance to Monitoring & Evaluation those RNAs which he/she feels should be referred back to the RCD Subcommittee or the relevant programme, and provide reasons
  • submit his/her own recommendations on each RNA to the Executive Subcommittee, through Monitoring & Evaluation, for discussion and final decision
  • where necessary, follow up on the decisions and recommendations of the Executive Subcommittee.

2.5 Executive Subcommittee

This committee has powers delegated to it by the NRF Executive to take decisions from which contractual agreements with the institutions will flow. For this reason the Executive Subcommittee consists of:

  • Executive Director: Knowledge Management & Strategy (Chairperson)
  • Executive Director: Research Development & Support
  • Executive Director: Business and Industry Partnerships Programmes
  • TRDP/URDP Manager
  • Manager: Monitoring & Evaluation and/or alternate

In the absence of the Executive Director: Knowledge Management & Strategy, the Managing Director of the Research Innovation Support and Advancement (RISA) will act as Chairperson of the Subcommittee.

In view of the potentially large number of RNAs that need to be processed, it may happen that not all members of the Executive Subcommittee can avail themselves for all the scheduled meetings due to work commitments. In this case the meeting can proceed with the following quorum:

  • Executive Director: Knowledge Management & Strategy (Chairperson)
  • Executive Director: Research Development & Support
  • Executive Director: Research Promotion & Social Sciences and Humanities or Executive Director: Business and Industry Partnerships Programmes
  • TRDP/URDP Manager
  • Manager: Monitoring & Evaluation

The role of the Executive Subcommittee is to:

  • consider the RNA submissions, the reviewers' reports and the recommendations of Monitoring & Evaluation, RCD Subcommittee as well as the Executive Director: Research Development & Support, and take the final decision on the status of the RNAs.
  • satisfy itself that proper procedures in the appraisal of the RNAs have been followed and that the integrity of the appraisal process was not compromised in any way.

After completion of the review of all (or the majority) of RNAs of a particular institution, a contractual agreement will be drawn up between the NRF and the institution concerned i.t.o. the successful RNAs. The agreement will be signed by the Vice-Chancellor of the institution and the Managing Director of RISA.

3. Operational Framework

In order to ensure an efficient and successful implementation of the appraisal of RNAs, it is important to clearly outline the roles which institutions, the TRDP/URDP and Monitoring & Evaluation are expected to perform.

3.1 Institutions

The role of the institutions in this process is to:

  • take ownership of the RNAs
  • screen the applications for completeness, accuracy, appropriateness for the institution and verify the information provided
  • submit applications timeously
  • assist with organising on-site visits and ensure that relevant RNA team members are informed and attend on-site appraisals
  • follow up on the recommendations of the reviewers and the NRF

3.2 TRDP/URDP

The role of the TRDP/URDP is to:

  • receive RNA applications and sort them into one of the categories
  • verify the information contained in the RNA applications
  • collate necessary documentation
  • send documentation to Monitoring & Evaluation (electronically, plus one hard copy)
  • assist with the identification of reviewers
  • follow up on the recommendations of the Executive Subcommittee.

3.3 Monitoring & Evaluation

The role of Monitoring & Evaluation is to:

  • identify and select suitable reviewers
  • make all the arrangements for the appraisal (either on-site, panel or postal)
  • draw up a concise document based on the feedback from reviewers and make preliminary recommendations
  • make arrangements for meetings of RCD and Executive Subcommittee
  • communicate final decisions to the institutions by letter signed by the Managing Director of RISA
  • provide secretariat for the appraisal and appeals process.

4. Appeals Process

The appropriate authority at an HBU or Technikon can lodge a written appeal if it considers the appraisal process to have resulted in a decision that is contestable. The following appeals process has been approved by the NRF.

4.1. The relevant authority at an HBU or Technikon can approach the TRDP/URDP Manager for additional information and clarification in cases where the NRF has communicated its decision not to support a proposal for an RNA. The TRDP/URDP Manager will then supply any relevant information over and above the information already supplied. Such an approach and response will be in writing.

4.2. If the institution is not satisfied with the explanation provided and believes that the proposal as submitted was not fairly handled or reasonably reviewed it can institute a formal appeals process. Such an appeal will be based on the tenet that the appraisal process was not fair or reasonable, both substantively and procedurally. No revisions made to the original submission will be considered in the appeals process.

4.3. The relevant authority in the institution (i.e. the Director of Research, Vice-Rector charged with research or the Rector) will lodge an appeal with the NRF Executive Director: Knowledge Management & Strategy providing reasons for the appeal. Such an appeal must be lodged within three months of the date of the NRF's letter informing the institution about the outcome of the RNA appraisal.

4.4. The Executive Director: Knowledge Management & Strategy shall forward an appeal to Monitoring & Evaluation, which will process the appeal on his/her behalf. Monitoring & Evaluation will solicit comments of the relevant Executive Subcommittee members (i.e. NRF Executive Directors and the TRDP/URDP Manager). A dossier containing all the relevant information (i.e. background documentation and solicited comments) shall be forwarded to the Executive Director: Knowledge Management & Strategy for his/her recommendations and for onward transmission to the Managing Director: RISA.

4.5. The Managing Director: RISA will consider the appeal in terms of the process that was followed to reach a decision as well as the reports by reviewers, where these are applicable. The Managing Director: RISA will either decide to uphold the original decision, decide to reverse the previous decision and approve the RNA for support or he/she may request additional information or additional reviews before taking a decision.

4.6. The Managing Director: RISA will rule on the appeal and inform the institution accordingly. His/her decision will be final.

4.7. The NRF cannot guarantee that a successful appeal will result in financial support for an RNA even if error is established in the initial appraisal. This is because programme budgets and priorities may factor into the decision to support an RNA or not.

4.8. An appeal has to be based on the documentation that was submitted and which served as the basis of the appraisal. Changes to the submission documentation or changes that have taken place subsequently to the submission of documentation cannot be considered in an appeal. However, if there have been substantial changes to activities in the RNA the HBU/Technikon can engage with the programme management about a possible resubmission of a substantively revised proposal. In this case the submission, if accepted, will be regarded as a new proposal which will be subjected to the usual appraisal procedure.

Download the above document in either MS Word or pdf (coming soon).