Ministry of Foreign Affairs
 
  Listen to the text     Normal screen 
  Full table of contents     Previous     Next     Similar documents  

2 Methodology

This evaluation is based on a participatory approach. Findings have been discussed with the stakeholders in the Ministry of Foreign Affairs (MFA), the Norwegian Institute of Human Rights (NIHR) and the Norwegian Refugee Council (NRC) at three successive workshops before the presentation of the draft report. Consequent adjustments have been made. During the process, the stakeholders were supplied with notes on progression and findings.

The findings mainly originate from the desk study in Norway and from field studies.

In the desk study, the following sources were used:

  • Quantitative information mainly, but not exclusively, based on the NORDEM database, and NORDEM documents in the MFA archives
  • Information and viewpoints from the interviews with
    • NORDEM stakeholders in the concerned departments at MFA, at NIHR and at NRC. For the latter two, this included both staff and board members
    • Former NORDEM secondees at a focus group meeting
    • Persons who participated in the founding of NORDEM, including the then political leadership and NIHR and NRC board members
  • A questionnaire survey undertaken among the members of NORDEM
  • Information on organisations with aims similar to NORDEM’s in donor countries similar to Norway, through written material, telephone and e-mail correspondence.

Field studies were undertaken first in Bosnia and Herzegovina (BiH), and subsequently in Ethiopia and East Timor. During the field studies, briefing and debriefing meetings were held with the Norwegian embassies and interviews were undertaken, to the extent possible, with NORDEM secondees, representatives of the seconded organisations, counterparts and beneficiaries and key informants. The three cases were quite different, which is why we present the results in separate sections. While it is not possible to apply the same concepts automatically in the field studies, their different nature enriches the evaluation as a vivid and realistic demonstration of the broad range and nature of the NORDEM activities and of the consequent challenges and difficulties.

At the first of the above-mentioned workshops, in September, a discussion note based on the desk study was presented together with the draft questionnaires and interview guides. The second workshop, in October, concentrated on the preliminary report from the field study in BiH. Early November, the results of the statistical analysis and of the surveys of the NORDEM members and Norwegian embassies covering countries where NORDEM delegates have worked, were presented for comments together with a note on NORDEM management and administration. The main aim of the third workshop, in November, was to discuss the preliminary draft final report before finishing the draft final report.

Concerning the quality of the above sources, the results of the interviews were deemed adequate for the evaluation. This was also the case for the information gathered from organisations similar to NORDEM. Regarding the information gathered in MFA archives, MFA database on NORDEM and NORDEM’s database, the team encountered some difficulties as part of this information was insufficient and not easily found – in spite of assistance forthcoming from the NORDEM stakeholders and from the MFA evaluation section.

The MFA files contain most of the correspondence related to secondments, including job descriptions from requesting organisations, suggestions for candidates, budgets and grant requests from NORDEM, and grant letters from MFA to NIHR. At NIHR and NRC, the team had access to annual reports, databases, end-of-assignment and other reporting from the NORDEM members.

In certain areas, documentation was incomplete or not existing, for instance in the area of performance of secondees: policy and strategy papers, overviews of secondment grants and annual aggregations of figures. For instance, for 1997–98 only budget figures exist as the result of specific problems that were rectified. Nor was it possible to obtain a precise overview of other, "extra-NORDEM", secondments undertaken by MFA or related expenditures. For the early years, 1993–94, it was impossible to obtain complete figures, which is why we have not included the partial figures we actually managed to retrieve. Nevertheless, the quantitative information covers the period 1995–2000, in accordance with the ToR.

Evidently, the difficulties thus encountered also signify that available information has not been used as a tool for policy guidance or for management purposes. The lack of complete and consistent information certain lacunas in historical NORDEM files have resulted in a much more time having to be spent by the team to access documentation. This extra use of time is not deemed to have had a serious impact on the evaluation as a whole, however.

Under the field studies, interviews were made to the extent possible with NORDEM secondees, both current and, in some cases, former ones; representatives of seconded organisations; local counterparts; beneficiaries and key informants from other sectors in society or from the international community. The limited NORDEM presence in the field study countries has been sought remedied by interviews with former secondees.

Hence it has been possible to obtain responses to most of the questions raised in the ToR regarding the effects of NORDEM in the recipient organisations and countries. However, regarding an identifiable Norwegian effect on the results of the work of the seconded organisations, or on the nature of the work of these organisations, it has only been possible to provide very generalised accounts.

The survey among members, which was carried out by e-mail, resulted in 61 filled-out questionnaires from approximately 40% per cent of the sample, after a reminder. As with the interviews, the response percentage advises caution in drawing too firm conclusions, except in the case where the responses are unequivocal. The survey of 35 Norwegian embassies covering countries with NORDEM activities showed little else than that the embassies know very little about NORDEM, though they would like to know more. The team is aware of the methodological bias inherent in having talked or corresponded with a considerable number of NORDEM members, while obviously not having contact with those failed to qualify for membership. A non-biased approach is especially important when exploring recruitment criteria and methods. Another consideration is that some NORDEM members wish to be recruited again and may be aware of their consequent dependence on NORDEM and its decision-makers.

Overall, we believe it was possible to obtain most of the quantitative information needed. Also regarding qualitative information, it was possible to gain enough evidence to substantiate findings on most of the aspects NORDEM’s work. But, as mentioned above, in certain cases, information has been limited for various reasons. These limitations have been kept in mind when formulating our conclusions. Finally, note is made of the limited management information hitherto available to NORDEM.