Monitoring of Labour Market Policy in EU Member States

Introduction | Basic Information Reports | InfoMISEP Reports

NATIONAL LABOUR MARKET POLICIES

InforMISEP Reports

Monitoring of Labour Market Policy in EU Member States

Peter Auer

In recent years the monitoring of labour market policy has become a major issue for programme implementation both at individual EU member country level and European level. Politicians and programme administrators in individual member countries are directing more efforts than ever before to efficient administration and close follow-up of employment measures. The reasons for this include the increasing quantitative importance of spending on employment measures, tight public budgets, and moves towards a leaner, more efficient public service. These trends are often associated with far-reaching decentralisation/deconcentration of the programme delivery and an increase in European Social Fund (co)financing with follow-up obligations attached. The following article, which is an abridged version of a more comprehensive report on the topic, focuses on state-of-the-art monitoring of labour market policy (LMP) in the EU member countries. An overview of monitoring in EU countries is presented, the difficulties these countries face in setting up monitoring systems are discussed, and a model for monitoring is proposed.1

Although the exact borderline between monitoring and evaluation (and also between monitoring, controlling and reporting) is hard to draw, there seems to be a pragmatic understanding of what monitoring activities are: regularly conducted observation of statistical indicators of LMP input/output and performance (outcome) for the purpose of improving programme implementation (and sometimes programme design).

There is a clear trend in all the countries reviewed to engage in such performance-oriented monitoring of financial and physical (e.g. number of participants) indicators of active LMP, the aim of which is to enhance programme implementation. Monitoring is mostly seen as a tool to document performance or the failure in reaching preset goals, but not explain them, a task that falls to evaluation. Evaluation also studies the effects of LMP measures at the macro level, for example in determining the indirect effects of policies such as dead-weight, displacement or substitution, while monitoring is concerned more with the direct effects of policies. One has also to distinguish between input/output monitoring and outcome monitoring: the former is an instrument for programme administrators at all levels, and permits, for example, controlled spending and some cost-benefit analysis; the latter also provides information which goes beyond the mere administration and implementation of programmes and could affect programme design.

9.1.1. Increasing importance of monitoring

Some Member States may have more experience of monitoring than others, but it is still achieving greater significance in all countries. Some of the reasons for this development are:

- Fiscal constraints, lean administration and the reform of the public sector. New ideas on the public service, involving a shift from bureaucratic and centralised rule-making toward decentralised and market-oriented responsibilities and assessment systems and a general awareness of public spending (accompanied by increased auditing by controllers), has contributed to the pressure to set up monitoring systems. Together with decentralisation (see below), the introduction of private-sector management methods (e.g. management by objectives, "profit centres" or "cost centres") in the public sector has also compounded the need for assessment of results. Partially related to that need there are increasingly laws and programmes which have a "built-in" obligation to have the performance of labour market programmes monitored and evaluated.

- The decentralisation of LMP implementation. Handing down responsibilities for LMP programmes to lower administrative levels and thus reducing the control opportunities inherent in centrally administered programmes could complicate LMP monitoring. But decentralisation promotes the introduction of monitoring systems in two ways. First, it provides local agents with the information they need. Second, it provides central bodies with information on their local branches, enhancing their control scope as they can compare the results of their local agents. It also provides better targets for local agents by indicating to them the results obtained by other agents. The trend toward expanding the discretionary power of decentralised levels (e.g. the latitude that local employment offices have for deciding which measure - say, job creation or job training - is suitable for their clients) also entails increased monitoring. After all, local agents must justify their allocation choices.

- Increased ESF financing for national LMP and provisions for follow-ups. The reform of the structural funds in 1989 led to multi-annual programming and greater involvement of the Member States in monitoring the implementation and effects of measures co-financed by the ESF. Because ESF co-financing does not often lead to the design of new measures, as ESF funding is integrated in the financing of existing measures, the ESF (co)financed measures are to a considerable degree identical with national labour market policy measures and - in principle - there should be no difference in ESF and national LMP monitoring. However, there is considerable cross-country variation here. ESF financed measures are not a specific subject of this article, but because of the overlap both forms of monitoring are often linked (for ESF monitoring see the work of the Centre for European Education Expertise in Lyon and the MEANS Programme of the European Commission).

- The European Commission's efforts to make economic growth more employment intensive, not least by an activation of labour market policy (European Commission, 1994), increase the need for monitoring. In the wake of numerous European Council meetings, and especially since the resolutions of the European Council meeting in Essen (December 1994), employment and employment policies must be monitored by member countries and the Commission, which has to report on developments every year from late 1995 on. The Commission, with the help of the Member States, Eurostat and its employment observatories (MISEP, SYSDEM), intends to establish employment policy indicators to permit follow-up work on national policies in financial and physical terms. These activities, if successful, might also lead to an increase in national monitoring.

- Last but not least, technological progress has made monitoring possible at all levels of organizations, and data transfer through data networks has tremendously increased the potential scope and speed of monitoring. These changes have paved the way for setting up more sophisticated monitoring systems.

9.1.2. Monitoring in EU Member States

The general impression gained from the questionnaires and selected expert interviews is that all EU Member States are currently engaged in setting up or at least extending existing monitoring activities. Labour administrations are trying to follow the implementation of LMP more closely than used to be the case, but one cannot clearly place the different countries on a scale ranging from "no system" to "fully implemented system". The one exception is Sweden, which we would place at the top of the list, although even there the system is not yet fully in place. Otherwise the state of the art of monitoring is as yet rather heterogeneous even within countries; sophisticated monitoring of one measure might contrast with the absence of follow-up procedures for another. The following chart shows some variables which allow a limited "monitoring of monitoring" for most EU countries. The chart is an outcome of the empirical material we had at our disposal (the questionnaires, selected expert interviews and documents) and was also returned to the countries for validation.

Monitoring institutions: In most countries, the ministry of labour and the employment service, which collaborate in monitoring activities, are the most important institutions in charge of monitoring. While the ministries generally bear overall responsibility for monitoring and evaluation, the employment service, through its regional and local employment agencies, carries out the daily business of monitoring; it is often the main provider of data and - as far as day-to-day activities are concerned - also the main user of data. Because monitoring is related especially closely to programme implementation, it is the employment services that are generally a very important actor in the field of monitoring. There is, however, national variation in this respect (e.g. France, where the ministry is important). Initiatives to change, introduce and discontinue programmes, which usually involve political (not just administrative) decisions, mainly occur at ministerial level, usually in close collaboration with the employment services. In many countries, additional actors (especially in the field of training) also carry out monitoring activities.

Coordination among these institutions: While having many different actors in charge of monitoring increases the potential for conflicts of interest in monitoring activities, possibly leading to sub-optimal overall monitoring capacities, this must not always be the case; monitoring activities might equally lead to increased collaboration between the actors. While evidence of both cases may be found in the countries considered, a vague North-South pattern does emerge, demonstrating that, among other factors, monitoring activities have been introduced relatively recently in some of the southern EU countries, and the division of labour between different actors might not as yet be fully established.

Decentralisation/deconcentration: This North-South divide among the countries appears much clearer with regard to decentralisation/deconcentration. While many of the northern countries have already achieved a fairly high degree of decentralisation (deconcentration) in the delivery of labour market policy, some of the southern countries still have more centralised delivery and/or less autonomous local offices. We have noted above that monitoring and decentralisation are closely linked and that the development of monitoring is usually more advanced in countries with decentralised programme implementation. The question of decentralisation is also linked to the next indicator.

Local budgetary discretion: in such decentralised settings as Sweden, local employment agents have significant autonomy in the allocation of funds between different measures. This discretion is much less important in many of the other countries and£s negligible in some of the southern member countries. It appeared in the interviews that "clientelism" (i.e. serving the needs of some important local actors without always looking for the public interest and equity in delivery) sometimes prevents far-reaching decentralisation, particularly in the South.

Incentives/sanctions: Although one might assume that decentralisation, budgetary discretion and autonomy of administration are likely to be accompanied by incentives for good results, this is in fact seldom the case. Both the Finnish and the Swedish delivery system, where decentralisation goes together with MbR (management by results) have some incentives, but they do not play a large role in results. Some incentives also exist in the UK delivery system through TECs. The only other country having some incentives (for directors of employment offices) is France. In the other countries incentives are not given. Sanctions, on the other hand, exist - with the exception of the UK - in none of the EU countries. It was reported that most monitoring systems are not yet sufficiently highly developed to deliver results which would allow an exact measurement of indicators, which could then be used as objective criteria for sanctions, and also because of the lack of control variables to account for differences in local labour market situations. Therefore a genuine "bonus-malus" system does not exist in the countries observed.

Table 1: European Overview an Monitoring

Organisation Continuous information on *: Monitoring

Country Monitoring Coordi- Decentra- Local Incen- Sanctions Budget Partici- Post-participation Training Employ- Level of Trend Observations

institutions nation lisation budget tives pation pro- ment develop-

among (Decon- discretion gramme** pro- ment

institu- centra- motion**

tions tion)

Employ- Qualifica- Earnings

ment tion

Belgium MoL, ONEM, VDAB, + +++ + - - n.a. n.a. n.a. n.a. n.a. n.a. n.a. + Regional particularism hinders

FOREM, ORBEM overall monitoring, monitoring

activities regional

Denmark n.a.

Germany BA, IAB +++ ++ ++ - - ++ ++ ++ - - ++ + ++ Strong system development; not all

mesures monitored

Greece MoL, OAED, PIEKA + + + - - ++ + - - - ++ + + Development of regional

employment observatory

Spain MoL, INEM, ++ + + - - ++ + + - - ++ + +

FORCEM

Finland MoL +++ ++ ++ - - +++ +++ ++ - - ++ ++ +++ High stage of development (MbR)

France MoL, ANPE ++ ++ ++ + - ++ ++ + - - ++ ++ ++ Not all measures monitored; both

ANPE and MoL are actively

developing the system

Ireland DEE, DSW, FAS ++ ++ + - - ++ ++ ++ + + ++ ++ ++

Italy n.a.

Luxemburg n.a.

Netherlands LBA, CBA, RBA ++ +++ ++ - - ++ ++ + + - ++ ++ ++

Austria MoL, AMS ++ ++ + - - ++ ++ + - - ++ ++ ++ New AMS is developing

monitoring

Portugal MoL, IEFP + + + - - ++ ++ ++ + - ++ + + Development of regional

employment observatory

Sweden MoL, AMS +++ +++ +++ + - +++ +++ ++ + - ++ ++ +++ High stage of development

(MbR)

United DEE, ES, IC ++ ++ ++ + (+) +++ +++ +++ ++ + ++ ++ ++

Kingdom

* at least every six months + = low ++ = medium +++ = high - negligible n.a. = information not available

** for training programme or employment promotion ++ = thoroughly monitored + = less thoroughly monitored

Source: Questionnaire, selected expert interviews and related documents, own classification; UK: provided by MISEP correspondent directly.

9.1.3. Input/output measurement

As far as the continuous observation of selected indicators is concerned, most respondents indicated that information on budgets and on participants (level and structure) is available, but sometimes only for certain programmes. However, the most important problem in this area is the absence of "compatible" budget and participant data as a first step towards a comparative cost-benefit analysis across employment measures: such information is required for effective monitoring. Also the timing differs: in some countries (Sweden, Finland) most information is in principle on-line, in others only monthly and/or quarterly data exist. All countries have at least annual reports, but they are of limited value for monitoring purposes as annual data usually arrive too late to change the conditions of unsatisfactory programmes early on. No consistent picture for timing preferences across countries emerges, although the trend is clear: the more monitoring is developed, the shorter the time interval to which monitoring refers. Timing of information is a crucial issue in general, but especially for budget data: usually such data is not continuous, but of ad hoc nature. This sort of budget information flow has major drawbacks for efficient programme administration. On the one hand, it often happens that all of a sudden administrators become aware a budgetary ceiling has been reached, and a sudden stop in programme delivery is required. On the other hand, there are cases where the budget is not fully utilised and money has then to be spent urgently. Thus for example the usual end-of-the-year run starts: money "has" to be spent on programmes that under such circumstances often do not deliver the best results nor reach target groups. This general problem of annual budgeting, which is also an argument for longer budgeting periods, could to a large extent be eliminated through continuous monitoring of the budgets. Thus in Sweden, since the inception of the monitoring system money allocated has usually been spent according to plan.

While consistent, comparative budget and participation data are the base-line indicators for monitoring input and output, outcome variables clearly have to be considered in order to assess policies adequately.

9.1.4. Outcome measurement

The follow-up of employment status, of qualification attained and of earnings after participation in various policy measures is performed differently in the countries observed.

Employment status is an indicator frequently analysed in many countries, although the timing (i.e. after three, six or twelve months) and the extent (for all measures or for selected measures) vary quite substantially across countries. Also the ways in which this follow-up is carried out varies: it is usually either done by postal survey or by administrative means (participants reporting to the employment office).

Qualification levels attained after training measures are not frequently assessed. Patchy information is available in some countries (see table), but in general this is not an important monitoring indicator. One reason seems to be that training courses often do not lead to certified formal qualification; where this is the specific aim of the course, this information is in general available (e.g. in the UK's present effort to establish a National Vocational Qualification Certificate). This makes it difficult to classify participants according to qualification levels attained. Also in training courses post-participant employment is the main indicator for success.

Unlike the US (and other English-speaking countries), where increased earnings are a major indicator of success, European monitoring and evaluation seldom analyse earnings after participation in an employment measure as an indicator of success. Ireland is an exception in that regard as it runs sample surveys six and twelve months after completion of measures, which are published yearly. In Sweden such earning follow-ups are done infrequently (one was conducted at the end of 1995) and are in any case not tied into administrative monitoring practices.

It seems that training programmes are more thoroughly monitored than employment promotion (job creation) programmes (see table). One reason for this might be simply that training programmes play a more important role than employment programmes in almost all countries.

In conclusion, our overview of the state of the art of monitoring and the monitoring trends in the EU countries shows a rather clear North-South divide. While it is true that most of the countries have as yet not developed fully fledged monitoring systems, all of them have established parts of such systems. But it is the more developed countries of the North and the Centre with a long tradition of employment policy delivery that are more advanced in the development of their monitoring systems. We have noted that Sweden and also Finland have rather sophisticated systems. Ireland, the UK, Germany, France, Austria and the Netherlands (and also Denmark, which is not included here) have monitoring experience for some of their policies and, moreover, are intensively developing these systems. Because of its regional particularism, Belgium is an exception: while there is little evidence of overall national monitoring, attempts to introduce monitoring on the different regional levels have been made. Greece and Portugal (and also Spain and, to our knowledge, Italy) have for different reasons not yet gained significant monitoring experience. Although their activities in the field are increasing, it seems that they are not currently "catching up" with the northern countries. Therefore mechanisms to exchange "best-practices" are necessary to reach a more homogeneous state of development in monitoring throughout the European Union.

EU countries clearly experience problems in setting up monitoring systems. Although certain problems are common to all countries, some also face complications relating to their particular administrative structure and other specific factors. Besides problems of coordination between different agencies in charge of monitoring, it is the lack of clear goals in employment policy measures that often makes monitoring difficult. Here the often contradictory relationship between short-term quantitative goals (the prime interest of politicians especially prior to elections) and long-term qualitative goals (whose fulfilment is often targeted by those directly in charge of the participants) plays a role. Thus, for example, centrally set quantitative volume goals create problems even in the traditionally more bottom-up decision-making channels of Swedish labour market policy. A more technical problem reported was also the lack of adequate data (or difficult access to such data) from which to construct indicators. Also the lack of personnel and financial resources was seen as a problem, as was the acceptance of monitoring by those whose work was monitored because of the element of control involved. Thus it is important to show that monitoring is not an instrument of control, but a way of enhancing the efficient working of the organisation.

9.1.5. A monitoring model

A monitoring system should consist of:

1. input, output and outcome goals specified by the political and administrative authorities at national, regional and local levels in a bottom-up procedure;

2. a definition of indicators that allow goals to be measured;

3. the actual monitoring process based on 1 and 2 and on statistical information on financial and physical indicators relating to 1 and 2; and

4. feedback loops to ensure that observed irregularities are addressed adequately.

Figure 1 illustrates such a monitoring model. It starts with policy formation in which programmes and individual measures are chosen. Even at this early stage clear goals for the programmes must be defined, and indicators enabling progress toward those goals to be measured must be established if monitoring is to be effective. In other words effective monitoring must be written into laws and ordinances (as is the case in the French five-year employment law of 1994). After programmes have been implemented, a continuous process of observation should begin, the intervals of which may vary from measure to measure. A quarterly observation period seems a reasonable compromise, but for short-term programmes shorter periods might be appropriate. The core function of monitoring is to detect indicators turning "red" and to initiate subsequent remedial action (feedback). Feedback between monitoring and programme implementation is stronger than between monitoring and policy formation because of the role of monitoring as an instrument linked to programme implementation. However, the results of monitoring in which the standard performance of programmes is measured should also be fed back to policy formation. Evaluation, in which dead-weight, substitution effects and other factors are also taken into consideration, offers a more profound assessment of programme impacts than monitoring does and thus affects policy formation more strongly than does monitoring.

However, monitoring should leave to evaluation only those indicators that cannot be observed directly and regularly. A relatively simple monitoring system will allow for a national/regional/local breakdown in order to facilitate comparisons between agencies. For any given programme at any given administrative level such a system should at least be based on the regular (monthly, quarterly) observation of:

1. the budget allocated and the resources spent (breakdown by spending categories, target groups and target sectors if appropriate);

2. planned participation and actual participation (breakdown by target group and target sectors if appropriate);

3. costs per head (per hour) on the basis of 1 and 2 and breakdown by spending categories, target group/sectors if appropriate.

Thus, monitoring systems would provide useful information on financial and physical take-up and costs, and under-performers could be detected by comparing regional/local units, although the differences in labour market conditions in local areas should be taken into account.

The indicators listed above allow the close monitoring of fund outflows and participation in relation to pre-set goals and permit comparisons of different constituencies and even of per capita costs between different measures. The indicators are therefore of greatest value to a programme administrator. They are basically input-output based and must, therefore, be supplemented by outcome-performance measures such as the employment status of the participants after participation or the skill levels achieved in training programmes (see Affholter, 1994).

Figure 1 also shows a link between labour market monitoring and labour market policy monitoring: results of LMP have an impact on the labour market and an ideal monitoring model must establish this link.

9.1.6. Conclusion

This article has drawn attention to some of the problems encountered in the current process of setting up LMP monitoring systems in the Member States of the EU, outlined the reasons why the monitoring of LMP performance is becoming more important, presented an overview of the state of the art of labour market policy monitoring in EU Member States and sketched out the functions of a monitoring system. While some countries have been engaged in monitoring for longer than others, most Member States are still in the process of installing such systems. Thus, the object of research still resembles a "moving target" and is difficult to grasp empirically. Some main trends have emerged, however. Monitoring should be tied to the routine administrative process of programme implementation and permit the combined observation of financial and physical indicators of LMP. In addition, monitoring should involve the continuous assessment of results. Ultimately, input (expenditure), output (participants) and outcome (performance) indicators are all essential elements of such monitoring systems and must be combined within the monitoring system.

It is surprising that administrative monitoring was not introduced earlier; but it seems that the spending of public money was not previously subject to such constraints. In two of the surveyed countries, it was precisely a lack of financial monitoring that led to problems with programme delivery. Consequently, one of the basic aims of monitoring systems is to allow a steady and controlled disbursement of assigned budgetary funds. Providing information on where public money goes and how the money can be allocated most efficiently for reaching pre-defined goals is, of course, a basic purpose of a monitoring system. The combined observation of money allocated/money spent and planned/actual participation by delivery area/sector and so forth is the core feature of any system for monitoring active labour market policy. These rather simple input/output data can be supplemented with results indicators to produce a comprehensive system of LMP observation that would allow agents at all levels to follow up LMP measures.

The fact that aspects of financial constraint currently seem to be the main focus of attention inevitably gives rise to the concern that the interest in efficient public spending will eventually mean that programmes will be evaluated primarily according to their immediate success (e.g. low per capita spending) and that this will be at the expense of longer-term and qualitative goals.

Experience with monitoring systems shows that monitoring of pre-set budgetary and participation goals does make it possible to improve the convergence of planning and actual outcomes. Moreover, it seems that if goals are set from the bottom up and if local agencies have a say in their establishment, the chances of achieving them are enhanced.

Although monitoring is in fact a "neutral" instrument, it seems, therefore, that the participation of those involved in monitoring in the setting of goals is important for the efficiency of the instrument. In the light of the experience with employment programmes in the United States, it seems that programme performance is enhanced if there is a built-in monitoring function. As formulated by the DOL (1994):

"Perhaps the most important lesson from our experience with using a performance driven management system is that local programs respond remarkably well to the required performance indicators. Once performance standards were implemented, employment rates and wage levels for individuals leaving the programme rose and continued to increase each year" (p. 1).

Thus, building in performance indicators in programmes is clearly a step in the right direction.

Monitoring implies the availability of indicators pertaining to all regional and local delivery areas and, hence, implies comparisons between them. It could thus enhance effective programme delivery. There is, however, the problem of pressure to harmonize LMP programme performance in areas that differ in their points of departure (in terms of target-group shares or the situation of the local economy, for example). That is, local adjustment of performance indicators is essential (see also Barnow, 1995). Sensitive data, for example, per capita cost, must be adjusted to take account of local variations.

In conclusion, clear-cut goals, clear financial and physical indicators of performance, concise statistics, appropriate time intervals of observation, feedback to guide the lowest level of the delivery organization, feedback to amend programmes in case of non-performance, incentives for good performers and increased local freedom to manage delivery would all form part of an optimal "package deal" and would pave the way to the efficient monitoring of LMP. Finally, a very important aspect of monitoring is acceptance: only a dialogue between the observers and the observed, especially when it comes to defining the goals to be set and monitored, is likely to bring about satisfactory results.

9.1.7. Selected Bibliography

Affholter, P. D. (1994): "Outcome Monitoring." In: J. S. Wholey, H. P. Hatry & K. E. Newcomer (eds.): Handbook of Practical Program Evaluation. San Francisco, pp. 96-118

Auer, P. (1994): "Further Education and Training for the Employed: Systems and Outcomes." In: G. Schmid (eds.): Labor Market Institutions in Europe. Berlin

Barkema, H. G. (1995): "Do Top Managers Work Harder When They Are Monitored?" In: KYKLOS, 48, pp. 19-42

Barnow, B. S. (1995): Performance Management and Programme Impact in the Job Training Partnership Act: Exploring the Relationship between Them. Interim report. John Hopkins University. Baltimore, MD

Bellmann, L. & U. Walwei (1994): Public Employment Services and Labour Market Information. Labour Administration Branch Document 40-3. Geneva: ILO

Blaschke, D., H. E. Plath & E. Nagel (1992): Konzepte und Probleme der Evaluation aktiver Arbeitsmarktpolitik am Beispiel Fortbildung und Umschulung. MittAB 3/92. Nuremberg: IAB

Chastand, A. (1994): "Statistiques et Politiques de l'Emploi." In: Courrier des Statistiques, 70

Courty, P. & G. R. Marschke (1995): Moral Hazard under Incentive Systems: The Case of a Federal Bureaucracy. Paper for "Business/Academic Dialogue, Reinventing Government and the Problem of Bureaucracy: Implication for Regulation and Reform", June 1995, University of Chicago

DEE (Department of Enterprise and Employment) (1994): Evaluation Report: Recording and Reporting Systems. Dublin: ESF Evaluation Unit

DOL (US Department of Labor) (1994): "Performance Standards: Lessons Learned." Washington, DC (unpublished manuscript)

European Commission (Hg.) (1994): Growth, Competitiveness and Employment. White paper. Brussels

Fama, E. F. & M. C. Jensen (1983): "Separation of Ownership and Control." In: Journal of Law and Economics, 26, pp. 301-351

Gauti�, J., B. Gazier & R. Silvera (1994): Les Subventions � l'Emploi: Analyses et Exp�riences Europ�ennes. Paris: La Documentation Fran�aise

Hasan, A. (1991): "Evaluation of Employment Training and Social Programmes: An Overview of Issues." In: OECD (ed.): Evaluating Labour Market and Social Programmes - The State of a Complex Art. Paris: OECD, pp. 21-42

Höcker, H. (1994): "The Organisation of Labour Market Policy Delivery in the European Union." In: inforMISEP Policies, 48, pp. 26-35

Kuhlmann, S. & D. Holland (1995): Evaluation von Technologiepolitik in Deutschland. Heidelberg: Physica

Melchior, A. (1992): "Performance Standards and Performance Management." In: US Department of Labor (ed.): Dilemmas in Youth Employment Programming. Findings from the Youth Research and Technical Assistance Project. Research and Evaluation Report Series 92c, Vol. 2. Washington, DC: Department of Labor, pp. 77-142

Meyer-Krahmer, F. (1990): "Evaluation der Wirksamkeit von Instrumenten der Forschungs- und Technologiepolitik." In: H. Krupp (ed.): Technikpolitik angesichts der Umweltkatastrophe. Heidelberg: Physica, pp. 210-214

Mikkonen, I. & H. Räisänen, H. (eds.) (1995): Evaluating Labor Market Training - Outcome and Effectiveness. Labor Policy Studies 122. Helsinki: Ministry of Labor

MISEP (Mutual Information System on Employment Policies) (1989): Rapport Final du Groupe de Travail sur les Evaluations des Mesures de Politique d'Emploi. European Centre for Work and Society. Maastricht, NL: European Center

OECD (1991/1994): Employment Outlook. Paris: OECD

Räisänen, H. (1994): What Kind of Policy Instrument is Labor Market Training? Helsinki: Ministry of Labor

Schmid, G. (1980): Strukturierte Arbeitslosigkeit und Arbeitsmarktpolitik. Königstein: Athenäum

Schmid, G. (1994): Reorganisation der Arbeitsmarktpolitik. Märkte, politische Steuerung und Netzwerke der Weiterbildung für Arbeitslose in der Europäischen Union. WZB discussion paper FS I 94-213. Berlin: Wissenschaftszentrum Berlin für Sozialforschung

Schmid, G., J. O'Reilly & K. Schömann (eds.) (1996): International Handbook of Labour Market Policy and Evaluation. Aldershot: Edward Elgar

9.1.8. Figure

EU � European Commission DG EMPL/A/2 J II 27,
Rue de la Loi 200, B-1049 Brussels � Belgium
GHK Consulting Ltd
30 St. Paul's Square, Birmingham. B3 1QZ
E-mail: eeo@ghkint.com