ISSN 0798 1015

Vol. 40 (Issue 38) Year 2019. Page 5

Research in evaluation of target program model in Russian metropolises

Investigación sobre la evaluación del modelo de programa objetivo en las metrópolis rusas

ELOKHOV, Alexander M. 1; ARBUZOVA, Tatiana A. 2 & ARBUZOV, Viacheslav O. 3

Received: 26/06/2019 • Approved: 28/10/2019 • Published 04/11/2019

Contents

1. Introduction

2. Materials and methodology

3. Results and discussion

4. Conclusions

Bibliographical references

1. Introduction

According to guidelines, cities with the population of more than one million people belong to metropolis group (“Set of Rules”, n.d.). The number of cities and their population is growing in Russia. From 1926 to 2018 the number of cities and towns has grown by 2.4 times (from 461 to 1112), the share of urban residents in the total country’s population size has increased from 18% to 70% for the same period  (Population of the Russian Federation by Municipalities, n.d.).

A growing role of metropolises in Russian social and economic development is largely defined by the efficiency of city strategic planning where target programs have become one of the main instruments. There are several reasons for an active use of programs. First, target programs provide a concentration of efforts and resources on key ways of the city’s social and economic development. Secondly, programs allow city residents to control, and businesses to participate in solving the strategic tasks of the city’s sustainable social and economic development. Thirdly, programs are an instrument of the city’s budget formation and implementation

The calculation, made on the basis of data taken from municipal administration sites, shows that in Russian metropolises more than 240 target programs are developed and implemented. The composition of programs, their number and program budget volume in Russian metropolises are approximately the same. Table 1 demonstrates the dynamics of target programs number and program budget volumes typical for all Russian metropolises by means of the example of the metropolis Perm, which is one of the biggest Russian cities with the population of more than one million people (Municipal Programs, n.d.).

Table 1
Dynamics of programs number and
program budget volume in Perm

Source: created by the authors based on (Municipal Programs, n.d.).

The development of the programing system in urban development leads to an increase in the interest of target programs evaluation.  

The analysis of metropolises’ regulatory documents shows that the target programs evaluation is based mainly on such intermediate results of their implementation as achievement indicators of program goals, implementing of program measures, program finance development.  These evaluation types do not allow defining the final result of program measures implementation: cost-effectiveness in achieving quantitative parameters, changing indicators of the city’s social and economic development under the influence of target programs. The problem of programs correspondence to the strategy of city’s social and economic development remains actual, because these programs are proclaimed to be the implementation instruments of the strategy by the Federal Law №172-FZ  (“Federal law №172-FZ”, 2014). The current situation demands to increase the efficiency of city programs evaluation model. 

The purpose of the present research is to improve programs evaluation model by introducing such new indicators as program’s «accordance» with city development strategies, «efficiency» of program measures, and their influence on strategic targeted indicators of the city’s development.

The objectives of the research are:

1) to consider theoretical approaches to program evaluation  in domestic and foreign researches;

2) to analyze the compliance of programs evaluation models in Russian metropolises  with the requirements of theoretical approaches;

 3) to supplement the indicators of programs evaluation model;

4) to develop algorithms for evaluating target programs by using new indicators.

2. Materials and methodology

The research methodology is based on comparative approach, which used general scientific methods and principles, the ones of logical, systematic and statistical analysis of metropolises’ guidelines on programs efficiency evaluation.

The theoretical foundation of evaluation model is laid in the works of Weiss, 1972; Franklin & Thrasher, 1976; Scriven, 1980; Mishan, 1971;  Patton, 1997. Later, there appeared more noticeable research in the evaluation sphere, due to Western scholars formulated basic principles of target programs evaluation (Dolan & Edlin, 2002;  Vanclay, 2003;  Hatry, 2005;Peacock, 1998;  Miller, Hildreth, & Rabin, 2010).

1. The target program effect must be the consequence of its implementation. According to Hatry (2005) the true indicator of program efforts usefulness is not a manufactured product but the received result (effect). The results are events or changes in condition, proving the movement to program target. Thus, the results are connected to the main function of a program, i.e. what it was made for. The result is not what a program itself «made», but the consequences of its activity (Hatry, 2005). F. Vaclay identifies the program’s social impact, which includes positive or negative social and cultural changes in public condition (Vanclay, 2003).

2. The target program effect must provide the increase in standards of living not for some citizens, but the welfare growth of a territorial community as a whole. A. Williams & E. Giardina (1993) single out social benefit and externality as parts of programs’ social effect. They understand social benefit as an integrated utility from different activities, including the benefit of the subject, implementing the program, as well as the people not paying for it. The externality means additional savings or expenditure, which do not depend on program’s actions but influence on their results.

Table 2 demonstrates the theoretical approaches of Russian specialists where the composition of indicators for target programs evaluating are determined.

Table 2
Indicators of program evaluation in studies
of Russian specialists from 2001 to 2018

Source: created by the authors

The data in Table 2 confirms the fact that Russian specialists have different views about indicators of program evaluation. It should be noted the shift in  composition of program evaluation indicators towards necessity to evaluate the  programs impact on city’s social and economic development.

The authors of the article share the position of the Institute for Urban Economics specialists who believe, that indicators for target programs evaluation should  be aimed to promoting efficient social and economic urban development  (Vetrov, et al., 2003; Vizgalov, 2005; Elokhov & Arbuzova, 2018).

Table 3 demonstrates the indicators for evaluating target programs in Russian metropolises. The composition of indicators was determined based on the study of laws and regulations on municipal programs development and implementation in metropolises of the Russian Federation. The data analysis resulted in the following conclusions.

First, in Russian metropolises program evaluation models have different structure, including from 3 to 6 elements. Evaluation of «internal» program elements prevail in the model, reflecting the subject and specific features of their implementation: volume of financing, achievement of program purposes and tasks, implementation of program measures. Only Ufa and Ekaterinburg evaluate programs’ accordance to strategic planning documents based on the indicator of programs’ accordance with the city development strategy. Ufa, Ekaterinburg and Nizhny Novgorod have the fullest evaluation model (5-6 indicators).

Secondly, only documents from Perm and Volgograd demonstrate the evaluation of results from programs implementation, manifested in changing of the city’s social and economic development indicators.

Other cities evaluate intermediate results. However, H. Hatry’s research shows that the main principle in defining program results consists in searching the elements, directly relating not only to satisfaction of program’s clients, but to the population as a whole and are important to them  (Hatry, 2005).

Thirdly, there are different approaches to evaluate programs’ efficiency. Efficiency is defined along the whole program based on calculating an average indicator and integral indicator, uniting several measured indicators, reflecting the achievement of program’s goals and its financing, as a rule. There is no evaluation of program measures’ efficiency.

There are different methods to measure programs’ integral efficiency indicators: Novosibirsk, Ekaterinburg and Perm calculate point total, Ufa and Krasnoyarsk measure shares from one, Nizhny Novgorod, Kazan, Samara, Voronezh, Volgograd, Rostov-on-Don and Omsk calculate percentage. The variety of methods does not allow to compare the results of evaluation the effectiveness of similar target programs in different cities. Efficiency calculations do not use generally accepted ratio of program’s effect to its costs, types of effects are reduced only to achievement of target programs’ indicators.  However, the true utility indicator of program efforts consists not in a produced product, but in a received result (effect). Results are events or changes in a state, proving the movement towards program goals (Hatry, 2005). M. Garadzhа suggests to calculate target programs’ effect based on the indicators of result achievement in relation to purposes  (Garadzha, 2002).

Fourth, only program documents in Kazan and Volgograd state the need to evaluate programs’ effects on the city’s social and economic development. Other cities evaluate the impact of «internal» results only without assessing their influence on the development of a corresponding territory that is «external» effects. Meanwhile the most important principle of programs implementation consists in increasing the standards of living for the whole population of the territory, not for certain categories of citizens only. A. Williams and E. Giardin (1993) recommend singling out social benefit and external effect when evaluating programs. They understand social benefit as an integrated utility from different activities, including the benefit for subjects, implementing the program, as well as for those who do not pay for it. External effect (externality) means additional savings or expenditure, which do not depend on program actions but influence on their results. F. Vaclay (2003) tells that program’s social effect includes positive or negative social and cultural changes in social status.

According to the results of target programs evaluation models analysis in Russian metropolises, the authors placed two hypotheses:

1. Current models of programs’ evaluation do not reflect their influence on indicators of the city’s social and economic development.

2.  In order to increase the objectivity of target programs and validity of decisions in the field of program-oriented management, it is needed to complement evaluation of target programs with indicators reflecting relationship between programs and  indicators of the city’s social and economic development.

3. Results and discussion

The research showed that program evaluation models in Russian metropolises should be brought into the line with the current world practice by adding some indicators. An extended program evaluation model in a metropolis is presented in Table 4.

 Table 4
Extended program evaluation
model in a metropolis

Source: created by the authors

In order to improve existing program evaluation models in Russian metropolises it is necessary to consider some rules:

1. During evaluating of programs’ basic elements, it is needed to calculate the ratio of actual and planned values of indicators in terms of funding volume, number of program measures, and program goals. Full implementation of planned indicators will determine the maximum value of evaluation, equal to one.

2. Evaluation of programs «accordance» with city development strategies should be based on checking the coincidence of their elements. For example, by the ratio of the target programs’ number  and the number of strategies’ tasks, task accordance between the programs and the strategy. Full accordance of these elements will determine the value of evaluation, equal to one.

3. Evaluation of program measures efficiency should be based on the ratio of  implementation costs put on program measure and the magnitude of change (increase or decrease) in the programs targeted indicator after measure implementation. The planned ratio of indicators is taken as a unit, the actual ratio is defined as an increase or decrease in the planned level of efficiency (more or less than one).

4. Evaluation of programs’ influence on strategic targeted indicators of city’s development  should be based on changing the number of city’s resident population during program implementation. The calculation is carried out according to the program share in the total funding volume, which ensures the achievement of targeted strategic indicator of the city’s development. Evaluation of programs’ influence is defined as a fraction of a unit.

5. The overall program evaluation is defined as the sum of the six indicators from the Table 4. Based on the results of evaluation management decisions about program can be made. Table 5 demonstrates the criteria for overall program evaluation.

Table 5
Criteria for overall program evaluation

Source: created by the authors

We will look at the necessity and possibility of using new program evaluation indicators by means of the example of the metropolis Perm, which occupies a middle position among Russian metropolises.

The indicator of programs’ «accordance» with city development strategies

We will examine to what extent target programs, implemented in the city, reflect «The strategy of Perm’s social and economic development to the year 2030». The subject of research includes the existence of target programs, implementing the city development strategy; task accordance between the programs and the strategy. Table 6 demonstrates that 19 target programs consisting of 124 tasks are implemented to fulfill 59 strategic tasks (The strategy of socio-economic development of Perm city to 2030, n.d.;  Municipal programs, n.d.).

Table 6
Task accordance between the programs
and the strategy of Perm in 2019

Source: created by the authors based on (The Strategy of Socio-Economic
Development of Perm City to 2030, n.d.; Municipal Programs, n.d.).

Having analyzed table 4, we came to the following conclusions:

1. The number of municipal target programs is not proportional to the Strategy’s functional and purpose blocks (FPB), for example, FPB on social development aspects contains 18 tasks and 7 target programs, while the one on urban economic development includes only 7 tasks and 1 program.  The share of strategy tasks implemented in all target programs is only 32%.

This, in our opinion, decreases resource and effort concentration to solve key strategic tasks and causes ambiguity in strategic planning.

2. 63% of key FPB tasks are reflected in the accepted target programs. The closest link is between strategic and program tasks in FPB of social safety and economic development, while the weakest link is in FPB of municipal governance.

Thus, we can conclude that target programs correspond to the city’s strategy tasks only by (32 + 63): 2 = 47.5%.

The indicator of program measures «efficiency»

In Table 7 we will show the possibility of such calculations by the means of the example of one the target programs «Crime prevention in Perm for the period of 2016 ‒2018» (By-law of the Perm city administration № 850, 2015). This program is typical for all Russian metropolises.

Table 7
Calculation of the efficiency of program
measures from 2016 to 2018

Source: created by the authors based on
(By-Law of the Perm City Administration № 850, 2015.)

Having analyzed table 6, we came to the following conclusions:

The program is equally financed from municipal budget by 6,5 million rubles per year. As a result of the annually implemented standard set of program measures, the target program indicator “Crime rate per 10,000 people” decreased from 28.3 cases in 2016 to 5.7 cases in 2018. Thus, the expenses per reduction unit of the program indicator increased by 4.69 times between 2016 and 2018, whereas the program measures efficiency as the indicator inverse to expenses, decreased by 5 times. 

The indicator of programs’ «influence» on strategic targeted indicators of the city’s development

The definition of this indicator, is based on the idea, that the implementation of the city’s development strategy should provide the growth of the city’s attractiveness as a place convenient for life and labor. The growth of residential population is suggested to use as a strategic target indicator, reflecting the success of the city’s development strategy implementation. If, as a result of the strategy implementation, a city has high standards of living, residential population increases due to migration and natural growth. Target programs act as a mechanism for implementing the city’s development strategy. Program measures ensure a comfort conditions for a city life and contribute to population growth (Elokhov & Elokhova, 2015).

Table 8 represents the evaluation of programs impact on the growth of the resident population of Perm in 2019.

Table 8
Analysis of target programs impact on the
residential population growth in Perm in 2019

Source: created by the authors based on (By-Law of the Perm City Administration №1071, 2017; 
Decision of the Perm City Duma № 250; 2017)

In Table 7 the impact of target programs on the residential population growth is determined in proportion to the program share in the program budget volume.

The largest impact on the residential population growth is provided by the target program “Ensuring access to quality education services” (39.82%), and the least by the target program “City land management” (0.07%). It is obviously, that a number of programs have little effect on the residential population growth, they do not create the appropriate conditions for improving the quality of the city’s life. That is why, it is necessary to review the composition of programs and the volume of their budget. Carrying out such calculations helps to increase the objectivity of programs evaluating.

The data in tables 5,6 and 7 shows that there are programs in the city of Perm that do not significantly affect the achievement of strategic parameters of the city’s development.

4. Conclusions

In this study, a comparative analysis of the evaluation model of target programs in Russian metropolises, was carried out. It was found, that in metropolises there are similar models for evaluating target programs. For evaluation, a comparison of planned and actual indicators reflecting the implementation of program objectives, implementation of program measures, and development of budget volume.

Evaluation of target programs according to the current model does not allow a complete and objective assessment of program activities in the city. Given the similarity of models for evaluating target programs in Russian metropolises, we show the reliability of this statement by means of the example of one of the Russian metropolises, Perm city.

Thus, the head of the city, using the current methodology for evaluating the program’s effectiveness, declared low-efficient only 1 programs out of 25 as the result of the year 2018. That means he evaluated program activity positively  (“Consolidated annual report”, n.d.).

At the same time, the current target programs evaluation model in Perm cannot be considered as complete and comprehensive. The data in table 6 demonstrates that target programs correspond to the city’s strategy tasks only by 47.5%. Table 7 demonstrates that the efficiency of the typical for all Russian metropolises target program «Crime prevention in Perm for the period of 2016 ‒2018» decreased by 5.0 times. Table 8 demonstrates that 11 out of 19 target programs, operating in 2019, affect the residential population growth in the range from 0.1 to 1.4%, which means they do not create the appropriate conditions for improving the quality of city’s life.

These indicators signal about the increase of imbalances in target programs evaluation according to current evaluation model, confirming the adequacy of the first formulated hypothesis.

In order to increase the objectivity of current target programs evaluation model, it is recommended to supplement it with such indicators, as program’s «accordance» with city development strategies, reflecting the degree of program compliance with the city’s development strategy; «efficiency» of program measures, reflecting the efficiency of program measures for their impact on changing program targets;  and «influence» on strategic targeted indicators of the city’s development, reflecting  programs impact on  final results of the city’s development.

The expansion of the composition of program’s evaluation indicators is consistent with evaluation theory, regulatory demands of the federal law 172-FZ «On strategic planning in the Russian Federation» (2014).

The research results prove that additions to programs evaluation model will contribute to an increase in the strategic planning efficiency and program management of the city development. This will also help to work out a more efficient system for managing target programs.

Thus, it is necessary to bring the share of key tasks of the city’s development strategy functional and purpose blocks implemented through programs to 100% and optimize the composition of target programs by strategy’s sections. This, in our opinion, will strengthen the concentration of funds and efforts, spent on target programs, to solve the key strategic tasks of urban development.

Evaluation of program’s measures efficiency by means of the example of the   typical for Russian metropolises target program «Crime prevention in Perm for the period of 2016 ‒2018» showed the possibility of saving budget funds. It is proposed to reduce the financing of ineffective measures. This money could be spent on the measures, which have the most powerful influence on the crime rate. It is also useful to change program’s financing based on the achieved results. For example, if the decrease in the crime rate in 2017 reached 20.5% of the previous year’s indicator, the expenses should also be reduced to 20.5% from the planned ones.

The research results indicate that it is necessary and possible to supplement the existing provisions of the city’s programs development and implementation by assessing their compliance with strategic documents, calculating the efficiency of program measures, taking into account the programs impact on city’s economic and social development. In our opinion, this will contribute to improving the efficiency of urban development program management, will help to develop a more effective system of program measures, which ensure an increase in the quality of the city’s life in accordance with Presidential Decree № 204 «On national goals and strategic tasks of the development of the Russian Federation for the period until 2024»  (“Decree of the President of the Russian Federation”, 2018).

Bibliographical references

Afanasev, M.P., & Shash, N.N. (2013). Assessment methodologies of the state-funded program efficiency. Public Administration Issues, 3, 48-69.

Alabina, T.A., & Ershova, N.N. (2014). The evaluation of long-term target programs’ efficiency: theory and practice (case study Kemerovskaya oblast). Barnaul: Si-Press.

Alexandrova, A.L., Sadovskaya, Z.L., Strike, R. G., & Chagin, K.G. (2003). The efficiency of municipal social programs: possibility and necessity of evaluation. The example of program evaluation “From social benefit to salary” in Perm. Moscow: Fond IEG.    

By-law of the Perm city administration № 850. (2015). “On approval of the municipal program “Crime prevention in the city of Perm”. Docs.cntd.ru. Retrieved from: http://docs.cntd.ru/document/428680794

By-law of the Perm city administration №1071. (2017).  “On approval of the forecast of socio-economic development of Perm city for the long-term period until 2023”. Gorodperm.ru. Retrieved from: https://www.gorodperm.ru/actions/strategy/conception_development/dok/prognoz/

Consolidated annual report on the implementation and evaluation of the municipal programs effectiveness in the city of Perm at the end of 2018. (n.d.). Gorodperm.ru. Retrieved from: http://www.gorodperm.ru/actions/strategy/conception_development/monitorig/svod_doklad/

Dolan, P., & Edlin, R. (2002). Is it really possible to build a bridge between cost-benefit analysis and cost-effectiveness analysis? Journal of Health Economics, 21(5), 827-843.

Decision of the Perm City Duma from № 250. (2017).  “On the budget of Perm for 2018 and for the planning period of 2019 and 2020”.  Docs.cntd.ru. Retrieved from: http://docs.cntd.ru/document/446598166

Decree of the President of the Russian Federation № 204. (2018). “On national goals and strategic tasks of the development of the Russian Federation for the period until 2024”. Minenergo.gov.ru. Retrieved from https://minenergo.gov.ru/view-pdf/11246/84473

Elokhov, A.M., & Arbuzova, Т.А. (2018). Territorial Strategic management by objective. Perm: PSNRU.

Elokhov, A.M., & Elokhova, T.A. (2015). Strategic management by objective. Perm: ZUIEP.

Federal law №172-FZ. (2014). “On strategic planning in the Russian Federation”. Garant. Retrieved from http://base.garant.ru/70684666/

Franklin, J., & Thrasher, J. (1976) An Introduction to Program Evaluation.  New York:Wiley&Sons.

Garadzha, M. Yu. (2002). Monitoring the efficiency of budget spending: main introduction stages on municipal level. Moscow: Fond IEG.

Hatry, G. (2005). Performance measurement getting results. Moscow: Fond IEG.  

Ilichev, A.V. (2009). Basis of target programs efficiency and risk analysis: sources, formalization, implementation. Moscow: Nauchnyy Mir.

Korableva, A.A. (2017). A methodological framework for assessing the effectiveness of municipal programs. Regional Economics: Theory and Practice, 15(4), 691-705.

Kuzmin, A.A. (2001). Specific features of conducting certain evaluation stages of projects and programs. Retrieved from: http://www.ipen21.org/ipen/

Miller, G. I., Hildreth, W. B., & Rabin, J. (2010).  Performance-Based Budgeting. Colorado: Westview Press.

Minchenko, O.S. (2012). Approaches to the targeted programs assessment in the Russian practice. Management Issues, 2(2), 14-20.

Mishan, E. (1971). Cost-Benefit Analysis. London: George Allen and Unwin.

Municipal programs. (n.d.). Gorodperm.ru. Retrieved from: http://www.gorodperm.ru /actions/strategy     /conception_development/dok/Municipal_programs/

Nozhenko, D.Yu. (2016). Government programs as a tool for strategic management: an interregional analysis. Management Issues, 4(41), 8-5.

Patton, Q.  (1997). Utilization Focused Evaluation. London: Sage Publications.

Peacock, S. (1998). An evaluation of program budgeting and marginal analysis applied in South Australian hospitals. Melbourne Vic Australia: Centre for Health Program Evaluation.

Population of the Russian Federation by municipalities. (n.d.). Gks.ru. Retrieved from: http://www.gks.ru/wps/wcm/connect/rosstat_main/rosstat/ru/statistics /publications /catalog/afc8ea004d56a39ab251f2bafc3a6fce

Set of rules. Town planning. Planning and development of urban and rural settlements. (n.d.). Garant. Retrieved from: http://base.garant.ru/6180772/#friends

Scriven, М. (1980). The Logic of Evaluation. Inverness, CA: Edgepress.

Shakina, E.A. (2008). Multi-level evaluation of target programs as a method of managing regional and municipal budgets expenditures (Published Phd dissertation). St. Petersburg State University of Engineering and Economics, St. Petersburg, Russia.

The strategy of socio-economic development of Perm city to 2030. The decision of the Perm City Duma №85. (2014) Gorodperm.ru. Retrieved from: http://www.gorodperm.ru/actions/strategy/conception_development/dok/strategia

Vanclay, F. (2003). International Principles for Social Impact Assessment. Impact Assessment & Project Appraisal, 21(1), 5-11.

Vetrov G.Yu., Vizgalov, D.V., Pinegina, M.V. & Shevyrova, N.I. (2003). Evaluation of municipal programs. Moscow: Fond IEG.    

Vizgalov, D.V. (2005). Evaluation methods of municipal programs. Moscow: Fond IEG.    

Voronin, A.G. (2014). Municipal economy and management: problems of theory and practice. Moscow: Finansi I statistika.

Weiss, C. (1972). Evaluation research. Methods of assessing program effectiveness. Englewood Cliffs, NJ: Prentice Hall.

Williams, A., Giardina, E. (1993). Efficiency in the Public Sector: The theory and practice of cost – benefit analysis. London: Edward Edgar.

1. Candidate of economic sciences, associate professor, the Department of Management, Perm State National Research University, Russian Federation. E-mail: reader1201@rambler.ru

2. Senior lecturer, the Department of Management, Perm State National Research University, Russian Federation. E-mail: businesspyt@gmail.com

3. Candidate of economic sciences, associate professor, the Department of Information Systems and Mathematical Methods in Economics, Perm State National Research University, Russian Federation. E-mail: arbuzov1989@gmail.com