Project Objectives and Outcomes
Public or government projects are implemented to facilitate the achievement of specified societal goals. Effective project governance is required to ensure that the project conforms to the societal values, needs, and interests in the planning and implementation stages. According to Samset et al. (2016), public projects result from political processes that involve compromises between societal actors and the government. As such, managing the needs and priorities of the different stakeholders throughout the project cycle is important. To avoid the clashing or conflicting interests, the local administrative agencies, citizens, the media, and private contractors and consultants must fully participate in the definition, design, and execution phases of the public project cycle following the principles of transparency and altruism (Flyvbjerg 2014). Project governance is required to support a structured and efficient planning and execution of the public investment initiative.
In essence, project governance may follow any of the three widely used policy formulations: “the stick, the carrot, and the sermon” (Samset et al. 2016, p. 11). The stick denotes alternate regulatory measures, while the carrot implies the economic options available to the authorities. In contrast, the sermon connotes the information, i.e., advice or warnings against unethical practices (Samset et al. 2016). This model of project governance is illustrated in Figure 1 below.
Projects usually follow a logical sequence beginning with concept development and definition at the front-end stage of the project prior to operationalisation. Concept development is guided by an evaluation of the conditions and premises that inform a particular measure (Samset et al. 2016). Public projects often involve three fundamental steps, namely, definition, design, and execution. Project definition entails the clarification of the objectives at the “strategic, tactical, and operational” levels grounded in societal needs (Samset et al. 2016, p. 13). At the design phase, the strategies for attaining the objectives are defined based on expected impact or ambitions. It provides a means for project execution. The execution phase is where the project action items are implemented and progress monitored.
In the front-end phase, a consideration of the effects of the actions on the outputs is crucial. This requires the design of the outcomes/results in addition to project design. The former step generates a specification of the output values that informs front-end decisions regarding which concept/option to adopt from the start (Flyvbjerg 2014). Output development begins with no baseline data and is fine-tuned into clear, anticipated project results. Output design is related to the selection of the strategies for attaining the objectives/goals. It entails tradeoffs between anticipated gains and costs linked to each solution to determine the best option for attaining the goals/targets.
The means and goals/objectives are intertwined in public sector projects. At all levels, the goal/objective gives the rationale (why), while the means describe the ‘how’ of the project (Flyvbjerg 2014). The Why-How framework helps formulate objectives and respective strategies for achieving them. The Project Management Institute [PMI] (2013) defines five iterative steps in the project management process, namely, “initiation, planning, execution, controlling, and closing” (p. 6). Further, the front-end phase comprises initiation and planning, while the execution phase includes management processes – implementation, monitoring, and closing. The PMI (2013) also states that the initial project definition and design should involve the consideration of options to select the best concept. Subsequently, the selected concept is developed through iterative processes to give the final project that is implemented. Thus, iterative evaluation of the objectives and means is required in formulating/planning a public sector project.
The aforementioned considerations are also important in the implementation and monitoring stages. In this regard, ongoing consideration of the objectives or a change of targets is considered a government role. On the other hand, the development of means of execution will be a role of the project managers of the implementing institution or contractor (Samset et al. 2016). The execution step may involve changes to the objectives to mitigate the negative effects of the change, while maximising the benefits to people. Thus, ongoing evaluation of the project, including consideration of the end user’s perspective, during execution is required to increase the return on investment at the end.
Samset et al. (2016) give a fundamental logic of public projects from the identification of the societal need to the achievement of the desired effect. The actual needs of the citizens inform the development of a relevant project concept to address them. A relevant project is the one whose outcomes/deliverables are considered sustainable. As a result, the identification of the societal needs is the starting point of public project development. The formulation of objectives and goals/targets based on the identified needs is then done to guide the subsequent execution and monitoring of goal-oriented initiatives. This process is illustrated in Figure 2 below.
In public investment projects, after formulating the goals, the means or strategies for achieving them are selected. In this case, efficiency in the translation of resources into outputs and the utilisation of the outputs to attain a sustainable effect on the population is of utmost significance (PSGB 2012). In the front-end model, this process is referred to as project design. The actual utilisation of the selected means or strategy occurs in the execution step, generating an output/result. In the public sector, the output could be an infrastructural product or a new system of citizen service delivery. The utilisation of the output determines the project outcomes or deliverables.
The logical framework approach encompasses three perspectives, i.e., strategic, tactical, and operational viewpoints (PSGB 2012). The strategic view focuses on the society. It entails an emphasis on a project’s societal effect over time. In contrast, the tactical perspective centres on the citizens/users’ acceptance of the project, while operational/implementer’s perspective focuses on the results (PSGB 2012). The framework also includes certain tradeoffs during project definition, design, and implementation.
Trade-offs between outcomes and outputs are considered in feasibility studies related to public sector projects. According to Samset et al. (2016), the outputs must help realise the anticipated outcomes. They are reviewed if they cannot support this goal. A review of the goals (why) or means (how) may also be necessary if the chosen strategy fails to meet the set goals/targets (Samset et al. 2016). Cost-benefit tradeoffs constitute are also considered during project execution to optimise the resources. The tradeoffs described must be considered in the context of the uncertainty in effects, resource availability, and limitations, as shown in Figure 3 below. The tradeoffs are considered in the project design process to refine objectives and select the best concept. Later, during project execution, outcomes vs. outputs as well as costs vs. benefits are considered.
Project Objectives
Government and public sector institutions often undertake big projects with an intention of initiating a service, upgrading the efficiency of current services, or attaining specific developmental goals. As the PMI (2013) writes, successful public projects require the right expertise, adequate resource allocation, and well-defined objectives/goals to minimise costs and risks while maximising the financial returns and innovation. An objective is a quantifiable criterion for determining project success (IPM 2013). It comprises an attribute, metric/measure, and a specific value.
The objectives of public sector projects are fundamentally different from those of private sector projects because the focus is not on financial returns, but rather on the public good or societal objectives (PMI 2013). Therefore, the overarching goal/objective of investment in public projects is to maximise the population of direct beneficiaries or citizens. In other words, public projects aim at maximising the anticipated social utility of a project.
The objectives are formulated at the definition step and the means of achieving them identified at the design step of the project lifecycle. Objectives/goals act as a performance assessment benchmarks for public projects through measurements and feedback. According to Samset et al. (2016), the ultimate aim of defining objectives is to institute restorative actions and promote learning. Therefore, the objectives/targets in public projects are related to assessment. Governments formulate objectives in partnership with contractors for performance measurement based on the identified population needs and anticipated effects (Samset et al. 2016). In this respect, the objectives and goals are useful benchmark standards for evaluating project progress. The determination of goal achievement or progress helps transfer of learning to subsequent projects or phases.
The formulation of quantifiable project objectives occurs on the project definition stage. Project goals and targets are developed from the objectives. In practice, goals help guide or track the implementation process, while explicit targets are the corresponding measurements of project progress. According to Samset et al. (2016), project objectives are useful in two ways: as components of the needs-effect relationship (horizontal) and in vertical learning that is enshrined in project governance models. The top-to-bottom learning is important in organisational improvement. On the other hand, goals are embedded in the horizontal and vertical dimensions of project objectives as shown in Figure 4 below. The definition of objectives and needs is anchored in established norms or experiences that constitute learning. Typically, norms describe the standards or thresholds for identifying societal needs or effects (Samset et al. 2016). They emanate from cumulated experiences or established rules on how projects are initiated. The established standards help in the expression of objectives based on the identified societal needs.
The definition of objectives/goals for a particular project utilises the standards and the outcomes of a needs analysis of the stakeholders. As Cormican (2016) puts it, the process of defining the objectives requires a deep understanding of the needs and priorities of the people affected by the project. This can only be achieved through the involvement of administrators, users, and other groups to ensure a good understanding of their interests (Cormican 2016). A good knowledge of stakeholder needs will help align the objectives of the different stakeholders with the project’s goals. Stakeholder needs analysis also increases the success rate of projects.
Quantifiable project objectives serve multiple functions in public investment projects. Samset et al. (2016) identify three uses of objectives as establishing a common understanding of the purpose of the project, stimulating motivation or social drive, and clarifying the anticipated results or deliverables of individual tasks. Since the planning and execution phases are often marked by iteration, the above functions usually overlap. The first two functions are tied to the parties involved in project planning; their knowledge of the project’s purpose and motivation is essential in setting specific goals. In contrast, the last function relates to explaining the project to the executing parties to enable them implement it (Samset et al. 2016). The functions must touch on the elements of the project.
Logically, for public sector projects, the objectives/goals relate directly to the societal needs and effects. The societal needs can be identified in two main ways. The first one involves planning experts who review and assess the specific needs before describing and expressing them (Pulmanis 2015). The second approach entails the involvement of the citizens and other stakeholders in the identification of the needs, which are the prerequisite for the formulation of objectives. The first method is considered a classical public planning approach. Its main advantage is the efficiency or speed with which the needs are identified. It involves a quick needs analysis by experts based on accessible data. Therefore, the approach is entirely dependent on expert knowledge and scientific tools.
In contrast, the involvement of stakeholders in the identification of needs and in the formulation of objectives is complex, more costly, and time consuming. However, the approach comes with multiple advantages, including enhanced communication, realistic assumptions, and well-aligned objectives (Pulmanis 2015). Stakeholder participation is considered a gold standard in the planning and execution of projects in the public sector. Research interest in this area has focused on whether the approach could yield low quality decisions because of a reduced use of rational methods. The findings indicate that increased stakeholder access to data, ideas, and resources contributes to superior quality decisions compared to pure expert analysis (Pulmanis 2015). Therefore, the best approach for the development or definition of objectives is stakeholder involvement.
The main issue in defining project objectives is consideration of how outcomes are related to the needs at the societal level and the user/tactical level (Pulmanis 2015). The overarching question concerns the effects that would give the users/society the highest benefits. A clear understanding of this relation is a prerequisite for identifying and formulating relevant objectives and goals. In addition, understanding the limitations inherent in the project is important in determining what could be attained under the prevailing conditions (Pulmanis 2015). Another dimension in the formulation of objectives relates to the evaluation of the success rate of a project. The objectives/goals must include key performance indicators (KPIs) to help track progress.
Objectives/Goals of Public Projects
From the logical framework, the development of project objectives or goals comes after the stakeholders have identified the societal need or problem. Economic theory indicates that primary function of public entities is to deliver social/public goods to the citizens (EIO 2011). Therefore, the objectives of public sector projects must centre on the provision of goods that improve the wellbeing of the public in a non-competitive way. In addition, the objectives/goals of public projects should be expressed in broad terms. For example, an appropriate objective of a public project focused on enhancing gender empowerment would be expressed in terms of the percentage increase in the levels of empowerment in women (Jung 2014). Therefore, since the public sector serves as “a provider, a regulator, and a policymaker”, the objectives of public investment projects are modelled around key thematic areas such as economic empowerment, capacity building, human rights, infrastructure, and public health/wellbeing, among others (De Vries, Bekkers & Tummers 2014, p. 3).
The public sector encompasses central organisations and downstream institutions that work closely to develop and implement projects or programmes (De Vries, Bekkers & Tummers 2014). The projects aim to promote economic growth and improve social welfare. The accomplishments of the public sector are seen in the quality of the services/goods provided (Jung 2013). Therefore, the development objectives of any public sector relate to funding infrastructure and supporting socioeconomic growth through industry-specific regulations and policies. Usually, the development of project goals and objectives takes into consideration social development priorities of a country at a given time (IFAC 2013). For example, a project to improve reading outcomes may have a set of objectives that measure relevant process and outcome variables. In the design stage, two kinds of objectives can be set to guide the project. They include process and outcome objectives.
Process Objectives
In public investment projects, some activities must be completed within a particular duration. Process objectives specify the project activities that are expected to be complete within a given timeframe (Robinson 2015). In this regard, they facilitate accountability in public projects by stating the milestones that should be met at a particular time. For instance, a youth development project can have a process objective of achieving 80% regular attendance of the youth in more than two programs in the first six months (Rainey & Jung 2015). In addition, a process objective for an academic/reading improvement project could be to have 85% of students take part in more than two reading activities weekly during the first year of the project.
Outcome Objectives
In public projects, it is important to clarify the outcomes in order to measure the progress over time. Outcome objectives are developed to support performance tracking. They are concise statements that define “who will make what change, by how much, where, and when” (Robinson 2015, p. 8). An example would be; by the end of 2016/2017 academic year, 60% of K-4 learners participating in a national academic/reading program will show ‘benchmark’ literacy skills.
Key Performance Indicators
KPIs refer to the measures of a project’s outcomes or performance. They are tracked during project execution to evaluate progress towards the attainment of the stated objectives (Holzapfel 2014). KPIs are also utilised to determine a project’s success. They are presented in a way reflects the connection between project outcomes/outputs and inputs. In this way, KPIs can help uncover problems that could hamper the attainment of the objectives during the implementation process. In the context of performance tracking, the indicators selected should be simple, measurable, and related to the objectives (Holzapfel 2014). They should track the direct effects on the target population or organisations as determined through an evaluation.
Performance tracking entails periodic evaluation of the project’s progress during implementation. Project progress is tracked based on clear short-term and long-term goals and the outcomes communicated to policymakers to help enhance project performance (Holzapfel 2014). KPIs serve several purposes in public sector programmes. First, performance indicators are useful in strategic planning. Incorporating KPIs into the formulation of a project leads to a greater focus on the project’s premises and expected outcomes (Holzapfel 2014). Thus, KPIs help clarify the goals and premises that underlie a public project or programme. KPIs are also useful in performance accounting. They can help support resource allocation decisions to maximise the outcomes of identified programme activities (Holzapfel 2014). In this regard, KPIs enhance efficiency in resource utilisation in the public sector.
Another important function of KPIs relates to forecasting during project implementation. Evaluating project progress against KPIs may indicate the likely performance trajectory that can lead to optimal outcomes. Thus, KPIs can highlight the areas that need improvement and indicate the interventions that can help realise the objectives. Effective performance indicators are useful in measuring programme results. They measure the projects achievement relative to the stated objectives (EU 2014). In this regard, they help promote accountability among the stakeholders. In programmes that need greater accountability levels, KPIs can be used to communicate the outcomes to satisfy the public and policymakers. The performance data emphasise the significance of the project to the citizens and other stakeholders. They also provide benchmarks for assessing similar projects. Thus, KPIs provide a means for improving programmes based on organisational experiences and learning. They can be used to manage quality through the measurement customer satisfaction levels. Thus, implementers can evaluate whether and how the project is impacting the lives of citizens.
Types of KPIs for Public Projects
From the logical framework, the major variables measured by performance indicators include project objectives, outcomes, and inputs (EIO 2011). The World Bank recommends a range of indicators embedded in the logical framework for use in performance tracking. They include results indicators, direct indicators – relevance, outcomes/impacts, outputs, and inputs – risk indicators, and efficacy indicators, which focus on the sustainability, effectiveness, and efficiency of the project (EIO 2011).
Result Indicators
Among the most widely used KPIs in monitoring project performance are the result indicators. In projects, the immediate or long-term results of a project are measured based on its objectives. The indicators assess the results of a project and they are developed from the outcome/impact indicators (EU 2014). They show the expected effects on the target population or entities that are attributable to the project. Therefore, defining result indicators begins with the determining the project’s impact/outcome and then working backwards to the results/outputs. By definition, result indicators should be closely connected to the objectives of the project. This means that priority should be given to indicators that allow the tracking of results across multiple project activities. Unlike output indicators, result indicators measure the change in the status of the target group. The DFID (2013) recommends that to reduce the effect of exogenous factors, it is prudent to select result indicators that reflect the project activities as much as possible.
Input Indicators
The performance of projects depends largely on resource input or allocation. The input indicators of a programme evaluate the “quantity and quality of resources” that go into a project’s activities (DFID 2013, p. 5). Thus, the quantity of monetary and nonmonetary resources utilised by a project can be tracked using the input indicators. The resources may include funding – co-financing and bank loans, human resources, training, and equipment, including materials/supplies such as vaccines that are required in a public immunisation project. Evidently, tracking the resource input can help gauge the achievement of the objectives relative to the cost of the project.
Output Indicators
KPIs for output measurement indicate the quantity or quality of the products or services that result from the utilisation of the inputs (World Bank 2013). An output constitutes what is generated from the project. In this regard, it can be argued that output indicators measure the efficiency of resource investment in a programme. An example of a good output indicator for a public project meant to provide extension services would be the number of farmers reached at a particular point during its implementation. Similarly, for other public projects such as a vaccination programme, highway construction, or rural electrification programme, the output indicators would be the percentage of people immunised, the number of kilometres of road done, and the energy infrastructure installed in target regions, respectively.
Outcome/Impact Indicators
From the logical framework, it is clear that the outputs or results of a project lead to specific outcomes/impacts on the public (PSGB 2012). Therefore, tracking the outcome/impact of a project gives an indication of whether and how the project is changing lives in the community. Outcome/impact indicators evaluate the quantity or quality of the project’s results based on the actual impact of the goods/services generated (World Bank 2013). For public health project – immunisation programme – the impact indicator could be a decrease in disease incidence in the target population, while that for an extension project could be better animal/crop husbandry. Therefore, impact indicators measure the improvement or decline in a specific variable due to goods or services (results) of a project.
Relevance Indicators
Most public sector projects yield the expected impact on the high-level objectives that outcome/impact indicators cannot measure (World Bank 2013). They include programmes with national or sector-specific objectives. For such programmes, the impact/outcome should be determined at the national level. On the other hand, some projects could produce unanticipated adverse impacts on the public. In such projects, evaluation studies can help measure the impact for appropriate policy interventions. Relevance indicators are applied in evaluating the general directions in the policy issues, which a project is likely to influence (Graham 2011). They are only used if the project objectives are national in scope. Examples of relevance indicators include public health improvement due to health care system reforms and economic growth attributed to a rural electrification project. The main challenge when dealing with relevance indicators is that sometimes it is difficult to ascribe specific impacts to a particular project’s results entirely.
Risk Indicators
Risk and sensitivity analyses of a project give estimates of the effect of attaining project objectives on condition that particular assumptions are not met (World Bank 2013). They provide a picture of the effect of changes to exogenous factors on project variables using the what-if scenarios. Therefore, a risk indicator evaluates the impact of external factors (risks) related to the macroeconomic dimension of the project (CIT). The exogenous factors include those variables identified as having a direct impact on the project outcome. For instance, energy prices or wages for the project team impact on the project outcome directly. The realisation of a project’s objectives is contingent upon the existence of a means-effects relationship of the logical framework components and minimal risks (CIT).
Efficacy Indicators
The efficacy of a multi-stage public project is determined by its efficiency, effectiveness, and sustainability. Efficacy indicators demonstrate “how well the results at one level of a project” have been translated into outcomes in the subsequent level (World Bank 2013, p. 14). They measure three aspects of a project: “the efficiency of inputs, effectiveness of outputs, and sustainability of the project impact” (World Bank 2013, p. 14). Thus, it can be argued that efficacy indicators highlight the capacity of a project to meet the set objectives. The logical framework often utilises efficacy indicators alongside results indicators to measure a project’s impact (Jos & Faith-ell 2012). For most projects, the monitoring of project performance ends with impact indicators. However, efficacy indicators can be utilised to track performance in projects where efficiency, effectiveness, or sustainability is a project goal for the institution.
Therefore, organisations can choose to use indicators of efficiency, effectiveness, or sustainability based on the nature of the project. Efficiency indicators capture the proportion of inputs of a project per a unit of outputs generated (World Bank 2013). An example may be the amount of funds or labour needed per unit output. Efficiency indicators are useful in public financial auditing – for measuring the appropriate use of resources/funds for the activities intended in line with accountability requirements (Jos & Faith-ell 2012). In contrast, effectiveness indicators measure the “ratio of outputs per a unit of a project outcome or impact” (World Bank 2013, p. 15). Thus, they measure the extent to which the project outputs influence the outcomes/impacts. An example is the number of immunisations given per a unit decrease in disease morbidity or mortality in the target population. Sustainability indicators capture the continuity of the project’s positive outcomes over time (World Bank 2013). Thus, they measure the degree to which the project will remain viable over time, especially after funding stops. They indicate whether the benefits will continue being felt after the project ends.
Measuring Project Results and Output
The management of results and output of project activities is essential in tracking project progress. It encompasses systematic collection and analysis of data to determine the efficiency and effectiveness of a programme or project in generating the intended results (OECD 2013a). Results measurement also highlights the improvements necessary to achieve better performance – outputs, outcomes, and impacts. The concept is widely used in management literature and is related to the idea of ‘management by objectives’, which has informed public sector reforms since the 1990s in OECD countries. The approach goes beyond the monitoring project resources and activities to include the measurement of the output of the project, i.e., the results. In this context, results are defined as the “output, outcome, or impact –intended or unintended, positive or negative – of a development activity or project” (OECD 2013a, p. 16).
In measuring results, public entities gather and analyse data to determine how well a project/programme is doing relative to the anticipated goals. A survey conducted in 2013 established that over two thirds of OECD countries have created a special unit/department charged with project results measurement and maintenance (OECD 2013b). Besides results management, the special units coordinate quality and planning of the project being implemented. In most projects, results measurement focuses either on deliverables, including short-term outputs, i.e., direct services or goods or on the long-term impacts/changes related to the project outputs.
Results measurement often involves a system that collects data related to the progress of the project. It entails the identification and formulation, in measurable terms, the anticipated results/impacts, in light of the prevailing socioeconomic, environmental, and political realities that influence their achievement. It begins with the formulation of a results chain that establishes the cause-and-effect relationships between resources and activities, culminating in the expected result (OECD 2013a). Additionally, two systems – one for tracking performance and another for explicating the improvements – are developed to complement the results chain. In measuring results, baselines are first obtained to give a picture of the state of affairs before the project is implemented. The baseline data are integrated with the objectives and selected result indicators to enable governments to determine if performance is on course and the contributing factors. The results derived this way can be analysed with a results model to support public reporting.
An example of a results model is the four-tier framework established by the UK Department for International Development (DFID). The framework helps track and manage the progress of “development results at corporate level, as well as to publicly report on delivery” (DFID 2013, p. 11). Level one of this framework measures the progress made on specific outcome indicators that involve a collective action of the DFID and particular nations. Level two centres on the results, which include indicators assessing outputs attributable to DFID-sponsored projects, while level three covers operational effectiveness of the programme. Improvements at level three can translate into “better delivery of results and greater value for money” (DFID 2013). Level four is concerned with the measurement of organisational effectiveness. The indicators used in this level measure the efficiency of the corporate processes to help bolster effectiveness in delivering the results (DFID 2013).
The four-tier model is comparable to other result systems adopted for public projects. The four-level model determines data collection and measurement techniques used by governments or development partners. Level one indicators are derived from global measurement systems. They are pegged on millennium development goals that act as the point of reference for public projects globally. In contrast, level two indicators are developed from institutional strategies, while the data are drawn from national measurement systems (OECD 2013a). Level three and four indicators relate to the organisational systems. Some OECD countries have developed simpler results frameworks with a maximum of two levels to give extra weight to country-level results (OECD 2013a). In such models, the first level usually covers the output of different projects, while the second level measures the impact/outcome of each specific project.
The primary goal of building a results framework is to “collate and analyse” key data/information for effective project management (OECD 2013b, p. 6). It allows government agencies to monitor the progress made towards achieving specific results of a project, as indicated by the objectives. The development of results frameworks is a collaborative process that brings together government agencies and civil society organisations (OECD 2013b). The first step of this process involves the selection of a few strategic outcomes related to each project. Subsequently, each outcome is monitored through a set of KPIs – a maximum of three KPIs per outcome (OECD 2013b). A joint evaluation of the results by the various actors is done before compiling a results statement. The statements usually report quantitative and qualitative data related to the project results and provide a basis for improvements in the next project phase. Therefore, up to three KPIs should be selected per result, i.e., each for qualitative, quantitative, and survey data.
The selection of KPIs is an important process in the accurate measurement of the outputs/results of a project. The right measurements/indicators can help populate the results framework with accurate data. As stated, indicators – whether quantitative or qualitative – either support the measurement of achievement or changes attributable to the project or gauge its progress. Thus, they define specific aspects of a project based on a scale, highlighting the type of data to be gathered to measure output. In other words, they measure how a project is performing against the baseline data. Consistency in the use of appropriate indicators is required throughout the results chain. Various criteria have been developed to help public entities and officials select good indicators. The common ones used by government agencies across the globe include the “SMART (specific, measurable, achievable, relevant, and time-bound) and RACER (relevant, acceptable, credible, easy, and robust)” (World Bank 2013, p. 24).
National guidelines on how to choose good indicators also exist. In Canada, the government developed a results chain to help public entities develop good indicators and measurements for projects. The results chain begins with the input and ends with the outcomes, which fall into three categories: immediate, intermediate, and ultimate outcomes (DFATD 2013). In the results chain, input indicators measure the investments that have gone into a public project, including money, equipment, staff, technology, and time. The results chain also captures the planned activities related to the project. The activities may include infrastructure development or refurbishment, policy guidance, staff training, workshops, and assessments (DFATD 2013). The outputs indicators measure the products/services stemming from the project activities. They may include the number of staff trained, workshops facilitated, infrastructure built, or policy initiatives supported by the project. The impacts or outcomes of the outputs are measured at three levels: immediate, intermediate, and ultimate outcomes. The immediate outcomes indicators measure the change in capacities of the beneficiaries in terms of knowledge, skills, attitudes, processes, and motivations (DFATD 2013). In contrast, the intermediate outcomes indicators measure the behaviour changes in the target group in terms of decision-making efficiency and effectiveness and social action practices. The ultimate outcome indicators evaluate the change in status or wellbeing of the beneficiaries of the project. They may include indicators of social, economic, or environmental wellbeing.
The results chain given is an example of how outputs and outcomes are measured during the project. It involves quantitative and qualitative indicators that measure the most relevant aspects of a project. However, technical difficulties related to results measurement forces some public entities to select indicators that monitor quantifiable effects, i.e., outputs that are “easy to achieve and measure” (Danida 2011, p. 12). The problem with the use of quantitative results indicators alone is that they do not give adequate measurement of progress towards the desired change in the long-term. Therefore, complexity-aware indicators that involve feedback loops and qualitative data are recommended (USAID 2013a). Examples include sentinel indicators, and process monitoring, and stakeholder feedback – to obtain the perspective of the beneficiaries.
The results measurement systems should be simple and clear. When choosing indicators, it is important to consider the interests and needs of the stakeholders and the objectives of the public project (APEC 2011). Complex indicators can limit the capacity to measure and attribute the results to a public sector project. Therefore, selecting simple indicators can help capture the project results and support management goals. To achieve this, public entities adopt a strategic approach when selecting indicators. They involve technical experts and consider project priorities and budget constraints to choose the right indicators (USAID 2013a).
Collection and Analysis of Data
The common problem experienced when measuring results of public projects is the lack of data related to the selected indicators. The implication of the lack of data relates to incomplete results frameworks, making them unreliable for tracking progress. For instance, incomplete or absent baseline data can affect performance measurement because the initial status of affairs before the implementation of project is not known (USAID 2013b). Therefore, in measuring project results, baselines are crucial. The baseline data are obtained through preliminary surveys that measure the selected indicators. The absence of reliable baseline data can lead to the choice of indicators that would not adequately capture the relationships in the results framework (USAID 2013b).
The data collection approach depends on the application for which the results are intended. Results data can be used for the management of project progress or to meet accountability/reporting requirements (OECD 2014). Low-cost data collection methods, such as surveys, can be used to obtain quality data on the identified indicators for internal decisions (OECD 2014). In contrast, addressing the accountability objective involves data aggregation methods that capture project outputs and outcomes across sectors in the long-term. An important consideration in data collection and analysis is the inclusion of feedback loops within the results framework. According to OECD (2013b), feedback loops allow data based on the survey of the beneficiaries to be included in performance management to enhance the efficacy and efficiency of a public project. They give real-time data that is useful in tracking progress. The management of data is the responsibility of the project staff and managers. Thus, empowering the staff through capacity building programs can improve their ability to choose good indicators and measure project progress.
Project Outcomes and Deliverables
Project outcomes/deliverables may be tangible or intangible. The PMI (2013) gives examples of tangible outcomes of public investment projects as road infrastructure, while intangible outcomes may include human resource or experts. Project deliverables are the outputs of a project. They include the “measurable, tangible, and verifiable” outcome or result that must be generated to achieve project goals (PMI 2013, p. 8). The ordering dictates that objectives are measurable, outcomes may be tangible or intangible, and outputs are tangible (PMI 2013). In the public sector, project deliverables are considered the impacts of public services or goods delivered by a government agency to the people. In this regard, the anticipated impact or change is critical in the definition of project outcomes.
The distinction between tangible and intangible outcomes relates to the physical nature of project outcomes. Tangible outcomes are perceptible or determinable, while intangible ones are not. According to Pulmanis (2015), tangible outcomes are those that have been operationalised in a way that they are easily quantified and monitored. In contrast, intangible outcomes are not easily quantified, but fall into nominal measures of high or low achievement and success or failure. Therefore, intangible outcomes are assumed as opposed to being measured during a project. Examples may include knowledge transfer or learning from the project’s performance for future budgetary considerations or choice of delivery methods. Others may be social capital, relationships, and stakeholder satisfaction.
The measurement of the tangible value often involves the ROI of a project. However, according to Cormican (2016), most public sector organisations seldom measure ROI and management capabilities related to a project. The reason for this relates to the belief that ROI measurement is complex and management capabilities are the project inputs (Cormican 2016). Intangible benefits relate to the organisational effectiveness and learning resulting from investment in a project. Enhanced decision-making processes, improved communication/collaboration, better work cultures, and management effectiveness are all indicators of intangible value (Cormican 2016). Intangible outcomes of a project may also be improved alignment of organisational values and culture and enhanced transparency and accountability.
As stated above, project outcomes are the result of a change project or initiative. An effective outcome management strategy for public projects is required to realise the desired effect from the change or initiative. Outcome management encompasses a range of activities for “planning, managing, and realising” the desired outcomes from a project (Pulmanis 2015, p. 11). It strengthens the capacity of governments to provide services that meet the needs of the public. Outcome management aligns a project to the intended outcomes/results delivered in a timely and on-budget way through the logic model (Pulmanis 2015). The logic model links the activities of a project to the outcomes.
Tangible and Intangible Project Outcomes
The downstream outcomes of a project have a direct impact on the citizens and organisations. One example of tangible outcomes of public sector projects is the provision of basic services to improve livelihoods of the citizens. These services may be in the area of public health, education, urban housing, agriculture, energy etc. The tangible outcomes are achieved through direct government funding. Further, with regard to tangible outcomes, public sector also manages infrastructural development and related projects that the private sector organisations cannot fund because of the high level of risk involved (Cerniauskiene 2014). In the area of public health, tangible outcomes may involve changes in social or economic behaviour of the citizens through projects that focus on cessation of habits like smoking. The government’s role in areas of food or road safety is equally essential in achieving tangible development outcomes. Tangible outcomes also stem from government policies. Tangible outcomes can be achieved through the development of public policies that address socioeconomic challenges. For example, reimbursement programs in the healthcare sector can result in quality care and tangible outcomes of reduced morbidity and mortality rates (Cerniauskiene 2014).
Intangible project outcomes of public sector projects cannot be readily quantified and are less perceptible. An example is fiscal or institutional sustainability. Most long-term government projects or those involving partnerships with international development agencies aim at achieving sustainability outcomes, i.e., project continuity. Other less tangible outcomes may include improved systems and processes related to public spending and debt management, especially for projects that involve donor funds (Caymaz, Akyon & Erenel 2013). As such, public entities learn to operate within the agreed fiscal limits and acquire the principles of external reporting. Collaborative and constructive engagement is another intangible outcome of public projects. The multi-sectoral nature of government projects means that multiple stakeholders – public entities, the private sector, civil society, and community members – must collaborate in the delivery of public goods or services (Tabish & Jha 2011). Further, accountability and transparency systems may occur due to the involvement of the courts, legislature, and audit organisations in project implementation and monitoring.
A logic model gives a visual representation of the relationship between each output and immediate, intermediate, and ultimate outcomes (Pulmanis 2015). An immediate outcome represents the first-level effect/consequence of an output of an initiative. In contrast, the intermediate outcome includes the gains and capabilities derived from the outputs. The ‘final outcome’ encompasses the long-term effects derived from a project. In other words, ‘final outcomes’ occur at the end of an initiative.
The outcome management process in the public sector involves four stages, namely, the development of the outcome realisation model, planning, monitoring of outcome delivery, and optimisation of the outcomes (Pulmanis 2015). For public investment projects, it is important to identify and monitor the intermediate outcomes or indicators in order to attain the ‘final outcomes’. As Pulmanis (2015) states, monitoring the intermediate outcomes helps identify and resolve constraints or problems, contributing to an improved success rate of the project. The strategy also enables implementers or governments to introduce necessary adjustments or adopt a different solution during the execution stage to realise the desired outcomes.
Having an outcome management framework is useful in complex government projects. An outcome management helps dismantle inter-agency silos by involving all stakeholders in the measurement and tracking of outcomes (Pulmanis 2015). The governance framework also helps in clarification of the process of achieving the outcomes, development of the quantitative criteria, and enhancement of accountability (Flyvbjerg 2014). In the public sector, accountability entails obtaining stakeholder buy-in, performance measurement, and building a positive organisational culture.
The outcome management process contains within it the cost-benefit analysis of the initiative. Cost-benefit analysis centres on determining the costs – direct and indirect – related to a project as well as its outcomes (tangible and intangible) (Flyvbjerg 2014). In this way, the management can make effective decisions to realise optimal outcomes. The main limitation of this approach relates to its greater focus on financial outcomes as opposed to ‘soft’ benefits. Thus, project outcomes important in the public sector, e.g., protection of the public, cannot be captured through a cost-benefit analysis. In contrast, the outcome management model supports the measurement of both financial and soft benefits of an initiative.
In evaluating outcomes, the focus should be on the project’s results and outputs/deliverables. As PSGB (2012) writes, the adoption of outcome management in government projects helps focus the implementers on the outcomes and deliverables. This approach provides the rationale for each activity and measures the results of the activity. In this regard, outcome management can indicate the areas that need improvement to realise the desired goals (Cormican 2016). It also helps align the project with government/departmental priorities at its planning stage.
Outcome management is also supports the definition of intangible outcomes of a project. The main challenge faced by public sector projects is specifying and conveying the ‘soft’ benefits that would result from the project to the citizens (PSGB 2012). Using the outcome management framework, implementers can define the public good of a project at the planning phase. As Flyvbjerg (2014) puts it, outcome management allows planners and implementers to establish the connection between individual project activities and the intangible outcomes through a “cause-and-effect logic model” (p. 8). It defines the intangible benefits and stakeholder impact of a project as well as the related costs.
The elements of an outcome management approach, including planning and monitoring of outcomes, can be utilised to strengthen project deliverables. Tangible outcomes such as personnel improvement and cost savings are often expected from a public sector project. However, if the outcome management process is not integrated with government practices, e.g., “project approval and management frameworks, project plans, and scorecards”, the anticipated deliverables may not be realised (Flyvbjerg 2014). The integration helps address systemic challenges in the public sector to achieve higher benefits.
In conclusion, achieving the desired tangible outcomes is often a challenge in the public sector. Projects realise these outcomes at the end, not during project lifecycle. In addition, anticipated tangible outcomes, such as cost/personnel savings, are allocated to other ministries or government agencies (Flyvbjerg 2014). This means that since the realisation of outcomes is not rapid, they are usually not ascribed to the project activities. Additionally, the complex nature of government projects makes it hard to separate and ascribe an outcome/effect to a specific initiative or activity. Further, the government practices of reallocating outcomes or efficiencies, e.g., good governance, obtained in one unit to another makes it difficult to specify and attribute them entirely to a given initiative.
Summary
The focus of this literature review has been on the development of objectives and outcomes measures of public sector projects. The logical model offers a basic framework for designing, executing, and monitoring public projects. It begins with the identification of the societal problem or societal need that requires public sector intervention, followed by the development of goals/objectives and specific activities/means for achieving the expected results/outputs. The direct impacts of the outputs on the beneficiaries produce the outcomes, which can be tangible or intangible. Since the government has a role of providing public goods or services, based on the economic theory, the objectives of public projects centre on key multiple thematic areas such as empowerment, capacity building, infrastructural development, education, public health, etc. Process and outcome objectives can be developed to guide the implementation process.
Once the objectives have been formulated, the next step is the selection of KPIs for tracking project progress. The key considerations in selecting KPIs include relevancy to the project objectives and ability to support realistic measurements. A range of KPIs can be selected for use in projects, including results, input, relevance, and efficacy, output, and outcome indicators, among others. The key consideration when selecting indicators include the nature of the project, capacity to support stakeholder communication through feedback loops, and the results framework utilised. The measurement of project results involves the collection and analysis of data – quantitative or qualitative – related to the selected indicators. A four-tier model has been proposed for tracking the outcome indicators, results, operational effectiveness, and organisational effectiveness throughout the project’s life. In addition, a results chain, such as the one used by the DFATD, are useful in results management. The ultimate outcomes of a project can be tangible, e.g., infrastructure, or intangible, e.g., sustainability outcomes. Therefore, the measurement of both tangible and intangible outcomes is critical in tracking project progress.
Reference
Asia-Pacific Economic Cooperation [APEC] 2011, Framework for results-based public sector management and country cases, Asian Development Bank, Manila, the Philippines.
Caymaz, E, Akyon, FV & Erenel, F 2013, An exploratory research on strategic planning in public institutions: Turkish prime ministry disaster and emergency management presidency case, Procedia – Social and Behavioural Science, vol. 99, no. 1, pp. 189-195.
Cerniauskiene, N 2014, Strategic management of public sector institutions: a student workbook, Mykolo Romerio universitetas, Vilnius, Lithuania.
Danida 2011, Danish development cooperation in a results perspective: Danida’s framework for Managing for development results 2011–2014, Ministry of Foreign Affairs of Denmark, Copenhagen, Denmark.
Department of Foreign Affairs, Trade and Development [DFATD] 2013, Results-based management at CIDA: a how-to guide, Web.
Development for International Development [DFID] 2013, DFID’s results framework: managing and reporting DFID results, DFID, London, UK.
De Vries, HA, Bekkers, VJ & Tummers, LG 2014, Innovation in the public sector: a systematic review and future research agenda, EGPA conference, Speyer, Germany.
European Commission [EU] 2014, Guidance document on indicators of public administration capacity building designing a results framework for achieving results: a how-to guide, The World Bank, Washington, DC, Washington.
European Integration Office [EIO] 2011, Guide to the logical framework approach: a key tool for project cycle management, Republic of Serbia Government, Belgrade, Serbia.
Graham, A 2011, Public sector financial management for managers, CANADEM, Ottawa, Canada.
Holzapfel, S 2014, The role of indicators in development cooperation: an overview study with a special focus on the use of key and standard indicators, Web.
International Federation of Accountants [IFAC] 2013, Good governance in the public sector- consultation draft for an international framework, Web.
Jos, A & Faith-ell, C 2012, New governance approaches for sustainable project delivery, Procedia – Social and Behavioural Science, vol. 48, no. 1, pp. 3239‐3250.
Jung, CS 2013, Navigating a rough terrain of public management: examining the relationship between organizational size and effectiveness, Journal of Public Administration Research and Theory, vol. 23, no. 3, pp. 663–686.
Jung, CS 2014, Why are goals important in the public sector? Exploring the benefits of goal clarity for reducing turnover intention, Journal of Public Administration Research and Theory, vol. 24, no. 1, pp. 209–234.
Organisation for Economic Cooperation and Development [OECD] 2013a, Development result: an overview of results measurement and management, briefing note, OECD, Paris, France.
Organisation for Economic Cooperation and Development [OECD] 2013b, Managing and measuring for results: survey highlights, OECD, Paris, France.
Organisation for Economic Cooperation and Development [OECD] 2014, Measuring and managing results in development co-operation: a review of challenges and practices among DAC members and observers, OECD, Paris, France.
Public Sector & Governance Board [PSGB] 2012, The World Bank’s approach to public sector management 2011-2020: better results from public sector institutions, World Bank, Washington DC, Washington.
Rainey, HG & Jung, CS 2015, A conceptual framework for analysis of goal ambiguity in public organizations, Journal of Public Administration Research and Theory, vol. 25, no. 1, pp. 71-99.
Robinson, M 2015, From old public administration to the new public service: implications for public sector reform in developing countries, UNDP Global Centre for Public Service Excellence, Singapore, Republic of Singapore.
Tabish, SZ & Jha, KN 2011, Important factors for success of public construction projects, International Conference on Construction and Project Management, vol.15, no. 1, pp. 56-64.
United States Agency for International Development [USAID] 2013a, Complexity-aware monitoring, discussion note, USAID, Washington DC, Washington.
United Stated Agency for International Development [USAID] 2013b, Developing results frameworks, technical note, USAID, Washington DC, Washington.
World Bank 2013, Core sector indicators and definitions, The World Bank, Washington DC, Washington.