Using Indicators to Measure Progress and Performance

Compiled by:
Beat Stauffer (seecon international gmbh)
Adapted from:
TAYLOR, P.; LIDEN, R.; NDIRANGU, W.; JIN, L. (2008)

Executive Summary

Progress and performance must be measured to attest a development in a (long term) project. With few, but carefully selected indicators, it is possible to get a good overview on the progress and performance. Indicators can be used in many different areas including water and sanitation. Here, you will find how indicators work and how they are developed.

Introduction

“How do we know process is being made? How can we know if an intervention is making our local water and sanitation system more sustainable?” These are very important questions regarding projects or programmes in water and sanitation. Therefore a suitable method is needed to make the development visible and measure the progress.

Indicators are one approach to measure progress.The term "minimum indicators" is used in recognition that:

  • It is better to start with a small set of indicators that are feasible to monitor and to improve over time.
  • There are many other indicators that could be used to measure progress beyond this basic level.

Indicators provide an effective tool to measure progress and performance. An indicator is the representation of a trend tracking the measurable change in a system over time. Generally an indicator focuses on a small, manageable set of information that gives a sense of the bigger picture. Therefore it can be seen that there is no need to measure everything. Furthermore the choice of indicators is important as to whether it gives sufficient ‘sense of the bigger picture’.

Only well selected indicators can assist the process to maintain a focus on the important work areas and take strategic decisions to address problem areas. That makes a project sustainable and allows responsible persons to act.

Use of Indicators

Indicators are useful to:

  • Measure progress over time against various water and sanitation objectives providing information relevant to policy.
  • Measure performance against a target to evaluate the effect of policy actions and plans.
  • Present information to the public or stakeholders in a simplified way.
  • Identify areas for increased attention by an organisation.

Indicators Have Two Core Functions:

  • To provide system information to inform the project team (responsible organisation), the public and policy makers.
  • To translate data into policy relevant information. That is, they describe, show trends and communicate the results of implementing objectives.

Criteria for Developing Indicators

The most important point in developing indicators is not to be too ambitious. Start with what can be realistically done or else failure is guaranteed.

  unmillenniumproject.org

Millennium development goal 7, target 10 and its indicators. Source: UN MILLENNIUM PROJECT (2006)

a) Simple, Easily Measured, Understood and Applied

The data used for indicators should be in a format that is easy to use. The more complex the indicator the less useful it will be.

b) As Few as Necessary

The capacity to measure and report is usually limited by financial and human resources.Being burdened with an excessive number of indicators may mean that the system fails to achieve the expected benefits or does not work at all. A set with a large number of indicators will tend to clutter the overview it is meant to provide.

c) Use Existing Information Where Possible

It is preferable that the information needed to measure an indicator is available through existing data sources and monitoring programmes or that data collection can occur through existing programmes. This will improve the cost effectiveness of the system.

d) Relate at the Appropriate Scale

An indicator should be related to the specific situation it is "indicating" information about. The indicator should be measurable at an appropriate scale.

e) Detect Change

The indicator should be able to detect change. If it does not reflect change because it was poorly selected or the situation has changed then another indicator should be identified.

f) Comparable, Repeatable and Defensible Between Sites and Times

It is an advantage to use indicators which are comparable between similar projects in different countries. That allows transboundary comparisons and assures the choice of the indicator.

Note: Indicators are useful tools for measuring progress and motivating action in specific areas. They should be limited to those that can reasonably be measured. Start small, build up gradually.

Typical Indicators to Measure Sustainability in Sanitation and Water Management

(Adapted from MUGA and MIHELCIC 2008)

  UN-Water Africa 2006)

Typical indicators in the field of water and sanitation. Source: UN-WATER; AFRICA (2006) 


 

Depending on the scope of an intervention in the field of sustainable sanitation and water management, different indicators will be applied. Some typical specific examples to measure sustainability are:

Water Sources Management

  • Water withdrawal: absolute or per-person value of yearly water withdrawal gives a measure of the importance of water in the country's economy.
  • Ground water level development (rise or fall of ground water levels, pollution).
  • Water sources quality (e.g. turbidity, minerals content, pathogens, chemical quality).

Water Purification

Water Use

Wastewater Collection

Wastewater Treatment

(Adapted from MUGA and MIHELCIC 2008)

Reuse and Recharge of Nutrients and Water

  • Percentage of reclaimed water used in agriculture to total agricultural water use.
  • Percentage of nutrients used from recycled sources.

Of course, this list is by no means exhaustive. Furthermore, also other, less technical indicators (such as e.g. hygiene behaviour, knowledge on water related diseases etc. could be measured).

Applicability

Indicators are useful to measure progresses and performances of (long term) projects in water and sanitation. It makes successes or failures visible and with a precise influence on the indicators problems can be solved.

Advantages

  • Indicators can be adapted to the circumstances
  • It is not necessary to measure every single parameter to get a “bigger picture” over the whole progress
  • Failures can be detected and eliminated

Disadvantages

  • Too many indicators; A set with a large number of indicators will tend to clutter the overview
  • Measuring indicators is a long-term exercise. If the indicators are not defined properly, and if measuring is not done correctly and conscientiously, it will not work

References Library

TAYLOR, P.; LIDEN, R.; NDIRANGU, W.; JIN, L. (2008): Integrated Water Resources Management for River Basin Organisations – Training Manual. Pretoria: International Network for Capacity Building in Integrated Water Resources Management (Cap-Net). URL [Accessed: 15.11.2010]. PDF

UN MILLENNIUM PROJECT (Editor) (2006): UN Millennium Project: Goals, Targets and Indicators. URL [Accessed: 15.11.2010].

EPA (Editor) (2006): Distribution System Indicators of Drinking Water Quality. URL [Accessed: 23.11.2010]. PDF

MUGA, H.E.; MIHELCIC, J.R. (2008): Sustainability of Wastewater Treatment Technologies. In: Journal of Environmental Management 88, 437 - 447. URL [Accessed: 22.04.2012]. PDF

UN-WATER; AFRICA (2006): African Water Development Report 2006 – Chapter “Indicators - Measuring the Progress of the African Water Vision. Addis Ababa: UN Water/Africa. URL [Accessed: 15.11.2010].

Further Readings Library

Reference icon

EPA (Editor) (2006): Distribution System Indicators of Drinking Water Quality. URL [Accessed: 23.11.2010]. PDF

This paper compiles available information on indicators of drinking water quality within potable water distribution systems. The indicators include microbial and non-microbial parameters, for which sample collection and analyses could be performed to identify existing or potential problems, as well as other methods or tools that may similarly function as problem indicators.


Reference icon

MUGA, H.E.; MIHELCIC, J.R. (2008): Sustainability of Wastewater Treatment Technologies. In: Journal of Environmental Management 88, 437 - 447. URL [Accessed: 22.04.2012]. PDF

This paper presents a set of indicators that incorporate environmental, societal, and economic sustainability that were developed and used to investigate the sustainability of different wastewater treatment technologies, for plant capacities of 5 million gallons per day.


Reference icon

TAYLOR, P.; LIDEN, R.; NDIRANGU, W.; JIN, L. (2008): Integrated Water Resources Management for River Basin Organisations – Training Manual. Pretoria: International Network for Capacity Building in Integrated Water Resources Management (Cap-Net). URL [Accessed: 15.11.2010]. PDF

This training manual about Integrated Water Resources Management contains a module on indicators and its application.


Reference icon

OLSCHEWSKI, A.; DANERT, K.; FUREY, S.; KLINGEL, F. (2011): Review of Frameworks for Technology Assessment. WASHTech Deliverable 3.1. St. Gallenand The Hague : Swiss Centre for Development Cooperation in Technology and Management (SKAT) and International Water and Sanitation Centre (IRC). URL [Accessed: 14.03.2012]. PDF

A WASHTech literature review of existing frameworks for technology assessment reveals that there is a gap for a WASH technology assessment tool and a WASH technology uptake tool. The authors of the review, which supports the development of WASHTech’s Technology Assessment Framework, (TAF), conclude that a computer tool based on an algorithm is not appropriate because it is too rigid. Choosing a manageable number of appropriate indicators is key for assessing new technologies.


Reference icon

GALAN, D.I.; KIM, S.; GRAHAM, J.P. (2013): Exploring Changes in Open Defecation Prevalence in Sub-Saharan Africa Based on National Level Indices. In: BMC Public Health 13, 1-12. URL [Accessed: 28.08.2013]. PDF

This study estimates the changes in open defecation prevalence between 2005 and 2010 across countries in Sub-Saharan Africa. It also explores the association between national level indices and changes in open defecation prevalence and assesses how many countries can achieve “open defecation free status” by 2015.


Reference icon

ZEWO (Editor) (n.y.): Outcome and Impact Assessment in International Development. Zewo Guidelines for Projects and Programmes. Zurich: Schweizerische Zertifizierungsstelle fuer gemeinnuetzige, Spenden sammelnde Organisationen (ZEWO). URL [Accessed: 04.10.2013]. PDF

These guidelines are designed to help project managers to assess the outcomes of their projects and programmes. They demonstrate how development agencies can implement an appropriate outcome and impact assessment system.


Reference icon

COWLING, R.; NORMAN, G. (2013): Achieving Sustainability: Changing the Ways in Which we Define Success. (= Perspective, 1). London: Water and Sanitation for the Urban Poor (WSUP). URL [Accessed: 25.11.2013]. PDF

Implementing agencies like WSUP, Water For People and IRC are of course accountable to their funders, including major bilaterals and foundations. And naturally, these funders must track the effectiveness of their spending. But short budget cycles and the need to demonstrate “value for money” can often encourage over-simplistic measurement of success in terms of short-term outputs, rather than genuinely sustainable services. This note proposes some ways forward.


Case Studies Library

Reference icon

UN-WATER; AFRICA (2006): African Water Development Report 2006 – Chapter “Indicators - Measuring the Progress of the African Water Vision. Addis Ababa: UN Water/Africa. URL [Accessed: 15.11.2010].

The Africa Water Vision 2025 was formulated to guide the development and management of water resources for sustainable development. A number of indicators were selected along the lines of the methodologies developed for the World Water Development Report in order to quantitatively measure progress in the implementation of water-related projects.


Reference icon

WICKEN, J. (2008): Measuring Sanitation: Outcomes as well as outputs. London: WaterAid. URL [Accessed: 15.11.2010]. PDF

At the second SACOSAN conference held in Islamabad in 2006, countries in the region committed to work together to identify a new set of indicators for sanitation and use these to report on progress at the next SACOSAN to be held in India in 2008. This paper has been prepared as a thinkpiece for consideration by governments and stakeholders working on possible indicators.


Training Material Library

Reference icon

VUJCIC, J.; RAM, P.K. (2013): Handwashing Promotion: Monitoring and Evaluation Module. New York: United Nations Children's Fund (UNICEF). URL [Accessed: 04.02.2014]. PDF

This guide will walk you through planning and implementing monitoring and evaluation (M&E) for your handwashing promotion programme.


Important Weblinks

http://www.unmillenniumproject.org/ [Accessed: 15.11.2010]

This is an example how indicators can be used in several subjects (e.g. social, environmental or educational)

https://improveinternational.wordpress.com [Accessed: 06.06.2013]

This is an ongoing compilation of statistics to show that failure rates for water systems, latrines, and hygiene promotion campaigns are still high after decades of intervention.

http://www.washplus.org/ [Accessed: 19.12.2013]

The WASH Sustainability Index Tool, developed for the USAID-Rotary International H2O Collaboration, is a tool to assess sustainability of WASH programs. The tool considers the sustainability of institutional, management, financial, technical and environmental factors.