12 August 2019

Monitoring and evaluation (safe water business)

monitoring and evaluation
Author/Compiled by
Andrea van der Kerk (IRC)
Vera van der Grift (IRC)
Reviewed by
Jeske Verhoeven (IRC)
Raphael Graser (Antenna Foundation)
Fanny Boulloud (Antenna Foundation)

Executive Summary

M&E

“What gets measured gets done” (Drucker, 1954)

Monitoring is the routine to collect data and information on activities and operations (e.g. from customers, markets etc.). Evaluation is then used to analyse the compiled data and to compare the actual results of a business or project against the set strategic plans. Monitoring and Evaluation (M&E) helps an entrepreneur, NGO or government to track how effective and efficient resources are used and whether one is on track to meet its set goals.

This tool provides information on how a safe water business can implement and improve its monitoring and evaluation scheme. Hands-on insights are provided through an overview of M&E activities of businesses documented in the Safe Water program phase II also.

The corresponding case study highlights in particular what crucial role M&E plays in TARA’s chlorine flask business in India.

What is Monitoring & Evaluation about?

Factsheet Block Body

Monitoring is the routine collection of high-priority data and information on activities and operations (e.g. from customers, markets etc.). Monitoring helps the entrepreneur to track how effective and efficient the business is using its resources towards operations and whether the business is on track to meet its objectives. For safe water businesses, monitoring can for instance entail record keeping such as sales or operational reporting, profit and loss reporting, and tracking of commercial or financial records. For measuring impact also other data are needed such as figures on (continuous) product use, health data, and medical expenses over time etc. (PATH, 2011). Collecting these data can be costly and time consuming. However, this investment can also pay off, the data collection can for instance help to convince investors (KUSTERS ET AL., 2011) and generate revenue from carbon credits (for more information on carbon credits see factsheet on carbon credits).

Evaluation is used to compare actual project results against strategic plans. Unlike monitoring, evaluation is not done on a routine basis but done at specific times (i.e. monthly, bi-yearly, yearly etc.). Evaluation refers to a range of activities that are undertaken to determine if the project activities had the desired impact. There are several types of evaluations that may be appropriate for social businesses: formative evaluations, outcome evaluations, impact evaluations (PATH, 2011).

Why is M&E useful for safe water businesses?

Factsheet Block Body

Monitoring and evaluation (M&E) are important for a safe water businesses in several ways:

  • An M&E system helps the entrepreneur to understand the market and customers’ needs, to inform the management and adapt business strategies accordingly where needed.
  • A well-established M&E system can also help the entrepreneur to determine whether he/she is accomplishing the desired social/health/market impact. This includes monitoring effective and/or continuous use of household water treatment products (HWTS), for instance to apply for carbon finance, to attract social impact investors and/or to meet (inter)national drinking water quality standards.

For whom is M&E relevant and useful?

Factsheet Block Body

M&E is useful for any kind of safe water business throughout the business life cycle from blueprint to scale. M&E helps to steer and adapt sales and marketing strategies, improve the business model and attract investors. The same relevance has M&E also for NGOs implementing a market based safe water approach and governments that are looking for specific measurement methods for safe water implementation.

For investors M&E is also relevant in terms of understanding how social enterprises do measure their impact.

How can M&E be used in practice? Lessons learnt from safe water businesses

Factsheet Block Body

The businesses in the Safe Water Program Phase II are at different monitoring stages. Some have only collected basic information about their customers, water quality in the target area and the number of products they sell. These data were used to develop the sales strategy. To improve their business model and fine-tune their sales strategy, some businesses collect information about the performance of their retailers, the actual use of the product and the water quality at the point of use. Finally, there are some businesses such as Hydrologic who aim to estimate the impact of the use of the product on the customer’s health and economic situation.

M&E stages safe water businesses. Source: IRC & Antenna (2018)
M&E stages safe water businesses. Source: IRC & Antenna, 2018

This is required for instance to acquire carbon finance (for more information see factsheet on carbon credits) and attract impact investors (see section 6). The figures below show these different stages of the M&E of safe water businesses; from the first stage of informing sales and marketing strategies to improving the business model as such and to assess impact and attract investors through being able to outline its company’s social impact.

 

tipTip: the Path M&E Toolkit includes a hands-on approach and tools to help safe water entrepreneurs to develop an M&E framework including monitoring plans, indicator definition and templates to demonstrate social impact.

From tracking sales to measuring impact and costs per M&E intervention. Source: IRC & Antenna (2018)
From tracking sales to measuring impact and costs per M&E intervention. Source: IRC & Antenna, 2018

 

1. M&E methodologies and costs involved

The figure above shows that at every stage additional data is being collected for M&E compared to the previous stage. This comes with extra costs for data collection such as software, staff time, equipment (smart phones, tablets), water quality tests, transportation, etc. The businesses in the Safe Water Program Phase II are using various methods to collect data, ranging from paper-based registries to mobile-based data collection (see the figures below). A sales tracking system that, preferably, integrates real-time data in an online platform is aspired by most businesses in the Safe Water Program Phase 2 to steer management and directly address problems. Hydrologic in Cambodia is the only business that currently has such a system. With this system (i.e. Taroworks) sales agents process sales data via tablets and the management can access these data real-time on an online platform. Spring Health and TARA are piloting similar platforms (see subsequent case studies for more information).

The costs of the various methods are difficult to compare, as the businesses are selling different products and services, in very different contexts. MinErgy is for instance spending about USD 290-340 per year on M&E including staff time. Whereas Hydrologic spends yearly about USD 25,000 a year plus 0.5 FTE (full-time equivalent) staff time on its M&E. The household surveys that Hydrologic does with this budget to collect data on the impact of their product, allows them to generate about USD 540,000 on carbon finance each year (excluding 15% costs for carbon consultant services) (see factsheet on carbon credits for more information).

Overview of M&E methodologies and costs from the social businesses ECCA, TARA, PakoSwiss, Spring Health, Hydrologic, Tinkisso and MinErgy. Source: IRC & Antenna (2018)
Overview of M&E methodologies and costs from the social businesses ECCA, TARA, PakoSwiss, Spring Health, Hydrologic, Tinkisso and MinErgy. Source: IRC & Antenna, 2018

 

2. Indicators suggested by WHO

tipTip: For businesses that sell HWTS products, the WHO M&E Toolkit for HWTS (WHO, 2012) provides guidance to measure a number of these indicators such as product use and water quality. It includes 20 indicators specifically designed for HWTS M&E. The indicators are grouped as per the following 5 categories:

  1. reported and observed use;
  2. correct, consistent use and storage;
  3. knowledge and behaviour;
  4. other environmental health interventions;
  5. water quality.
HWTS indicators suggested by WHO. Source: WHO (2012)
HWTS indicators suggested by WHO. Source: WHO, 2012

 

The WHO Toolkit is not specifically tailored to businesses but (some of) the indicators (shown in the figure above), resources and survey samples can be useful for safe water companies too. Impact indicators are not included, but monitoring the indicators gives an indication of the potential health benefits of the product. At a minimum, WHO suggests that all M&E efforts aim to know if households use HWTS selected indicators 1-4, which concern self-reported use and observations (see also WFP, 2002). The decision tree shown below helps businesses to identify which indicators to measure.

Decision tree to decide which WHO indicators to measure. Source: WHO (2012)
Decision tree to decide which WHO indicators to measure. Source: WHO, 2012

 

3. Which indicators do impact investors want to see?

Impact investments are investments made into companies, organisations, and funds with the intention to generate measurable social and environmental impact alongside a financial return. These impact investors can be banks, pension funds, insurance companies, NGOs, family foundations, government investors and development finance institutions, etc. Well-known impact investors include for instance funds and banks such as Grassroots Business Fund, Acumen and Root Capital.

It can be interesting for safe water entrepreneurs to attract impact investors to generate extra revenue to grow their business. The entrepreneurs do need to show that the business is generating measurable social and/or environmental impact. Collecting data on a regular basis is therefore essential. It is difficult to say which data impact investors want to see in general, because this depends on the investor’s profile and intentions. In the IRIS catalogue you can find which performance indicators leading impact investors use to measure social, environmental, and financial success. This catalogue, hosted by Global Impact Investors Network (GIIN), can also be filtered for specific topics such as water. It also includes several user profiles and a practical guideline.

 

tipTip: on the Global Impact Investors Network website you can find more about impact investment, leading impact investors and performance indicators that they use to measure social, environmental, and financial success.

 

The case study of TARA provides further hands-on insights on how a safe water enterprise can leverage impact through a thorough M&E scheme.

Stay up to date about water entrepreneurship!

Subscribe here to the new Sanitation and Water Entrepreneurship Pact (SWEP) newsletter. SWEP is a network of organizations joining hands to help entrepreneurs design and develop lasting water and sanitation businesses. 

Library References
Further Readings

Participatory Impact Assessment

This guide aims to provide practitioners with a broad framework for carrying out project level Participatory Impact Assessments (PIA) of livelihoods interventions in the humanitarian sector.

CATLEY, A. BURNS, J. ABEBE, D. SUJI, O. (2008): Participatory Impact Assessment. A Guide for Practitioners. Tufts University URL [Accessed: 18.04.2018]

Outcome Mapping: a realistic alternative for planning, monitoring and evaluation

“Outcome Mapping” is a tool that is used during a process documentation for analysing information. This paper reviews Outcome Mapping principles to guide donors considering support for projects using this tool, and other decision-makers seeking methods to improve the effectiveness of aid policies and practice.

JONES, H. HEARN, S. (2009): Outcome Mapping: a realistic alternative for planning, monitoring and evaluation. London: Overseas Development Institute URL [Accessed: 10.11.2010]

Domestic Water Service Delivery Indicators and Frameworks for Monitoring, Evaluation, Policy and Planning: A Review

KAYSER, G. ; MORIARTY, P. ; FONSECA, C. ; BARTRAM, J. (2015): Domestic Water Service Delivery Indicators and Frameworks for Monitoring, Evaluation, Policy and Planning: A Review. In: International Journal of Environmental Research and Public Health: Volume 10 , 4812-4835. URL [Accessed: 18.04.2018]

Impact toolkit

A broad collection of impact measurement tools and indicators designed to consolidate otherwise fragmented impact measurement and management (IMM) resources and direct users to those that are best tailored to their needs.

GLOBAL IMPACT INVESTORS NETWORK (2018): Impact toolkit. URL [Accessed: 18.04.2018]

Dynamics of Rural Innovation – A primer for emerging professionals

This publication provides the reader a broad overview of methodologies to innovate in rural contexts. One specific focus is given to how innovation processes can be started what importance monitoring and evaluation plays. Additionally are different case studies provided that highlight innovation processes in rural settings.

PYBURN, R. WOODHILL, J. (2014): Dynamics of Rural Innovation – A primer for emerging professionals. Arnhem: LM Publishers URL [Accessed: 18.04.2018] PDF

Monitoring and Evaluation of Rural Water Supply in Uganda: Implications for Achieving the Human Right to Water

This paper examines how monitoring and evaluation has been strengthened rural water supply programmes in Uganda in recent years and what various challenges are still remaining, considering their implications for realising the human right to water.

QUIN, A. BALFORS, B. KIELLÉN, M. (2016): Monitoring and Evaluation of Rural Water Supply in Uganda: Implications for Achieving the Human Right to Water. In: Riedel, E. ; Rothen, P. (2006): The Human Right to Water. 33-54. URL [Accessed: 18.04.2018]

WASH Toolkit

The WASH toolkit specifically focuses on setting up and integrating project monitoring and evaluation for water and sanitation projects in schools. The tool can be useful in setting up new M&E measures but also to reflect and improve on existing instruments already in place.

WATER AND SANITATION PROGRAMME (2012): WASH Toolkit. Washington: Water and Sanitation Programme URL [Accessed: 18.04.2018]

Alternative Versions to