Impact measurement is a fashionable concept used by some donors to decide which charities to support. This article analyses the key issues and describes how small charities can attempt this cost-effectively.
Invariably this task will fall to the charity manager to set up and manage the systems amongst all their other duties. Large charities have staff dedicated to this work, but one advantage for smaller ones is that the real flavour of the work can be evaluated and communicated to key stakeholders.
The main point of setting up impact measurement is to assess the effectiveness of front-line services and to improve these. The secondary function is to satisfy and impress funders, current and potential.
Impact measurement starts with monitoring the services provided by your charity. It is best to design and implement a monitoring system in the planning phase of a new service. Sometimes funders, particularly government ones, impose their own monitoring requirements and targets as a condition of funding.
These might cover the “targeted type” of the user (e.g. age, employment status, location) and planned outputs required (e.g. job entry). Diversity must be recorded, such as gender and ethnic origin, which are all important factors in whether services reach the right service user groups.
Any system should also capture and analyse essential data at critical delivery points (e.g. attendance at programmes, retention, progression). You will need a spreadsheet or a Customer Relationship Management system (CRM). Some such as CiviCRM are free.
Data should be collected while service users are still on programmes; it is time-consuming and difficult chasing up service users to retrieve information after they leave. It is also essential to train staff in information collection, while the manager must regularly monitor the data collection process for quality. So much funding is now output–related, with payment dependent on adequate information returns.
Having collected the data, it needs evaluating to assess its impact, examining the quality of the service delivery and answering questions on your project from funders and other agencies, for instance, about effective delivery. It can be considered through a Theory of Change.
The biggest issue is whether your service has made a real impact and affected service users positively. Hard outcomes can be measured (gaining qualifications, job-finding, staying in work, successful rehousing). Softer outcomes, such as health improvements, are harder to assess, as there may be several drivers for change, while another issue is that progress reporting often depends on subjective questionnaires.
A key aspect of evaluation is providing information to staff and stakeholders on good practice while informing planning and review of services and deliverables. It should make judgments about the performance of a service, offering constructive criticism and necessary improvements.
Services sometimes succeed on their second attempt after an initial evaluation has taken place and adjustments made. The objective of an evaluation is to find out what really happened, with the aim of measuring outcomes against targets; and, if necessary, taking corrective measures if targets are not met.
Case studies are a useful way to illustrate your charity’s impact. However, it often becomes difficult for staff and managers to be completely objective about the effect of the services they have delivered. When you have worked long hours to get people to overcome difficulties and progress, it can be hard to see what else should have been done for a service user.
However, a very experienced charity manager is often sceptical, feeling that reports are written too positively. Many with wonderful impact reports, especially from the bigger agencies with lots of colourful diagrams, pictures and case stories. None of which is ever challenged by a funder, so frankly, we can write whatever we think the funder would like to hear. Hence he is much keener on objective external evaluators.
In fact, trusts often require an external evaluation as a condition of their grants and their inclusion within the budget. It is useful to publish any evaluation with anonymised case studies as it can help publicise the charity.
Developing successful impact measurement systems can be hard to achieve, but help is available through NCVO and local voluntary sector networks. It can also be linked to quality systems and staff management – all of which will be discussed in future articles.