How to use qualitative data to prove social impact

Qualitative data can prove social impact by showing clear, real-world examples of how a program changes people’s lives. The real game changer is using voice-based feedback and natural language processing (NLP), which makes it possible to capture, analyse, and link authentic stories to measurable outcomes - at scale. This is something that was not practical in the past.

However, many investment models rely heavily on 'hard' endpoint outcomes, e.g. employment rates, school attendance, reduced crime, health service utilisation, etc. What is missing are tools that help build evidence on the journey people go through to achieve these outcomes. This omission has been noted by the likes of the New Zealand Social Investment Agency. They recognise the need for tools and frameworks that are easy for service providers to use to show the cultural/social impact of their services, and the journey that people go through to achieve these outcomes. Central to the model are people's stories and their lived experiences.

The aim of the short blog is to address three key areas:

1. Outline the current gap in SROI models

2. How AI can help capture unstructured feedback to address these gaps

3. An emerging model to weave in analysis of qualitative data into the SROI process

The funding landscape is changing

Social impact investing is a tool that is being used more and more by Government funders. In New Zealand there is the Social Investment Agency, Australia has the Commonwealth Outcomes Fund, and recently, the UK Government announced the 'Better Futures Fund'.

In all these funds, the focus is on evidence of impact/outcomes rather than outputs. For NGOs and service providers, this shift is both an opportunity and a challenge. Where organisations can use evidence to show the impact of their work, they are more likely to secure additional funding, or at least retain the funding they already have.

The investment model relies heavily on 'hard' endpoint outcomes. While these are important, they typically show up after an extended period of time. There are many other outcomes that are part of the journey towards these endpoint outcomes. However, without the tools to capture this data, the social investment model is at risk of only identifying and funding services that are working with those already far down the pathway of resilience. This is not the part of the community that is the most at risk, with one estimate suggesting 15% of the population with the highest support needs use 50% of the Government's services.

The gap in the current SROI models

While a typical SROI approach will have evaluative tools to capture the pathway to change (e.g. a programme logic model and/or a theory of change), they typically lean much more into quantitative data. That is not surprising, as it is more efficiently collected and lends itself more easily to the economic analysis needed to get SROI ratios.

The hidden drivers of social outcomes

However, when SROI relies primarily on quantitative metrics, several limitations emerge:

1. Misses the 'how' behind changes. For example, you might see higher school attendance, but not know it was due to a young person feeling safe in a stable house. This is the sort of insight that qualitative data, collected at scale, can bring. It is this evidence that helps us understand the critical factors in service design that lead to outcomes.

2. Ignores cultural and social drivers. Many outcomes are influenced by such things as trust, belonging, and connection. These aspects do not appear in administrative datasets. Having a system to routinely capture it is a critical part of the evidence base, and understanding the person's lived experience.

3. Undervalues prevention and early intervention. Many SROI models rely on administrative data. For example, in New Zealand the Social Investment Agency plans to lean heavily on the Integrated Data Infrastructure (IDI) to look at outcomes. Of concern here is that the outcomes captured can take some time to appear. Further, and importantly for early intervention services, the outcomes they are looking to achieve are not likely ones reflected in the likes of the IDI. For example, consider a programme aimed at parents of toddlers at risk of behavioural challenges. Key measures here might be things like parents' confidence or attachment quality. Neither of which are captured in administrative data. The sorts of relevant measures that are (e.g. B4 School Check, Year 1 literacy/numeracy) will take 2-3 years to appear after the intervention. Longer-term impacts, e.g. school attendance/attainment, lower justice involvement, will take 10+ years to appear.

4. Fails to capture intangible benefits. Achieving social outcomes often comes after an increase in things like confidence, self-worth, and community cohesion. Think about programmes like Dress for Success for the importance of building confidence to go to job interviews, or Whānau Ora for building community cohesion. These are the sorts of changes that can readily be captured through analysis of qualitative data. Identifying where and how this impacts on hard endpoint outcomes is a critical part of understanding the impact of social investment.

Opportunities to capture unstructured feedback.

Given these gaps in the current model, what are the opportunities to capture unstructured data, and what do we do with it?

There are many touch points that provide the opportunity to collect unstructured data. Think about a typical large retail business these days. Sophisticated consumer experience systems are capturing feedback along the journey - from the initial enquiry to post-purchase feedback. The same opportunities hold for health and social services. In referrals, there are typically case notes. Each time there is client interaction, there is an opportunity for capturing client stories. Upon discharge is another opportunity to get rich feedback about their experiences and outcomes. These interactions can be captured in a myriad of ways.

• Stories told in interviews or focus groups

• Voice notes left after receiving a service

• Open-text responses in surveys

• Case notes recorded by front-line workers

• Comments on social media posts

Given the quality-rich interaction health and social services typically have with their clients, why is qualitative data not captured more systematically?

Weaknesses of qualitative data in SROI (and how to address them)

There are a number of aspects in qualitative data that, until recently, have impacted on how they can be used in SROI. It is these 'weaknesses' that, I suspect, have led to a very heavy bias towards things that are easy to count at scale.

1. Time-consuming to analyse. Until very recently, qualitative data has taken a long time to analyse. The rough rule of thumb was 3 hours of analysis, interpretation and write up for every 1 hour of spoken feedback. Advances in AI and NLP have cut that time down significantly. You can get almost 'instant' high-level thematic analysis through common AI platforms (e.g. Chat GPT5). A criticism of these models, though, is that they lack the specificity or nuance of the local context. In these cases there are NLP models that can be trained to understand the nuance, frameworks, and local context. For example, AWS Comprehend provides a service where an analyst can train an existing model. Within Talkscape, our graphical interface supports clients in training their own NLP model. This may initially take half a day. After that point, though, the model can instantly analyse the qualitative data captured and report it against the themes/sub-themes of their own model. No longer is time a barrier to collecting large amounts of qualitative data.

2. Perceived as anecdotal. Too often, qualitative data is perceived as anecdotal due to the small sample from which the data is drawn. Now, with advances in voice-to-text and AI algorithms to probe for more depth, we are no longer limited to a small selection of people's views. In fact, in the context of health and social services, there is the opportunity to get rich feedback at every touch point. This means that the data is no longer anecdotal. Rather, it is as representative as any quantitative outcome measure. Further, analysing the qualitative data with the quantitative data increases the robustness of both data sources.

3. Risk of bias. Similar to the anecdotal criticism, qualitative data has been seen to have bias. This is due to qualitative data only captured from people who can be bothered to engage in things like interviews or focus groups. These individuals often have strong views - at either end of a continuum. Now, qualitative data can be efficiently collected and analysed at scale; there is no longer bias. Rather, every voice can be heard - with all the experiences and outcomes (positive and negative) captured.

   
   

How AI makes this possible

AI now makes it possible to reliably capture and analyse qualitative data to show the impact on social outcomes. The following tools/techniques are the basis of the AI-supported pathway.

1. Automated data capture: Automated speech recognition tools now convert spoken feedback into text instantly. These models are designed to accurately transcribe different languages, with a 95% level of accuracy. These tools are in platforms like Talkscape's Voice of Community. It means people don't need to type feedback, and provides more inclusivity, e.g. for people who have English as a second language. There are even models designed to capture different dialects of the same language, e.g. Papa Reo accurately transcribes different spoken dialects of the Māori language.

2. Thematic analysis at scale: Advances in Natural Language Processing make it possible to accurately identify themes and sub-themes in data. Importantly, for looking at the relationship between process and outcomes, it is possible to use algorithms to map process and outcome journeys in the data. This can then be looked at across time or between groups.

3. Sentiment and emotion detection: Sentiment analysis makes it possible to easily identify positive and negative outcomes, and defined and experienced by the service users. This provides quantitative data that can then be fed into the model that looks at the social impact of a service.

   
   

AI assisted steps to capture and analyse qualitative data for showing social impact

A high-level methodology on how to use qualitative data to show social impact

A model that links the qualitative data into an approach to show social impact has a number of steps. This model assumes that the qualitative data is captured and analysed using the sorts of AI tools discussed above.

Step 1 - Set up the study design and data model

At a minimum, there are two components to the database that need to be set up.

i) Time points for both the qualitative data collection and the quantitative outcomes.

• For example, T0 (baseline), T1 (midpoint), T2 (end/discharge). At each point, capture qualitative data about: 1. the things that are impacting their health and wellbeing; 2. the things that would be helpful; 3. The outcomes they have seen (or hope to see) from the programme/service)

• Along the same time frame, capture the critical outcome makers. This allows for comparable pre/post measures

ii) Linking and identifiers

• Create a participant_id that is used across the qualitative and quantitative datasets. Typically, this is automatically created in the CRM that is being used.

• Record the session_id, date, programme_id, and any exposure markers (e.g. received shoes, freezer, medical debt relief).

Step 2 - set up the outcomes framework

There are two ways this can be constructed. You could use the themes in the qualitative data to construct the outcomes framework, or create an outcomes framework based on something like a programme logic development exercise. This model needs the high-level domains, the sub-themes, and how change will be mapped, e.g. client's need(s), activity that reflects working on need, outcome/change.

Step 3 - turn the narratives into analysable variables

In the outcomes framework are typically the themes and the domains. Both need to be used to create metrics for the outcomes model

i) Calculate the theme metrics

• Prevalence: pit =1(theme mentioned) or frequency share of all the themes in the verbatim.

• Intensity: proportion of the transcript devoted to the theme (tokens or seconds)

• Sentiment: average sentiment score sit [-1, 1] for the theme.

• Direction of change:

  Δpi​=pi,T2​−pi,T0​

  Δsi​=si,T2​−si,T0

ii) Story of change indices (SCI)

• Domain-level index (per domain d)

  SCIid ​= w1​Δpid​ + w2​Δsid​ + w3​Δintensityid​

• Weights (w's): set from the community-defined priorities at T0 (e.g. if housing is a top priority, then weight it higher), or equal weights if no priorities across the domains).

iii) Intervention level exposure

• Binary or dosage per type (e.g. shoes_provided, debt_relief_amount, freezer_provided), dated so the timing impact can be modelled.

Step 4 - mechanism tagging (linking the qualitative and quantitative data)

We now have data at different time points on the themes, the intervention(s) and the outcomes. This means we can tag the three areas together to tell the story of impact.

• For each theme, we now have the plausible mechanism tagged (e.g. shoes linked to school attendance, or debt relief linked to GP access).

• We can also link the intervention to the story of change to the outcome, e.g. debt relief (exposure) leads to an improved story of change in financial/health domains, leading to increased GP visits and less hospital attendance.

Step 5 - SROI analysis

At this point it is possible to complete the SROI using the standard methods. These are discussed more fully elsewhere:

• Create a benefits map: Assign a unit of economic value to each outcome. There are standard ways these are calculated.

• Use published estimated effect sizes to apportion what fraction of the outcome is linked to the programme.

• Compare the outcome to a counterfactual (e,g. a matched sample, or pre-trend data)

• Calculate the duration and drop-off of the effect, e.g. determine if the benefits might plausibly persist beyond T2 (e.g. sustained GP re-engagement apply a short duration (6-12 months) with a drop off.

• Calculation of total benefits and total costs the calculate the SROI ratio.

Step 6 - Outputs and the visuals

Bringing the qualitative and quantitative data together can tell the full story of impact. Some of the tools to show the impact include:

• An interactive dashboard that shows the domains and themes. These can be filtered by time (T0, T1, T2) and key demographic variables. Platforms like Talkscape's Voice of Community include the ability to hear the people when they are talking about the relevant theme/domain. This brings a richness to the story of change and impact.

• A heat map to show the changes in the various domains over time (T0 - T2). This can include the prevalence, intensity, and sentiment.

• The trajectory line of the SCI (priority domains) vs outcome changes. This shows visually how the changes in the SCI are related to outcome changes.

• A mechanism ribbon to show which intervention(s) preceded changes

A practical worked example

I get that the above may have gotten a little too technical. Hopefully, the following practical illustration helps clarify the process.

This is a scenario that looks at an intervention designed to reduce the impact of rheumatic fever. It includes reference to outcomes that are related to a cultural framework used in New Zealand about holistic wellbeing - Whānau Ora.

Baseline (T0)

Themes captured in the qualitative data:

• Housing (kāinga): "The house is damp, and the kids are always coughing" (negative sentiment -0.8)

• Financial stress (Pūtea): "We can't pay off the medical debt, so we're not going back to the GP" (sentiment -0.9)

• Physical wellbeing (Tinana): "My youngest has sore throats, we just ride it out" (sentiment -0.6)

Interventions received:

• Medical debt relief approved (value $250)

• Dehumidifier delivered for the home

Outcomes measured:

• 2 ED visits (1 RF-related)

• 0 GP visits

• School attendance 72%

Midpoint (T1)

Themes captured in the qualitative data:

• Housing (kāinga): "The dehumidifier helps a bit, the kids are sleeping better.” → sentiment improves to –0.2.)

• Financial stress (Pūtea): “We cleared the debt and went to the GP.” → sentiment +0.5.

• Physical wellbeing (Tinana): "We caught a sore throat early and got antibiotics.” → sentiment +0.6.

Outcomes measured:

• 1 ED visit (non-RF related)

• 2 GP visits

• School attendance 82%

Endpoint (T2)

Themes captured in the qualitative data:

• Housing (kāinga): “Still need repairs, but it’s warmer and less damp now.” → sentiment +0.3.

• Financial stress (Pūtea): “Money is still tight, but we’re managing doctor visits.” → sentiment +0.2.

• Physical wellbeing (Tinana): “The kids are much healthier, no strep lately.” → sentiment +0.8.

Outcomes measured:

• 0 ED visits

• 3 GP visits

• School attendance 90%

• Parent returning to part-time employment (12 hours/week)

Story of change indices

When plotting this data into a matrix we get:

Domain	Sentiment T0	Sentiment T2	Δ Sentiment	SCI
Kāinga	-0.8	+0.3	+1.1	Strong ↑
Pūtea	-0.9	+0.2	+1.1	Strong ↑
Tinana	-0.6	+0.8	+1.4	Strong ↑

Direct economic benefits

In this simple example we have the following direct economic impacts.

Outcome	Change (baseline to end)	Unit value	Benefit
Ed visits avoided	2 → 0 (–2)	$1,200 per visit	$2,400
GP visits added	0 → 3 (+3)	–$75 per visit (cost, but positive substitution)	-$225
Net saving (ED-GP)			$2,175
School attendance (+18%)	Not monetised (outside scope)
Employment (12 hrs/week × 12 weeks × $23/hr)	$3,000	$3,000
Total Benefit			$5,475

Programme cost

Debt relief $250 + dehumidifier $400 + programme overhead $600 = $1,250

SROI = $5,475 / $1,250

= 4.4

Every $1 spent generates ~$4.40 in direct savings/earnings.

Bringing this together

For illustrative purposes, assume you have this data for 50 families. We can calculate the impact score by combining the frequency, sentiment, and intensity measures. This gives a score for each theme, at each time point (T0, T1, T2) for each family. We also have the outcome measures at each time frame. This now allows us to do both correlation and regression analysis. For example, using correlation we can show:

• A change in Pūtea impact leads to a change in GP visits (r = +0.52 (p < .01)).

In regression, we could show:

• Each +0.5 improvement in Pūtea impact → +1.1 GP visits, –0.7 ED visits.

All this is now possible

With the increasing focus on social impact (and associated funding models), it is important to ensure the lived experience is captured. This data provides the critical insight into what is changing for people, how, and links this to the endpoint outcomes. With advances in AI, tools like Talkscape make it possible to easily capture and analyse this data - routinely and at scale. The above provides a method for how this data can then be woven into a model that clarifies the change story.

The result is the empirical evidence on how the 'shifts in the voice' predict the 'shifts in the numbers'.