The evaluation comprised:
- an economic evaluation – to calculate savings made from reduced demand on specialist services; to identify which partners were most impacted by savings; whether/when the investment in the programme would be returned (led by James Mahon).
- A qualitative and quantitative evaluation to find out which families were supported, the range of issues they experienced, what was effective about the model, what outcomes were achieved, how sustainable they were and what impact the interventions had on service use (Kerry Merrill/Helen Palmer)
We used a number of ways to ensure the data was robust and up to date:
In stage one (as part of a wider scoping phase) we explored West Sussex’s data collection policy, processes and systems for individual family data, collated service data and service costs. This included:
- reviewing 3 written case files/activity logs to assess comprehensiveness and quality
- visiting the operational data team to understand data processes/challenges
- reviewing data sources/assumptions on analysis of aggregated data
- viewing results of quality assurance mechanisms
- understanding data sources/complexities/assumptions/ around costing services and outcomes
- eliminating families form the data who had already achieved outcomes prior to the new service
In stages two and three of the evaluation (where we focused on effectiveness, family outcomes and impact) we built data verification into our core method. For example, we conducted 23 systematic case file reviews and then interviewed 12 of these families - to ensure the findings were aligned with data held on them. This case file review and family interview process also helped us to strengthen assumptions underpinning the economic model (sustainability, attribution) in addition to obtaining depth qualitative data.
We used several ways to disseminate analysis and findings:
- From the scoping process, we wrote a short report suggesting how West Sussex could improve their data (to ensure the economic model was robust). We presented this relatively informally to the Director of Think Family, Head of Commissioning, Think Family data team and the two key providers. We discussed findings directly with the analyst teams and input into updated guidance for practitioners.
- We presented the first report on effectiveness and the proposed economic model 6 months in to the evaluation informally to the Director of Think Family and the Head of TF Commissioning, and subsequently to a multi-agency steering group. A summary was posted on the Think Family key worker area of the website for practitioners.
- We produced two final reports:
- The economic evaluation report showed the savings made through actual costs avoided (reduced demand), and how this related to different partner agencies. We also developed 12 illustrative costed case studies to help partners and politicians understand how Think Family worked and the kind of costs associated with individual families.
- The second report focused on qualitative data around effectiveness, outcomes and impact. It detailed the type of families involved, the issues they were experiencing, soft and hard outcomes achieved, perceptions of sustainability of outcomes and how this reduced demand.
The final reports were presented and discussed in a range of different settings. The full report was presented the Think Family Evaluation steering group; we delivered a PowerPoint presentation and summary to the multi-agency Think Family partnership. Our reports have subsequently been used in cabinet and multi-agency partnership to help lever funding from partners and inform the future model. Practitioners were informed through a summary we gave to providers and emailed this directly to individual keyworkers who we had interviewed as part of the evaluation.