Project Guidance and Evaluation
Project Advisory Group
The Project Advisory Group draws on the practical experience and knowledge of its members to:
- Advise the project team and other participants as to the likely effectiveness, relevance and applicability of plans and activities
- Propose additional activities and refinements to activities in support of the project and its aims
- Promote the aims of the project with interested parties
- Facilitate stakeholder engagement
- Advise Friends Provident Foundation Trustees on the project
- Assist in sharing the learning arising from the project
- Support plans for the wider rollout of the model that is being trialled (if the pilot is successful)
Its membership includes representatives of our funding organisation, the credit industry, regulatory bodies, Payplan, professional bodies, pilot agencies, academics and AdviceUK:
Andrew Thompson, Grants Manager, Friends Provident Foundation
John Fairhurst, Managing Director, Payplan
David Hawkes, Sustainable Debt Advice Project Executive and National Money Advice Co-ordinator, AdviceUK
Paul Jones, School of Sociology and Social Policy, Liverpool John Moores University
Colin Kinloch, Money Advice Service
Fiona Magee, Deputy CEO, AdviceNI
Paul Ross, Director, Retail, British Bankers’ Association
Sallie Johnson, CEO, Institute of Money Advisers
Colin Trend, Project Director, Money Advice Plymouth
Alex MacDermott, Special Adviser, Credit Services Association; Debt Buyers and Sellers Group
The Role of the Independent Evaluator
Public Perspectives Ltd has been appointed as independent evaluator to assess the success of the project pilot. They will provide an evaluation of the success of the pilot stage of the project in achieving it aims. The role is principally an evaluation for learning not compliance or accountability. Therefore, the intervention of the evaluator is to feed lessons into the process and to learn from the process and outcomes.
The evaluator will:
- provide input to the definition of the monitoring and evaluation framework for the pilot
- plan and design evaluation activities
- provide an interim assessment of the progress towards planned outputs arising from the pilot, midway through its execution
- analyse and interpret data collected through the project team’s monitoring activities
- carry out additional data collection activities as seen necessary, to enable a robust analysis and evaluation of the outcomes of the pilot stage of the project
- Using monitoring and other information collected, make judgements about the project pilot. Based on the outcomes of the pilot, recommendations need to be made regarding the viability of the pilot model and its wider rollout
- attend an estimated three meetings of the Project Advisory Group
- prepare an evaluation report, conforming to Friends Provident Foundation’s publication requirements. An initial draft should be prepared with opportunity for comment before it is finalised.
- carry out any other activities that can reasonably be required as an independent evaluator.
Pilot Monitoring and Evaluation Scheme
Monitoring and evaluation will be based on an assessment of the following aims and target outcomes.
Sustainable for Agencies
- At least cost neutral for an agency
Commercially viable for Payplan
- Payplan willing to support national rollout
Sustainable for AdviceUK
- No significant additional costs (including resourcing) for AdviceUK in running the new model
- No need for significant on-going support from AUK or an independent neutral body
Financial impact on creditors
- Additional income generated from debtors
- More cost effective collection, e.g. earlier write-off of debts
Comparison with Citizens Advice pilot
- Potential financial benefit compares well with that projected from the Citizens Advice pilot
Works in practice for clients
- Options selected by client are more sustainable
Works in practice for advisers
- Adviser time per client case is not increased
Works in practice for agencies
- Overall time spent per client case is not increased
Works in practice for AdviceUK
- Low impact on AdviceUK resourcing, once pilot extra work discounted, so model can run with no additional AdviceUK resources
- Ability to implement model quickly in many more agencies
Works in practice for Payplan
- No undue additional workload
Works in practice for creditors
- Clear, straightforward process for creditors
- Clear contact points for creditors
Could work for the rest of the free-to-client Money Advice sector
- A model that can easily be adapted – so it can easily be implemented in different environments, e.g. with IT systems other than AdvicePro
The Fair Share net is extended
- Additional creditors (priority and secondary) added to Payplan disbursement system
Appropriate options are selected by clients
- All appropriate options offered to clients
Comparison with Citizens Advice pilot
- The processes for this pilot are at least as practical as those offered by the Citizens Advice pilot
Comparison with Christians Against Poverty
Advice to client is not compromised – Advice is:
- in best interest of client
- right route for client
- pilot leading to better solutions for client
- See OFT debt management guidance
The client always chooses the advice route based on the options available.
Agency not compromised, retaining its independence
- Payplan and distribution of Fair Shares does not influence agency’s activities with clients
Compliant with the Equality Act 2010
- No discrimination in the way that clients are treated based on age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, or sex
Clients’ interests are paramount so, for example:
- Clients will have the option not to be included in the pilot, e.g. if they do not want their data transferred to Payplan.
- We will aim to use anonymous data throughout.
- Consent will be sought from clients to be contacted by the evaluator.
Where possible, we will aim to collect anonymous data in its raw, unaggregated form. This will provide the most opportunities for varied analyses.
Where appropriate, comparisons will be made between pilot agency data available before the pilot, and data collected during the pilot. This is explained further in section 4 below.
Reporting on client outcomes is important to both agencies and the SDAP, covering those clients passed to Payplan, as well as those retained by the agency. We will agree with Payplan what data they will feedback to the agencies and to the project.
Evaluation will also consider:
Cultural impact on agencies – An assessment will be made of any possible cultural change in how advice is given, as a result, for example, of advisers needing to let go of cases as they are handed over to Payplan.
Financial viability – The aim of this is to provide hard data to prove or disprove the financial viability of the Fair share contribution model for money advice agencies, Payplan and creditors. This will be tested by:
- Selecting a sample of cases from across the pilot areas that have gone through the Fair Share contribution model, to identify the current and forecast income for advice agencies, Payplan and creditors
- Sensitivity testing this for different levels of re-distribution via the Fair Share model
- Comparing this where possible to baseline data held by the pilot agencies against their current income sources to assess if the Fair Share model is viable and meets the funding needs of money advice agencies
Friends Provident Foundation’s 2009 Report “Funding money advice services” – its conclusions will be taken into account where relevant.
Interim and final evaluation – AdviceUK and an independent evaluator will seek feedback through methods that are likely to include:
- Adviser interviews
- Client interviews
- Analysis of agency financial data (for debt advice services): income and expenditure, direct and indirect costs
- Analysis of other agency data: supervisor time, agency time, appointments per client etc.
- Examining the output of the agency’s quality assurance system
- Examining complaints records and insurance claims
- Adviser questionnaires
- Client feedback questionnaires
- Review of agency procedures
- Focus groups
The report produced at the end of the pilot will be published by Friends Provident Foundation and we plan to disseminate the results widely to interested parties. However, we will maintain anonymity for clients and advisers throughout, and we will acknowledge the overall contribution of the specific agencies involved in the pilot.
Questions and Answers
Q012: For each pilot site, do you know the expected number of clients that are likely to participate? And of these what proportion are likely to go via Payplan and what proportion are likely to go via the “token” payments system?
A: Agencies involved in the pilot range from those with under 300 new clients p.a., up to those with over 2,000 new clients p.a.. In total, the 9 pilot agencies expect approximately 10,000 new clients in 2012. The proportion likely to be referred to Payplan, and those eligible for token payment / low offer debt payment solutions will vary considerably between agency, as they deal with a variety of client bases. Overall, as preliminary estimates, we anticipate up to 20% of clients will be referred to Payplan for a DMP, IVA or TD solution, and 40 – 50% of clients will be eligible for a token payment / low offer solution.
Q013. Do you know how many advisers will work on the pilot for each pilot site?
A: We estimate that across the pilot agencies there will be a total of approximately 50 advisers involved (including volunteers), 10 managers and 30 support and admin staff. The smallest agency will only have about 2 staff involved, with the largest agencies having over 20 staff involved.
Q014. Do you have an estimate of how many different creditors will be part of this pilot?
A: No we do not have an estimate and this will depend on the case mix from the pilot agencies.
Monitoring is the responsibility of the project. It will collect information routinely and systematically during the lifetime of the pilot. Information collected will be stored and made available to the evaluator. It is likely to include:
- project record keeping, such as minutes of meetings and logs of events
- statistical information on take-up of pilot services
- feedback obtained from pilot agencies, advisers and clients
- data from case management and payment disbursement systems
- financial data from agencies
- quality reports, complaints and claims
The approach to monitoring and evaluation is generally in accordance with the approach recommended by Charities Evaluation Services http://www.ces-vol.org.uk .