SAMHSA, in collaboration with other federal agencies, is responsible for determining the effectiveness of interventions and programs aimed at tackling the public mental health crisis, including suicide. By federal law, SAMHSA must evaluate the Zero Suicide Program and report findings to Congress. This evaluation assesses the program’s implementation, outcomes and impact on enhancing continuous quality improvement. The evaluation will determine the program’s success in reducing suicide attempts and death and guide further program development.
To meet this requirement, SAMHSA has undertaken a comprehensive impact evaluation. The evaluation involves coordinating, gathering, analyzing and synthesizing quantitative and qualitative data from various grantees. SAMHSA also provides training and technical assistance (T/TA) to up to 25 grantees, supporting them in fulfilling the evaluation criteria and helping them complete the overall evaluation and subsequent reporting.
SAMHSA partnered with Aptive, a government consulting company with extensive experience evaluating mental health and suicide prevention programs and an intimate knowledge of the Zero Suicide program and framework, to help meet the program evaluation mandate. Aptive’s experienced mental health subject matter experts, data analysts and evaluation experts use a variety of qualitative and quantitative methods to help government agencies and other organizations measure performance to optimize their value to patients and other stakeholders. The Aptive team leverages lessons learned from the initial Zero Suicide evaluation designed and launched in 2017[1] and uses existing data collection instruments to provide SAMHSA with cost-conscious efficiencies.
[1] Labouliere, C. D., Vasan, P., Kramer, A., Brown, G., Green, K., Rahman, M., … & Stanley, B. (2018). “Zero Suicide”–A model for reducing suicide in United States behavioral healthcare. Suicidologi, 23(1), 22.
Our approach involves active participants, including stakeholders, grantees, consumers, people with lived experience and an expert evaluation panel. We provide grantees with T/TA throughout the evaluation to ensure all collected data is accurate. We also employ continuous process improvement throughout the project lifecycle and apply health equity principles to all project tasks.
The Aptive team designed four different studies to collect and analyze the necessary implementation, outcome and impact data: a systems change study, workforce study, consumer experience study and an impact study. Using an innovative, multi-methods approach, the team maximized the quantity of data collected without compromising data quality or unnecessarily burdening grantees.
Our project methodology includes best practices, such as Project Management Body of Knowledge (PMBOK)-based project planning and execution; Agile and Plan, Do, Check and Adjust principles; Lean Six Sigma; and Capability Maturity Model Integration compliant processes. This methodology is informed by decades of experience managing more than 100 health-related programs for the federal government, as well as the expertise of our executive management, program leadership and support staff.
Aptive recruited a team of experts to provide additional advice and commentary throughout the life of the project. Consultants who make up the Expert Evaluation Panel offer input into implementing the evaluation plan, analyzing data and presenting results for reports and other dissemination activities. Panel experts have a range of backgrounds and experience, including suicide prevention, treatment and prevention research, suicide interventions in health care settings, Veteran and service member suicide prevention and prevention with vulnerable populations, including American Indian/Alaskan Native and multicultural communities.
Aptive’s T/TA team lead works closely with a team of technical assistance liaisons (TALs) and our Community of Practice lead and T/TA advisor to ensure all Zero Suicide grantees receive high-quality evaluation T/TA. Our three levels of T/TA help grantees at varying stages of Zero Suicide program implementation participate in the evaluation. We rely on TALs, a monthly e-newsletter and updates to the data platform to consistently disseminate opportunities for T/TA.
Aptive collects data and oversees submission to the data repository, manages and documents database changes and implements protocols to ensure data quality, security, confidentiality and integrity. The team conducts data collection, analysis and reporting for baseline, aggregated and comparison data; prepares analytic data files; writes statistical programs; and statistically analyzes evaluation data. We conduct data analyses for each subject, site and analysis group as specified in the Evaluation and Data Collection Plan we created and aggregate and analyze data from agreed-upon groupings of respondents.
The Aptive team has hit several milestones that set the foundation for the Zero Suicide evaluation, including the work plan, evaluation plan and meetings with evaluation experts and Zero Suicide grantees. The team conducted one meeting with evaluation experts and two meetings with Zero Suicide grantees, which provided comprehensive and critical feedback to further refine and strengthen the evaluation plan and data instruments and prepare for developing the OMB clearance package.
The team will produce ad hoc reports and presentation products, local evaluation summary reports, quarterly and annual progress reports, a local evaluation summary report, the final Zero Suicide evaluation report, the draft Zero Suicide Report to Congress and the final Zero Suicide evaluation report. We also:
Hosted and facilitated expert and local evaluator meetings to gather feedback for the evaluation plan and data collection instruments
Synthesized evaluation advisory panel study feedback (systems change study, workforce study and consumer experience study)
Developed and disseminated the valuation needs assessment for grantee feedback and continue to create the timeline for T/TA activities
Draft and gather feedback for submission of the OMB clearance package
Attend Annual Zero Suicide grantee meeting
