SAMHSA needed a partner skilled in mental health evaluation for assessing the 988 Lifeline, focusing on potential outcome inequities. This partner needed to design and conduct a national crisis services evaluation, support 988 Lifeline grantees and evaluators with technical assistance, collect and analyze data, and produce reports for stakeholders, including Congress. The evaluation had to encompass 988 crisis centers, mobile teams, receiving and stabilization services and various SAMHSA grantees like tribal nations and community clinics. It also needed to identify disparities in crisis services and strategies to create equity.
SAMHSA partnered with Aptive, a consulting company with extensive experience conducting national suicide prevention evaluations, including the Clay Hunt Act-mandated evaluation and report to Congress.
The Aptive team created an evaluation and data collection plan that includes gathering feedback from local evaluators through virtual meetings and using an evaluation advisory panel (EAP) to provide oversight, guide evaluation design and ensure evaluation activities are shaped by people with lived experiences and those from historically harmed, excluded and marginalized communities.
The evaluation and data collection plan includes all instruments, measures and data collection protocols, as well as plans for data analysis, dissemination, human-subjects protections and the integration of feedback. The plan also describes grantee training and technical assistance (T/TA) to support evaluation participation.
Aptive offers comprehensive T/TA to all grantees of 988 crisis continuum services. Using an evidence-based approach ensures grantees are well-equipped to participate in data collection without feeling
overwhelmed. A team of technical assistance liaisons (TALs) brings expertise in suicide prevention and evaluation methodologies informed by both professional knowledge and personal experiences. An annual briefing keeps TALs updated on the latest developments in the 988 Crisis Services Framework.
The T/TA framework developed by Aptive is grounded in best practices from development, capacity building and implementation science, all underpinned by expertise in suicide prevention. This structure aims to maintain active engagement in the evaluation process, enhance data quality, and promote the use of data by grantees for program improvement, dissemination and advocacy.
The Aptive team developed a scalable, customizable Crisis Services Program Data Center (CSPDC) to facilitate data collection, management and evaluation. This center acts as a hub for all data collection tools and processes, offering flexible options for new data protocols and integrating data from various sources.
Aptive’s expertise extends to maintaining and documenting system changes and ensuring data quality, integrity, security and confidentiality. The team adeptly handles both quantitative and qualitative data, performing tasks that include data validation, identifying anomalies, anonymizing data and ensuring overall data accuracy and consistency.
Aptive writes and edits all documents needed to report on evaluation outcomes, including ad hoc reports, PowerPoint presentations, quarterly reports, the annual evaluation progress report, final summary and evaluation reports, talking points, conference documents, posters and other related material. An example of our monitoring is illustrated in the graph below.
The Aptive Team developed the evaluation project work plan, which serves as the foundation for documenting management processes and strategies to guide steps toward the success of the evaluation and to plan how results are achieved, measured and communicated.
In assessing the evaluation, the Evaluation Advisory Panel provided comprehensive feedback across various studies. The panel’s insights stressed the importance of adopting comprehensive, culturally sensitive and cooperative methods to evaluate and enhance the nation’s crisis care systems.
The T/TA team developed the Evaluation Needs Assessment, which the team disseminated to all grantees to capture feedback and understand their unique challenges. The team continuously develops materials and resources to prepare for upcoming grantee meetings.
In preparing the Office of Management and Budget (OMB) package, the Aptive team is focused on delivering a comprehensive and accurate report that aligns with SAMHSA requirements. This process involves detailed planning and systematic data analysis of evaluation requirements. Our aim is to present a well-rounded view of the performance and impact of the various studies under review.
The Aptive team will attend the annual 988 grantee meeting in Chicago to present the evaluation project to various grantees and stakeholders and solidify understanding of the 988 framework.
[1] Centers for Disease Control and Prevention, National Center for Health Statistics. National Vital Statistics System, Mortality 1999-2020 on CDC WONDER Online Database, released in 2021. Data are from the Multiple Cause of Death Files, 1999-2020, as compiled from data provided by the 57 vital statistics jurisdictions through the Vital Statistics Cooperative Program. Accessed at http://wonder.cdc.gov/ucd-icd10.html on Apr 4, 2024 10:27:27 AM
