The Clay Hunt Suicide Prevention for American Veterans Act, enacted in February 2015, calls for annual, independent, third-party evaluations of the impact of the Department of Veterans Affairs (VA) mental health care and suicide prevention programs. The evaluations began in 2018 and results are reported to the VA Secretary and Congress. The law also calls for interim reports.
VA’s Veterans Health Administration (VHA) required a partner to produce and submit the FY2021 (Year 3) Report to Congress, containing the most recent evaluations of VHA mental health program efficacy, cost effectiveness and Veteran satisfaction. The analyses refine and build on the previous mental health and suicide prevention evaluations of eight programs. In addition, VHA needed a review of the extent to which the VA Clinical Practice Guidelines for Opioid Therapy affect VA’s suicide prevention mission.
We did our research. Aptive reviewed VA’s mental health handbooks and suicide prevention program guide. This review helped us develop informed research questions to gain a thorough understanding of each mental health program, including:
- Acute Inpatient Mental Health Services (AIMHS)
- Intensive Community Mental Health Recovery (ICMHR)
- Psychosocial Rehabilitation and Recovery Centers (PRRC)
- Residential Rehabilitation Treatment Programs (RRTP)
- Substance Use Disorder (SUD)
- Suicide Prevention Program Guide
Following our review, we drafted a set of five initial questions to present to each program lead. Questions addressed program goals, effectiveness, patient access to care, relevant databases or metrics that may be leveraged for the evaluation and any additional information that could shed light on nuances and objectives. We also inquired about any notable changes in 2020 resulting from the COVID-19 pandemic.
We met with experts. We scheduled and conducted introductory meetings with VA program and evaluation leads to ask our questions, collect information about each program and build strong and consistent working relationships as we conducted the evaluation.
We created an evaluation method and analysis plan. We drafted and submitted a comprehensive evaluation methodology and analysis plan outlining programs to be evaluated, research questions, metrics, data sources and proposed analytic methods.
We nailed down logistics. We established data transfer and warehousing logistics with VA so that we could properly access and utilize data for the evaluation. We guided the development and implementation of written data handling policy and standard operating procedures for the data analysis and evaluation teams. These procedures helped the teams properly secure VA data in compliance with VA and federal rules, regulations and best practices. We also created a system security plan to establish the processes needed to obtain an authority to operate in the FedRamp data analytics environment.
We conducted subject matter expert interviews. We organized a panel of five subject matter experts in mental health and suicide prevention efforts and interviewed an additional 24 experts on these initiatives. This allowed us to gather and determine 33 best practices for recommendation to VHA on current and future improvements to consider in care delivery and treatment for Veterans and their families.
Data collection and tracking – We continue to collect data from up to nine different VHA information sources. Our team is conducting proper intake, processing and tracking of received data, and preparing lists and summaries of data contents and their variables to undergo accurate analysis.
Report creation – We created and submitted the following reports: weekly status reports, monthly progress reports, interim reports, a Summary Report of Best Practices, Analyses of Program Effectiveness and Opioid Data, and the Annual Report to Congress.