Program Evaluation

Review resources on program evaluation to support your efforts in demonstrating program outcomes.

  • Understanding NAS as Chronic Illness (OPQC Learning Session 1/27/14)
    A video from the Ohio Perinatal Quality Collaborative Learning Session on January 27, 2014 about understanding NAS as a chronic illness. Presented by Dr. Ron Abrahams.
  • Maternal opioid treatment: Human Experimental Research (MOTHER) – approach, issues and lessons learn
    Abstract AIMS: The Maternal Opioid Treatment: Human Experimental Research (MOTHER) project, an eight-site randomized, double-blind, double-dummy, flexible-dosing, parallel-group clinical trial is described. This study is the most current--and single most comprehensive--research effort to investigate the safety and efficacy of maternal and prenatal exposure to methadone and buprenorphine. METHODS: The MOTHER study design is outlined, and its basic features are presented. CONCLUSIONS: At least seven important lessons have been learned from the MOTHER study: (i) an interdisciplinary focus improves the design and methods of a randomized clinical trial; (ii) multiple sites in a clinical trial present continuing challenges to the investigative team due to variations in recruitment, patient populations and hospital practices that, in turn, differentially impact recruitment rates, treatment compliance and attrition; (iii) study design and protocols must be flexible in order to meet the unforeseen demands of both research and clinical management; (iv) staff turnover needs to be addressed with a proactive focus on both hiring and training; (v) the implementation of a protocol for the treatment of a particular disorder may identify important ancillary clinical issues worthy of investigation; (vi) timely tracking of data in a multi-site trial is both demanding and unforgiving; and (vii) complex multi-site trials pose unanticipated challenges that complicate the choice of statistical methods, thereby placing added demands on investigators to effectively communicate their results. TRIAL REGISTRATION: ClinicalTrials.gov NCT00271219.
  • Improving care for neonatal abstinence syndrome.
    Abstract BACKGROUND AND OBJECTIVE: Care for neonatal abstinence syndrome (NAS), a postnatal drug withdrawal syndrome, remains variable. We designed and implemented a multicenter quality improvement collaborative for infants with NAS. Our objective was to determine if the collaborative was effective in standardizing hospital policies and improving patient outcomes. METHODS: From 2012 to 2014, data were collected through serial cross-sectional audits of participating centers. Hospitals assessed institutional policies and patient-level data for infants with NAS requiring pharmacotherapy, including length of pharmacologic treatment and length of hospital stay (LOS). Models were fit, clustered according to hospital, to evaluate changes in patient outcomes over time. RESULTS: Among 199 participating centers, the mean number of NAS-focused guidelines increased from 3.7 to 5.1 of a possible 6 (P < .001), with improvements noted in all measured domains. Among infants cared for at participating centers, decreases occurred in median (interquartile range) length of pharmacologic treatment, from 16 days (10 to 27 days) to 15 days (10 to 24 days; P = .02), and LOS from 21 days (14 to 33 days) to 19 days (15 to 28 days; P = .002). In addition, there was a statistically significant decrease in the proportion of infants discharged on medication for NAS, from 39.7% to 26.5% (P = .02). After adjusting for potential confounders, standardized NAS scoring process was associated with shorter LOS (-3.3 days,95% confidence interval, -4.9 to -1.4). CONCLUSIONS: Involvement in a multicenter, multistate quality improvement collaborative focused on infants requiring pharmacologic treatment for NAS was associated with increases in standardizing hospital patient care policies and decreases in health care utilization.
  • Brief interventions for illicit drug use among peripartum women.
    Abstract We review the evidence and identify limitations of the current literature on the effectiveness of brief interventions (=5 intervention sessions) on illicit drug use, treatment enrollment/retention, and pregnancy outcomes among pregnant and postpartum women; and consider this evidence in the context of the broader brief intervention literature. Among 4 published studies identified via systematic review and meeting a priori quality criteria, we found limited, yet promising evidence of the benefit of brief interventions to reduce illicit drug use among postpartum women. Two of the 4 randomized controlled trials tested similar computer-delivered single-session interventions; both demonstrate effects on postpartum drug use. Neither of the 2 randomized controlled trials that assessed treatment use found differences between intervention and control groups. Studies examining brief interventions for smoking and alcohol use among pregnant women, and for illicit drug use in the general adult population, have shown small but statistically significant results of the effectiveness of such interventions. Larger studies, those that examine the effect of assessment alone on illicit drug use, and those that use technology-delivered brief interventions are needed to assess the effectiveness of brief interventions for drug use in the peripartum period.
  • Substance abuse treatment for adolescents: how are family factors related to substance use change?
    From the abstract: The present study examines the association between (1) changes in adolescents' substance use and parenting practices, as measured by adolescent self-disclosure and parental warmth; (2) changes in mothers' mental health and their parenting practices, and (3) parental use of services offered by substance abuse treatment centers, parenting practices and adolescent substance use.
  • Women and addiction: a trauma-informed approach
    From the abstract: This article presents the definition of and principles for gender-responsive services and the Women's Integrated Treatment (WIT) model. This model is based on three foundational theories: relational-cultural theory, addiction theory, and trauma theory. It also recommends gender-responsive, trauma-informed curricula to use for women's and girls' treatment services.
  • The Cannabis Youth Treatment (CYT) Study: main findings from two randomized trials
    From the Abstract: This article presents the main outcome findings from two inter-related randomized trials conducted at four sites to evaluate the effectiveness and cost-effectiveness of five short-term outpatient interventions for adolescents with cannabis use disorders.
  • Differential responsiveness to a parenting intervention for mothers in substance abuse treatment
    From the abstract: This study examines the relationship between levels of psychological distress in substance-dependent mothers and their differential response to a dyadic parent-child intervention.
  • Efficacy of intervention for engaging youth and families into treatment
    From the abstract: This study reports data on the efficacy of Strategic Structural Systems Engagement (SSSE), which is designed to bring hard-to-reach families into treatment. The study also explores variables that may contribute to differential effectiveness.
  • The Evaluation Exchange: A Periodical on Emerging Strategies in Evaluation
    Hosted by the Harvard Family Research Project, The Evaluation Exchange is a periodical that contains new lessons and emerging strategies for evaluating programs and policies, particularly those focused on children, families, and communities.
  • Evaluating Your Program: Consumer-Operated Services
    Provides tools for developing mental health services that are owned and operated by people who have personal experience living with a psychiatric disorder. Offers guidance grounded in evidence-based practices.
  • Online Evaluation Resource Library
    The Online Evaluation Resource Library (OERL) was developed for professionals seeking to design, conduct, document, or review project evaluations. The purpose of this system is to collect and make available evaluation plans, instruments, and reports for NSF projects that can be used as examples by Principal Investigators, project evaluators, and others outside the NSF community as they design proposals and projects. OERL also includes professional development modules that can be used to better understand and utilize the materials made available.
  • Point K: Practical Tools for Planning, Evaluation, and Action at the Innovation Network
    The Point K Learning Center's free tools and resources help nonprofits and funders look inward, so they can achieve a greater outward impact.
  • Developing & Evaluating Programs
    The University of Wisconsin-Extension website offers program development and evaluation resources. This includes downloadable guides and quick tips.
  • Conjoint versus one-person family therapy
    This article compares the effectiveness of Conjoint versus one-person family therapy.
  • Family Behavior Therapy for Substance Abuse and Other Associated Problems
    This article describes Family Behavior Therapy and its applications.
  • Building a Model Program for Substance-Exposed Newborns and Their Families
    This article discusses building the model program from needs assessment, intervention and evaluation to Consultation.
  • The Trauma Recovery and Empowerment Model for Women
    This article discusses conceptual and practical issues in a group intervention for women.
  • Brief Strategic Family Therapy Versus Treatment as Usual
    This article discusses the results of a multisite randomized trial for substance using adolescents.
  • Behavioral couples therapy for alcoholism and drug abuse
    This article explores behavioral couples therapy and its outcomes, i.e. abstinence from substances, domestic violence, and cost outcomes.
  • Neonatal abstinence syndrome after methadone or buprenorphine exposure
    This article gives a brief background on methadone, buprenorphine, and neonatal abstinence syndrome (NAS). It then describes the study which examines the effects of medications on NAS.
  • The role of the family in preventing and intervening with substance use and misuse
    This article is a comprehensive review of family interventions, with a focus on young people.
  • Mothering from the Inside Out
    This article provides an in-depth look at mentalization-based therapy for mothers who use substances.
  • Treating Women Who Are Pregnant and Parenting for Opioid Use Disorder
    This article reviews the literature on opioid treatment in pregnant women as well as the concurrent care of their infants and children.
  • The link between substance abuse and posttraumatic stress disorder in women
    This article reviews the research of the association between PTSD and substance abuse in women.
  • Methods for Evaluating Comprehensive Community Initiatives
    This online toolkit covers the following topics: Section 1. Measuring Success: Evaluating Comprehensive Community Initiatives Section 2. Gathering Information: Monitoring Your Progress Section 3. Rating Community Goals Section 4. Rating Member Satisfaction Section 5. Constituent Survey of Outcomes: Ratings of Importance Section 6. Reaching Your Goals: The Goal Attainment Report Section 7. Behavioral Surveys Section 8. Conducting Interviews with Key Participants to Analyze Critical Events Section 9. Gathering and Using Community-Level Indicators Section 10. Community-Level Indicators: Some Examples
  • Introduction to Evaluation
    This online toolkit covers: 1) A Framework for Program Evaluation: A Gateway to Tools; 2) Community-based Participatory Research; 3) Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?; 4) Choosing Evaluators; 5) Developing an Evaluation Plan; and 6) Participatory Evaluation.
  • Choosing Wisely
    This organization aims to promote conversations between clinicians and their patients. Their goal is that these conversations not only occur regularly, but are also supported by evidence, are free from harm, and are not repetitive.
  • U.S. Selected Practice Recommendations for Contraceptive Use, 2016
    This report by the CDC discusses contraceptives in complex and controversial situations. The recommendations in this report are intended to serve as a source of clinical guidance for health care providers and provide evidence-based guidance to reduce medical barriers to contraception access and use.
  • Father-Friendliness Organizational Self-Assessment and Planning Tool
    This self-assessment package, developed for Head Start and other family service programs, provides tools for organizations to assess their readiness to provide services to fathers and father figures.
  • NREPP Learning Center
    This site is designed to provide resources for developing, implementing, and sustaining culture-centered and evidence-based programs and practices. Learning Center tools provide practical support for using evidence-based programs and practices to improve the behavioral health of clients, family members, and communities.
  • Evaluating the Initiative Toolkit
    This toolkit aids in developing an evaluation of a community program or initiative.
  • Seeking Safety
    This treatment manual is for treating patients with co-occurring PTSD and substance abuse.