Jennifer L. Magnabosco and Ronald W. Manderscheid
Outcomes Measurement in the Human Services, 2nd Edition
Cross-Cutting Issues and Methods in the Era of Health Reform
In 1997, the year the first edition of Outcomes Measurement in the Human Services was published, the United States had just begun to implement more systematic accountability of the human services through legislation (such as the Government Performance and Results Act), more contemporary approaches to human services administration that were geared toward performance measurement (such as total quality management), and more stringent approaches to fiscal accountability through evolving strategies such as managed care. At the time of this second edition’s writing, our nation has once again just completed a thorough examination of how our health, mental health, and child and family services and systems are to be delivered and managed. Having just recently witnessed the passing of national health care reform, mental health parity, and new procedures to overhaul state child and welfare accountability systems, citizens in the United States are living in an unprecedented time of systemic change within our human services. Such historic change again directs much attention to human service outcomes and their measurement, and in the 21st century, new contexts for care such as prevention, consumer-, and recovery-driven (please see chapter 16 for a general definition) and strengthsbased (approaches to recovery that shift from a problem-focused, targeting of deficiencies perspective, to a focus on “elevating” and “magnifying strengths” perspective) care (Seligman, 2002; Seligman & Csikszentmihalyi, 2000).
This volume is intended to be in sync with information presented in the first edition. Both volumes
- Bring together prominent thinkers, researchers, practitioners, administrators, . . . policymakers [and consumers] to discuss issues relevant to the current impetus for accountability and measuring outcomes in health, mental and behavioral health, and child and family services
- Include chapters written by human service specialists with expertise in practice, policy and research, and advocacy
- Give students and professionals information to begin creating a comprehensive framework and to better understand the implications of outcomes measurement for the human services.
Many authors have contributed chapters to both editions, and in this edition, several authors again provide social work perspectives on outcomes measurement in the three main human service areas covered: health, mental/ behavioral health, and child and family services. This volume’s design and content are also grounded in a set of guiding questions similar to those of the first edition:
- Changed context and response since 1997: How have the contexts of outcomes measurement changed? How has outcomes measurement been usefully reconceptualized? What are the legislative and public policy contexts of outcomes measurement? Have they changed? If so, how? How have outcomes been addressed to meet legislative and public policy demands?
- New and established approaches to outcomes measurement since 1997: What approaches to outcomes measurement are being promulgated, and how have they changed since 1997? What can be said about the reliability, validity, and quality of existing approaches to outcomes measurement, as well as their relevance for social work and other human services?
- Future Implications: What are the implications of the increasing attention to outcomes measurement for social programs and for practitioners, policymakers, advocates, consumers, and researchers? What are the implications for future research, practice, policy, and advocacy?
Keeping these guiding questions in mind, authors in the second edition of Outcomes Measurement in the Human Services present an updated discussion of issues and methods in outcomes measurement (Mullen & Magnabosco, 1997). Where possible, and fitting, discussion regarding the effect that demands for accountability have had and will continue to have on social work practice is also included.
The Book’s Organization
This introduction sets the overall context for the book, summarizing each chapter and providing a descriptor of its content in italics. Parts I through IV of the book model the first edition’s overall structure. Part I addresses general overarching issues and cross-cutting themes of outcomes measurement, and parts II, III, and IV examine outcomes measurement in three select fields of human services: health, mental and behavioral health, and child and family services. Whereas one section in the first edition featured practice-research examples, in this edition we provide three such examples, one in each of the fields of practice, to illustrate the development and/or application of outcomes measurement in real-life human service scenarios. Part V is a new section that features particular areas in the human services that cut across the three main fields of practice featured and are timely and representative of important domains of interest in the 21st century. We close the book with concluding comments and recommendations for future action.
Part I: Overarching Issues and Methods
Part I “examines the overarching issues and methods that apply to all human service disciplines” (Mullen & Magnabosco, 1997, p. xxi). The themes found in these beginning chapters are echoed and particularly contextualized by chapters in parts II–V.
Orientation to Outcomes and Performance Measurement (Chapter 2)
Hatry updates the current status of outcomes and performance measurement in the public and private sectors, discussing major recent activities that have shaped the direction of such measurement in the United States and internationally. In this chapter (as in the first edition), he distinguishes among types of outcomes measurement and approaches and discusses the need for human service organizations (HSOs) to incorporate outcomes management in everyday organizational activities. He questions whether the typically collected outcomes and performance data are valid and comprehensive and raises concerns about how such data are being used. He advocates for the need to balance and integrate regular outcomes measurement and program evaluation, especially in light of the need to address disparities (inequities), and discusses various strategies being used to effectively meet accountability demands from funders, government, specific communities, and the public at large.
Qualitative and Quantitative Approaches and Outcomes Evaluation (Chapters 3 and 4)
The next two chapters provide perspectives on qualitative and quantitative approaches to outcomes measurement and outcomes evaluation. This dual perspective echoes Hatry, highlighting the necessity of both approaches and their unique contributions to the development, implementation, monitoring, and provision of outcomes and performance measurement, as well as to accountability, practice, and research purposes. In chapter 3, Patton and Gornick discuss the development, trends, and impact that qualitative data—specifically, qualitative outcomes evaluations—have had in the outcomes measurement movement and in meeting demands for accountability in the human services. They illustrate the value that qualitative methods offer in a case example evaluation of a residence for persons with long-term alcohol and drug use conditions. Patton and Gornick conclude with comments on some of the major issues in the future of outcomes evaluation, including the need to develop a common language for types of evaluation and outcome measures, as well as how increasingly important concepts, such as strengths-based care and prevention, require understanding that there may be “different meanings in different contexts.”
In chapter 4, Yates provides a commentary on quantitative approaches to outcomes measurement and evaluation. On the basis of his 35 years researching and evaluating HSO programs, he reflects on issues that are common to measuring outcomes quantitatively across HSOs, such as the idea that “reliability and validity of quantitative outcomes are not just based on science anymore,” and the fact that the process of selecting and quantifying outcomes for, and by, funders is not standardized. He then discusses implications for outcomes measurement in this era of change, and offers the RAPOA (Resource-Activity- Process-Outcome Analysis) model as a tool that can help HSOs measure the range of outcomes related to improving services and systems.
Controversies in Evidence-based Practice and Outcomes (Chapter 5)
Thyer discusses controversies in the measurement of outcomes and evidencebased practice (EBP). He shares views that are critically needed during this time of turbulent change in the human services: He explains why it is important to pause and reconsider the fact that EBP is really a five-step process, and that EBP “does not tell clinicians and others involved with an EBP process what to do.” He thoroughly discusses the relationship between the EBP process and outcomes measurement, how to locate and assess credible evidence, and various stakeholder approaches to outcomes measurement, as well as the implications different perspectives have for human services and EBP. He concludes with thoughts on how to balance approaches to outcomes measurement and EBPs and ends on an empowering note to consumer and advocacy groups, suggesting that they become more involved with promoting EBPs in service and government organizations.
A Process for “Getting to Outcomes” (Chapter 6)
Acosta and Chinman describe a method for HSOs to build capacity for outcomes called “Getting to Outcomes” (GTO). The authors discuss how the 10-step GTO model is “more than a process—[it] is an intervention that builds the capacity of users to use outcome data for continuous quality improvement, not just performance measurement.” They illustrate the use of GTO in a current project designed to implement positive youth development programs by an adolescent coalition in Maine. They conclude with a discussion about barriers associated with measuring outcomes in HSOs and how GTO can effectively help navigate the development of capacity and outcomes.
Leadership and Outcomes Assessment (Chapter 7)
Packard and Beinecke discuss leadership development in the human services and the implications leadership can have for outcomes at multiple levels. They make it clear that human services leadership is facing a serious challenge over the next decade as many experienced human service professionals retire. They also review the main models of leadership, tenets of leadership versus management, organizational change, and pertinent studies that have shown links between leadership and outcomes in the human services. Packard and Beinecke offer a conceptual model for organizational change that can be used as a framework to develop and measure progress toward outcomes-based cultures in HSOs. They conclude with suggestions for promoting “change leaders” and outcomes measurement systems and cultures in HSOs.
Part II: Outcomes Measurement in Health
In 1997, Mullen and Magnabosco wrote, “perhaps no field of practice of human service practice has been more affected by outcomes measurement than health.” (p. xxviii) This remains true today.
Integrated Service Delivery (Chapter 8)
Bartlett describes a current solution to integrated healthcare—the patientcentered medical home—and the implications it has had during health care reform and for the United States in general. Chronicling the history of primary care delivery in this country and its relation to current health care reform tenets, Bartlett describes the challenges to providing cost-effective and needed care, and to measuring integrated health care outcomes between primary care and other sectors, especially mental health and substance use. He stresses the unmistakable need to integrate health and mental health and describes several strategies that may help to bring us closer to reasonable outcomes, such as implementing the “use of brief, valid, and reliable tools” (like Zabora in chapter 12 and Corcoran and Hozak in chapter 15) and “thinking globally but acting locally” (Geddes, 1915)—that is, measuring health outcomes at all levels, with an eye toward improved population health.
Patient-reported Measures (Chapter 9)
Cherepanov and Hays provide a comprehensive discussion of measures that are critical to monitoring health and care: patient-reported outcome measures, health-related quality-of-life measures, and patient evaluations of care. The authors discuss the various types of measures in detail and their applications to practice; in particular, they discuss the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Program to illustrate patient evaluation of care measures. Like other authors in the book, Cherepanov and Hays chronicle the changes made in measurement and note the available resources. In their conclusion, they predict that patient-reported outcome measures may gain greater significance during health care reform.
Assessing Services for an Aging Population (Chapter 10)
Berkman and Kaplan discuss one of the most pressing health concerns the United States will face in the next decade: our aging population. They describe the characteristics of this population and how they will affect the context of health care delivery, regardless of health care reform. Berkman and Kaplan review the state of outcomes measurement in aging and provide a range of instruments and measures that are especially relevant and useful. Like other authors in the book, they highlight the role that social workers will have as the face of health care delivery is changed by this population’s needs and by health care reform.
Complementary and Alternative Medicine (CAM) (Chapter 11)
Coulter and Horsan discuss an aspect of health care that is widespread but not routinely included in mainstream discussions: the use of CAM and integrative medicine (IM). The authors explain the nature and modalities (e.g., acupuncture, chiropractic medicine) of CAM and IM, their paradigms, the controversies and challenges associated with integrating CAM and IM, and the challenges associated with conducting research and outcomes measurement. They conclude with recommendations and comments about useful approaches (e.g., observational studies) and theoretical models (e.g., systems theory), the principles upon which CAM outcomes measurement should be based (e.g., “elevate the notion of person-centered care and patient-centered care”), and an agenda for outcomes measurement in CAM that can be used for comparative effectiveness research and other health care reform efforts.
Research to Practice Example: Chronic Care and Psychological Distress (Chapter 12)
Zabora illustrates the importance of using appropriate and comprehensive screening and outcomes assessment tools to determine the range of needs chronic care patients, especially oncology patients, have during the course of care and over the longer term. Zabora focuses on screening and assessments for psychological distress—one of the most serious effects of chronic illness— in the context of hospital settings. This focus is of particular note because screening for such distress often goes unperformed, largely because of a lack of comprehensive tools and measures. He concludes with a discussion of barriers to such assessments, how particular members of the health care team—such as social workers—are better poised to take on this responsibility, and how such instruments and measures can be especially useful in health care reform efforts that seek quick and valid approaches.
Part III: Outcomes Measurement in Mental and Behavioral Health
The contributors to part III examine mental and behavioral health services outcomes measurement. As in the first edition, the terms mental health and behavioral health are sometimes used interchangeably when discussing outcomes measurement. In 1997, the term behavioral health typically was used to designate services that applied to people with substance abuse and mental health problems in a managed care context (Freeman & Trabin, 1994; Mullen & Magnabosco, 1997). Today, behavioral health carries a similar meaning, with two differences: First, substance use is used in place of substance abuse. Second, the overall meaning of substance use is not determined as much by the managed care context. Rather, it is informed by the fact that consumers, and the wide range of stakeholders in the fields of mental health and substance use, have come to accept a broader and more integrated approach to the spectrum of mental health and substance use care. Although the use of the terms “substance use” and “substance abuse” largely depend on ideology, practice, and societal context, the former represents recognition that persons with this disorder have a real illness and that the issues associated with this illness are multidimensional, taking into account the holistic individual, not just his or her drug using or drug “abusing.” Hence, in the United States, we have more readily come to use the term substance use to “connote measurability or an emphasis on behavioral functioning” (Mullen & Magnabosco, 1997).
Transformation and Outcomes (Chapter 13)
Kelly examines mental health policy in the United States, factors leading to the recent emphasis on mental health transformation or reform, and recommendations for what is needed to transform the country’s mental health system into an outcomes-oriented culture of care. Based on his recent book, this chapter, like Thyer’s, provides the author’s views on the use of EBPs and how various instruments can support the measurement of service effectiveness, as well as different stakeholder views on the use of EBPs and outcomes measurement. He concludes with a “blueprint for change” that can help the United States transform to a national outcomes-based mental health system, including the caution that we need to correctly implement “outcome data for all parties” that help guarantee community-based recovery for persons with mental illness.
Integrated Measures for Mental and Substance Use Care (Chapter 14)
Siemianowski and Kirk provide an overview of identity and integration issues for the field of mental health and substance use generally, the implications of these issues for integrated outcomes measurement for the two fields, and the specific implications for persons with co-occurring disorders (persons with mental health and substance use conditions). They use an example of multilevel integration that has occurred in the state of Connecticut to illustrate how transformation in organizational culture, strong leadership, and working with a variety of stakeholders—especially consumers—have resulted in a system that is driven by recovery, outcomes, and performance measurement. They conclude with recommendations for outcomes that can reflect an integrated system of mental health and substance use care under health care reform, and that can support integration with primary care, so that those in need can receive the comprehensive services required for their recovery.
Emergence of Rapid Assessment Instruments (RAIs) (Chapter 15)
This chapter updates Corcoran’s 1997 chapter on RAIs with expanded commentary with his colleague Hozak. Just as Corcoran discussed the importance of RAIs as an important extension of outcomes measurement during managed care and in social work clinical practice, Corcoran and Hozak describe how RAIs can be useful in health care reform today. Although these tools have been in use for a long time, in routine practice they have not superseded reliance on wisdom, clinical judgments, and intuition in decision making. Corcoran and Hozack believe this will continue. However, they think that RAIs can provide evidence about medical necessity and be used as quick and valid measures of treatment outcomes, or as estimations of the clinical or real-life significance of usual treatment and/or interventions. They conclude with cautions about properly linking different populations to standardized and normed instruments and measures, as well as endorse appropriate uses of RAIs to meet various clinical and reform goals.
Consumer-operated Service Programs (COSPs) (Chapter 16)
Campbell describes the evolution of the consumer movement since the late 1990s, with specific emphasis on contributions of consumers, consumer researchers, and other stakeholders in the study and integration of COSPs within the continuum of behavioral health care in the United States. She describes the range of COSP service models and highlights salient research that has investigated their effectiveness, including results from the only national survey on COSP services, completed in 2002. Much of this chapter also focuses on rating the evidence of COSP studies—in the form of a miniresearch synthesis—to question the appropriateness and types of evidencebased lenses through which to assess COSPs, and to set directions for future research and outcomes measurement in this area.
Research to Practice Example: Assertive Community Treatment Measures (Chapter 17)
Teague and Monroe-DeVita describe the evolution of measurement for one EBP—Assertive Community Treatment (ACT)—and provide an example of a live project that is testing refined aspects of ACT to improve service quality and outcomes for persons with severe mental illness. This example illustrates the relationship between fidelity, quality, outcomes, and effectiveness of ACT services, concluding with empowering comments—partially inspired by Kelly’s arguments about mental health system transformation—regarding the need to promote outcomes measurement, quality improvement, and outcomes-based culture.
Part IV: Outcomes Measurement in Child and Family Services
Part IV examines the common and unique aspects of outcomes measurement in child and family services. Although the move toward outcomes measurement has been most evident in health and mental and behavioral health since 1997, the field of child and family services has been making great strides. Child and family services share many of the same contexts that have shaped outcomes measurement in other human service fields. The chapters in this section speak to these commonalities and, at the same time, highlight some of the unique measurement issues faced by child and family services.
Multilevel Measurement Approaches in Children’s Programs (Chapter 18)
Ethridge and colleagues describe how history, current policy changes, and systemwide reforms in the field of child and family services have shaped existing and emerging approaches to measuring outcomes and performance measures in this field of practice. They discuss outcomes and performance measurement domains (which are also discussed in chapter 20) and the challenges associated with using certain methodologies, measures, and research in daily management, practice, policy, and advocacy. They illustrate how these challenges are being addressed in an outcomes measurement process that was developed and implemented across child and family services in Kent County, Michigan, in the United States. The authors conclude with recommendations for improving the delivery and outcomes of child and family services, including the need for cross-system collaboration and the development of infrastructures and working practices that enhance the sharing of information across human service sectors.
Linking Costs and Outcomes for Children (Chapter 19)
Kilburn describes the challenges and solutions to measuring cost and outcomes of childhood services and programs and family well-being. She discusses why it is important to incorporate cost and outcomes analysis into human services research and decision making, describing the different methods by which this is possible: cost analysis, cost-effectiveness analysis, and cost–benefit analysis. Kilburn illustrates the use of these methods with examples from the child services literature, but she makes it clear in her discussion that these methods are usable in any human services domain. She concludes with implications for how cost and outcomes analysis can enhance policy, research, and practice in the child and family services field.
Research to Practice Example: Standardization of Process and Outcomes in Child Service and Welfare Evaluations (Chapter 20)
McGowan and Walsh discuss lessons learned from child services and welfare evaluations and the push for standardization of process and outcomes (for safety, permanency, and child and family well-being) at the state level. They describe how the new results-oriented approach in accountability first came into play in 2000, its subsequent statewide assessment review process, and how this movement is helping to address some of the “methodological shortcomings” (Wulcyzn, 1997) in outcomes measurement and promoting new approaches to child welfare practice.
Part V: Special Topics in the Human Services
Part V brings together four topical areas not typically represented in general discussions of outcomes measurement in the human services that can be helpful as we begin to think more holistically about outcomes measurement, performance measurement, and outcomes evaluation across the human services.
Transformation of the Veterans Health Administration (VHA) (Chapter 21)
Francis discusses lessons learned from the VHA during its transformation into a Cabinet-level government agency determined to incorporate innovative and evidence-based clinical, administrative, and management practices to provide better health care to U.S. veterans. He describes how the VHA evolved its outcomes and performance measurement system and its integrated electronic health record, as well as how instituting a systemwide Quality Enhancement Research Initiative has improved the implementation of EBPs and performance measurement in everyday practice. He concludes by noting the importance of accountable leadership in forging change in measurement and quality-improvement systems and why this process, which includes the determination of measures, needs to be more patient centered.
Outcomes in Criminal Justice (Chapter 22)
Turner provides an overview of outcomes measurement in the U.S. criminal justice system and the challenges associated with measuring outcomes at multiple levels: individual, agency, and state. She bases her discussion in two main outcomes measurement domains: crime rates and recidivism. She discusses how new measurement systems, such as CompStat, have helped to improve correctional institutions’ internal processes and outcomes, and in turn, community outcomes in both of these domains. Turner concludes with some lessons learned from the field of criminal justice that other human services may find useful in their quest to become more automated and collaborative with the various sectors involved in providing comprehensive care.
Outcomes in Faith-based Organizations (FBOs) (Chapter 23)
Flory and colleagues discuss the role that FBOs and services play in the United States and how more formalized programming and funding of such services have evolved over the past 15 years. They discuss the research upon which effectiveness evaluations have been based and directions for research that can help to better measure important components of such services, such as how the “faith factor” may influence clients and play a special role in measurement. They conclude by suggesting the development of a typology and language for FBOs and of forums that can help to better share lessons learned.
International View of Outcomes (Chapter 24)
Ganju provides an international perspective on health, mental health, and social outcomes. Like several other authors, he highlights the importance of linking several life domains and their associated outcomes together to better understand how we can improve the quality of life globally. In particular, he discusses a conundrum: Although a “natural imperative” exists that links mental health and substance use to broader societal and community goals (such as reduction in poverty, increased employment), we have yet to comfortably and effectively recognize, prioritize, and equalize mental health and substance use outcomes when evaluating global welfare. Ganju discusses the tensions underlying this and offers recommendations to help promote an international outcomes agenda for health, mental health, and substance use.
Since 1997, human services has grown in its “wealth” (Mullen & Magnabosco, 1997) of approaches, instruments, and types of outcomes and performance measures, as well as its capacity to integrate research and measurement processes into everyday clinical and administrative practice, policymaking, and advocacy. Regardless, much still needs to be done to critically assess the field of outcomes measurement and how effective its evidence bases are in meeting the myriad demands that clients, collaborators, communities, funders, and other stakeholders place on HSOs. This volume brings together an array of information that can be used as a starting point toward that end. We will return to this theme in our conclusion. We hope your reading of this volume will provide a contemporary understanding of some of the most salient crosscutting and field-specific, issues, methods, and tools in outcomes measurement in the human services today.
Geddes, P. (1915). Cities in evolution. London: Williams.
Freeman, M.A. & Trabin, T. (1994). Managed behavioral healthcare: History, models, key issues and future course. Rockville, MD: U.S. Center for Mental Health Services.
Mullen, E.J., & Magnabosco, J. L. (Eds.). (1997). Outcomes measurement in the human services: Cross-cutting issues and methods. Washington, DC: NASW Press.
Seligman, M., & Csikszentmihalyi, M. (2000). Positive psychology: An introduction. American Psychologist, 55, 5–14.
Seligman, M. E. P. (2002). Positive psychology, positive prevention, and positive therapy. In C. R. Snyder & S. J. Lopez (Eds.), The handbook of positive psychology (pp. 3–12). New York: Oxford University Press.
Wulczyn, F. H. (1997). Methodological considerations in outcomes measurement in family and child welfare. In E. J. Mullen. & J. L. Magnabosco (Eds.), Outcomes measurement in the human services: Cross-cutting issues and methods. Washington, DC: NASW Press.