4. Evaluation Policy and Learning

The Colorado Governor’s Office of State Planning and Budgeting (OSPB) is home to many research and evidence-based policy initiatives and publishes the results of evaluations funded by the grant program on its website each year. OSPB also publishes annual guidance to state departments requiring the use of data and evidence in the state’s budget process. It also provides definitions for key research and evaluation terms and “encourages agencies to use the evidence continuum outside the budget process as a means to evaluate programming internally, and to guide discussions with OSPB analysts and internal and external stakeholders.”  With the large influx of the America Rescue Plan Act (ARP), OSPB has applied the evidence continuum as a cornerstone of its decision-making process to ensure the strongest outcomes for residents.

The Colorado Department of Higher Education (CDHE) developed a research agenda to inform education and training policies and programs. Data sharing is required by Colorado law among the Department of Education, Department of Labor, and Department of Economic Development, which collectively generate research covering the education-to-workforce continuum.

Several projects funded through the American Rescue Plan Act (ARP) plan include program evaluations. The Colorado Evaluation and Action Lab (Colorado Lab) partnered with the Governor’s Office of State Planning and Budgeting to determine appropriate evaluation strategies for ARPA-funded projects with a number of state agencies, including the Departments of Higher Education, Health Care Policy and Financing, Human Services, Labor and Employment, and the Department of Early Childhood.

In 2021, the Executive Board for P20 WIN, the state longitudinal data system, reviewed and approved a learning agenda to cover the priority areas for analysis across the ten agencies that are part of P20 WIN. The learning agenda focuses on cross-cutting topics and agencies are working to develop actionable data requests based on the topics in the agenda.

The Connecticut State Data Plan includes priorities for evaluation, including increased tracking of evidence and impact to deliver more efficient and coordinated services, and evaluation practices, such as use of ‘mixed methods.’

Consistent with the recommendation made in the 2018 GEAR Annual Report, the First State Quality Improvement Fund (FSQIF) was reinstated in Fiscal Year 2020 as a line item in the Department of Human Resources (DHR) budget and funded at $150,000 per year. The DHR Training and HR Solutions and GEAR developed the Continuous Improvement Practitioner (CIP) as a joint venture and partnership to build project management, Lean process improvement, and change management skills of State employees. The resulting CIP training program has graduated 52 practitioners to date with an additional cohort of 18 presently in training and another cohort planned for spring 2023. All CIP graduates subsequently join the GEAR Field Team. DHR has submitted the Fiscal Year 2024 budget request for an additional $200,000, endorsed by GEAR, to provide two annual cohorts, support the expansion of CIP, and implement project and process leadership training. The GEAR team itself is driving the broad adoption of standard strategic planning processes, the use of metrics to guide resource allocation decisions, and the implementation of a culture of continuous process improvement to enhance state programs. Specific ideas such as the “Continuous Improvement Cycle” and individual program updates can be found in the GEAR Reports & Publications page.

The Government Evaluation Act of 1995 establishes a system for periodic review of the executive branch agencies and independent agencies of state government to evaluate their efficacy and performance. Within the act, agencies are required to demonstrate their goals, how they are meeting them, and the corrective measures they are taking in the event they are unfulfilled. Moreover, the Office of Program Evaluation and Government Accountability also conducts independent reviews of state agencies focusing on effectiveness.

A 2010 Maryland law established the Maryland Longitudinal Data System Center as an independent agency to bring together education and workforce data from the Maryland Higher Education Commission, the Maryland State Department of Education, and the Maryland Department of Labor. In partnership with the University of Maryland System and under the guidance of a governing board, the Center’s 12 staff members produce a variety of publications, including a learning agenda, about student performance at the elementary, secondary, and postsecondary level to improve the state’s education system and guide decision-makers at all levels.

A 2019 Maryland law created the Office of Program Evaluation and Government Accountability (OPEGA), which at the direction of the Joint Audit and Evaluation Committee, conducts evaluations. OPEGA evaluates the efficiency, effectiveness, and economy with which resources are used in meeting desired results. Evaluations also evaluate whether the governmental activity or unit operates in an open, accountable, fair, and non-discriminatory manner.

In January 2023, Governor Walz and Lt. Governor Flanagan released the One Minnesota Plan, which outlines strategic priorities for the state through 2027. From January to July, senior leaders across state agencies worked with the Governor’s Office and the Results team at Minnesota Management & Budget (MMB) to develop specific, measurable goals to achieve these priorities.

In terms of impact evaluations conducted by the Results Management team, in 2018 MMB adopted an evaluation policy which governs its use of evaluations and requires the public release of all completed evaluation reports regardless of findings. The policy outlines key principles for evaluation: rigor, relevance, independence, transparency, and ethics. It also requires pre-registration of evaluations and final results. Once complete, evaluations become a part of Minnesota’s Inventory, which shows the evidence base for 730 state interventions, and allows for sorting based on level of evidence, service population, settings, and outcomes.

The Legislative Finance Committee’s Program Evaluation Unit conducts in-depth studies of state government programs and agencies to determine whether the taxpayers’ dollars are being spent effectively. Program evaluations cover a variety of issues and can involve the breadth of government, from state and local governments to public school districts. New Mexico has published a series of inventory and cost-benefit reports in the areas of children’s behavioral health, adult behavioral health, early education, child welfare, criminal justice, healthcare, and education. In 2019, the Legislative Finance Committee’s evaluation unit published analyses on substance abuse treatment outcomes and Medicaid spending and managed care administration. In FY21, the evaluation unit partnered with Abdul Latif Jameel Poverty Action Lab North America (J-PAL NA) to evaluate the effectiveness of the state’s early college high schools. In 2022, LFC published a cost-benefit report on child welfare. The LFC also publishes its learning agenda in the form of analyst work plans. All reports from the evaluation unit can be found here.

To advance Tennessee’s evidence-based budgeting efforts, the Office of Evidence and Impact defined five evidence steps, conducted program inventories, developed evidence reviews, and provided evidence-building technical assistance, such as guidance for prioritizing programs for evaluation funding and implementation. In addition to the evidence framework that guides work around evidence-based budgeting and evidence building, the Tennessee Office of Evidence and Impact supports a number of partnerships that have produced a body of evidence in key education policy areas. The Tennessee Education Research Alliance is a formal research partnership between the Tennessee Department of Education and Vanderbilt University’s Peabody College of Education. Led by full-time staff and guided by a steering committee and advisory council, the Department and the University have co-created a research agenda that builds a body of knowledge to better position the state to make data-driven and evidence-based decisions. The Alliance conducts independent studies and directs external research to provide relevant and timely information to state policymakers across a variety of topic areas, including early reading, professional learning, and school improvement.

The Office of Evidence and Impact recently launched an initiative, led by the Chief Evaluation Officer, to develop a current state analysis of evaluation activities and capacity across Tennessee’s executive branch, craft evaluation guidelines, develop a statewide learning agenda, and produce a curriculum and materials to train agencies to develop their own learning agendas in partnership with the Office of Evidence and Impact. These will be available online in early 2024.

The Utah Data Research Center (UDRC) was created by a 2017 law to integrate data from the Utah System of Higher Education, Utah System of Technical Colleges, Utah State Board of Education, Utah Department of Health, and the Utah Department of Workforce Services. Together, these constitute the “P20W” pipeline. UDRC offers “data products” (such as academic-quality research, reports, and dashboards) and “data as a product” for external researchers. UDRC annually publishes a research agenda in priority order including the primary stakeholder and research description. It also maintains a public archive of previous years’ learning agendas.

The Utah Data Research Center’s July 2023-June 2024 research agenda investigates themes related to education and workforce outcomes. Previously published research agendas are publicly available. As of July 1, 2022, UDRC was transferred from the Utah Department of Workforce Services to the Utah System of Higher Education per a 2022 Utah law.

Since 2012, at the direction of the legislature, the Washington State Institute for Public Policy (WSIPP) has produced a series of inventories of evidence-based, research-based, and promising programs. These inventories are developed with the aim of informing state agencies about effective and cost-beneficial options for service provision. In addition, a 2013 state law directed the Department of Corrections, in consultation with WSIPP, to: (1) compile an inventory of existing programs; (2) determine whether its programs were evidence-based; (3) assess the effectiveness of its programs, including conducting a benefit-cost analysis; and (4) phase out ineffective programs and implement evidence-based programs. As a result of laws like these that require evidence reviews and/or direct outcome evaluations, WSIPP has published hundreds of benefit-cost analyses in a wide variety of issue areas over the past 10 years. The WSIPP benefit-cost framework allows the translation of program effects into dollars and cents and has been emulated by several state governments across the country.

WSIPP’s most recent inventories include reviews of the evidence in the areas of children’s mental health, child welfare, and juvenile justice (2020) and K-12 educational programs for struggling students (2020), which is also featured on the state superintendent’s Menus of Best Practices and Strategies website. The 2023 legislature directed WSIPP to update its inventory of adult corrections programs, with a focus on programs for individuals in prison facilities.

In addition to evidence reviews and benefit-cost analyses, WSIPP is often directed by law to study the implementation and effectiveness of state programs as an objective evaluator independent of state agencies. WSIPP currently has ongoing legislatively directed work to examine the long-term costs and benefits of legalizing recreational cannabis, evaluate the effectiveness of Washington State’s Drug Offender Sentencing Alternative, and more. All projects on WSIPP’s research and evaluation plan are published on its Current Project Projects page.

In 1988, the Legislature changed its current focus to program evaluation or performance auditing, which involves evaluating the effectiveness and efficiency of programs and analyzing related policy issues. The goal of the Program Evaluation Section and the Management Audit Committee is to provide the Legislature with useful, objective, and timely information about the extent to which desired program results are being achieved. This information is intended to facilitate legislative and executive actions to improve state government and to consider policy alternatives to current programs.