Module 5
Reporting Cost Analysis Results
The final stage of conducting a cost analysis is reporting the results and disseminating them to audiences who can use the information to make decisions about educational programs, policies, and strategies; about funding; or to plan their own research. Once you have published your own cost analyses, you may be expected to review manuscripts and grant proposals that include economic evaluations. This last module addresses these topics.
In This Moduleβ¦
-
How should I report and disseminate my cost analysis findings?
What summary metrics and cost breakdowns should I include?
How do I calculate a CE ratio?
How should I display my results?
Where can I disseminate my cost analysis findings?
How can the results be used to inform education decision-makers?
-
Before you begin this module, review a checklist of questions to assess your readiness for the reporting stage.
-
Module 5: Video 1
Reporting Cost Analysis ResultsModule 5: Video 2
Reporting Sections and ExamplesModule 5: Video 3
Calculating Cost-effectiveness ratiosModule 5: Video 4
Leveraging CAPCAT 1.2 Plus for Reporting and Presentation of Results -
The state of cost-effectiveness guidance: ten best resources for CEA in impact evaluations
Cost-effectiveness analysis
Cost-effectiveness analysis of early reading programs: A demonstration with recommendations for future research.
Links to the readings are available in the Recommended Readings section below
-
Can community colleges afford to improve completion? Measuring the cost and efficiency consequences of reform
Resource requirements and costs of developing and delivering MOOCs
REACH Cost Analysis Report
Converting an odds ratio to a range of plausible relative risks for better communication of research findings
Links to these items are in the Additional Resources section below.
-
At the end of the module, we invite you to evaluate a published economic evaluation against the Standards for the Economic Evaluation of Educational and Social Programs, using a template we provide.
The development and production of these materials was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305U200002 to Teachers College, Columbia University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
Module 5: Video 1
Reporting Your Cost Analysis
04:12 minutes
This video provides an overview of the content and resources provided in Module 5 on reporting cost analysis. It raises a few key issues to consider before getting started on the final stage of your study.
Key Terms You Should Know from Video 1:
Audience, reference case, transparency of results, credibility of results, Standards for the Economic Evaluation of Educational and Social Programs/CASP Standards
Methods Note: Reporting table templates.
On the CAP Project resources page you can find Cost Analysis Results Table Templates, an Excel workbook that includes two different table designs, provided as example templates for cost analysts to display results. For each table, there are two versions: one that can be used in a report, and a plainly-formatted version for journal manuscripts.
Each table includes a list of ingredients or resources needed to implement an education program/practice, categorized by personnel, materials, facilities, and other inputs. You can replace these with your own program-specific ingredients and modify the tables, e.g., by adding more rows for additional ingredients and more columns for additional sites.
Costs per Site Table: This is useful when your study includes a small number of sites and where costs differ enough to make it worthwhile showing how they vary across the sites.
Lo Mean Hi Costs Table: This template provides an alternative format for cost analysis results which may be more useful when you have too many sites to include each site in a table.
Module 5: Video 2
Reporting Considerations for Cost Analysis
and CEA
14:03 minutes
This video reviews different ways to report the results of an economic evaluation and what you should include in each section. Download this checklist of reporting items referred to several times in the video so you can follow along:
Watch Video 2
Key Terms You Should Know from Video 2:
study audience; summary metrics; cost breakdowns; sensitivity analysis
Methods Note: Fixed/Lumpy/Variable Cost Breakdown
Reviewers, funders, and educators often ask for a breakdown of costs into fixed, lumpy, and variable costs. Why? It matters when decision-makers consider scaling up a program. Fixed costs wonβt change, variable costs will go up directly in line with the number of participants served, and lumpy costs will increase at intervals. So this categorization can be important when predicting costs of implementing a program for additional participants or sites such as schools or districts. Of course, there is usually some level of scale at which even a βfixedβ cost becomes lumpy.
Module 5: Video 3
How to Calculate Cost-Effectiveness Ratios
09:51 minutes
This video reviews the conditions that must be met to allow you to calculate a cost-effectiveness ratio (CE ratio). It shows you how to produce a CE ratio using different types of effectiveness data.
Key Terms You Should Know from Video 3:
cost-effectiveness ratio, cost-effectiveness plane
Methods Note. Cost-effectiveness plane (CE Plane)
On the CAP Project Resources page youβll find a one-pager - CE Plane - which shows you how to report cost-effectiveness ratios on a simple scatter plot chart to make interpretation easy. You can use a CE plane to compare costs and effects of a single program implemented at multiple sites if you have site-level costs and effects. Or you can compare multiple programs that target the same outcome (e.g., reading comprehension) and the same types of participants (e.g., below grade middle school readers), if the control conditions are similar.
Just be aware that you may come across CE planes where the control condition is different for each program reported so the origin of the chart, which represents the control or business-as-usual condition, represents a rather murky condition so the incremental costs of the treatment programs are not directly comparable. This is parallel to the problem in meta-analysis whereby different studies have different control conditions so it's not clear that the effect sizes are truly comparable. With costs, there are additional questions about whether they are all expressed in local or national prices, the year in which the prices are expressed, and whether they have all been estimated using similar assumptions and methods.
Module 5: Video 4
Leveraging CAPCAT 1.2 Plus for Reporting and Presenting Results
15:14 minutes
This video shows you where to find summary metrics, cost breakdowns, and graphics in CAPCAT 1.2 Plus. It also illustrates how to present cost analysis results visually in bar and pie charts and how to create your own CE plane in Excel. You can find the completed example analysis in CAPCAT 1.2 Plus on the CAP Project templates page (demo CEA of Reading Recovery vs. Fast ForWord Reading"*).
Watch Video 4
*Note that the most recent version of the analysis shows slightly different summary results from this video: a few ingredients have been re-categorized as start-up rather than ongoing ingredients. This means their costs are amortized and result in lower economic costs. In addition, a few additional Reading Recovery ingredients have been marked as incurring expenditures which results in higher expenditures.
Key Terms You Should Know from Video 3:
Summary metrics, Cost breakdowns, Incremental costs, Cost-effectiveness ratio (CE ratio), CE plane
Methods Note: Disseminating Economic Evaluations
On the CAP Project Resources page you can find a resource Disseminating Economic Evaluations which provides a few tips about reporting your results to different audiences and lists a variety of venues for sharing cost analysis/CEA results. These include conferences, blog sites, and journals. It includes examples of economic evaluations published at each listed journal.
Model Publications of Cost Analysis and Cost-effectiveness Analysis Results
This 8-page Issue Brief models the presentation of various cost metrics for a Workforce Investment Act of 1998 (WIA) Adult and Dislocated Worker program: average costs per each of five components or services; costs per service at urban, rural, and mixed locales; costs per service at locations with small (<$2mm), medium ($2mm to less than $6mm) and large ($6mm+) workforce funding allocations.
ISSUE BRIEF
Mastri, A., & McCutcheon, A. (2015). Costs of services provided by the WIA Adult and Dislocated Worker programs. Washington, DC: US Department of Labor.
This 134 page report provides cost estimates of best practices for providing special education and related services to students with disabilities. It models cost estimation of interventions at a statewide level and presentation of findings for a policy audience. Exhibit 10 in particular demonstrates effective visualization of cost estimates and breakdowns by disability classification and by type of service. Exhibit C.2.1. adds another breakdown: by school level (elementary, middle, high). Exhibit C.3.1. provides a neat visual summary of overall findings showing the excess cost for an average student with a disability compared with a student without a disability.
REPORT
Danks, A., Fatima, S., Rettiger-Lincoln, E., Junk, K. N., Harris, P., Levin, J., ... & Kolbe, T. (2022). Special Education in Ohio. American Institutes for Research.
This very clearly written 11-page journal article in School Psychology by Cil et al. describes a cost-effectiveness analysis of a classwide intervention that specifically targets a subgroup, English Learners ELs) with the goal of improving English language proficiency. The authors show how, despite the intervention being delivered to the whole class, they attribute costs to the ELs in order to tie costs to observed effects. Of particular value for decision-making, they model cost estimation for hypothetical scenarios in which implementation is varied (see Sensitivity Analysis section and onwards on pp.53-54. Table 4 models the presentation of cost-effectiveness ratios in which a cost per student metric is combined with effect size.
JOURNAL ARTICLE
Cil, G., Chaparro, E. A., Dennis, C., & Smolkowski, K. (2023). The cost-effectiveness of an English language curriculum for middle school English learners. School Psychology, 38(1), 48β58. https://psycnet.apa.org/doiLanding?doi=10.1037%2Fspq0000515
This article demonstrates retrospective cost analysis in which the authors estimated costs of interventions which had previously been evaluated for effectiveness by other researchers. Table 2 illustrates the presentation of cost-effectiveness ratios where effectiveness is not represented as an effect size but as a gain in the number of individuals who completed high school relative to business as usual.
JOURNAL ARTICLE
Hollands, F., Bowden, A. B., Belfield, C., Levin, H. M., Cheng, H., Shand, R., ... & Hanisch-Cerda, B. (2014). Cost-effectiveness analysis in practice: Interventions to improve high school completion. Educational Evaluation and Policy Analysis, 36(3), 307-326.
This article compares costs and effects for 2 digital math tools as typically implemented in a school district. Table 2 shows side-by-side quantities and costs per ingredient for the two digital math tools facilitating a comparison of the resource demands. The table includes assumptions used in cost estimates and clearly states the implementation scenario for each tool (number of students, classrooms, and schools), as well as the average dosage per student of each tool.
JOURNAL ARTICLE
Hollands, F. M., & Pan, Y. (2018). Evaluating Digital Math Tools in the Field. Middle Grades Review, 4(1). https://scholarworks.uvm.edu/mgreview/vol4/iss1/8
This conference poster, presented at AERA 2021, demonstrates the key information to provide for an academic conference poster when reporting a cost analysis. This includes brief sections on background/introduction, methods, results, and conclusions, including recommendations. A few visuals are used to illustrate the results.
CONFERENCE POSTER
Hollands, F. M., Leach, S. M., Stone, E., Head, L., Wang, Y., Shand, R. L., Yan, B., Dossett, D.H., Chang, F., Chang, Y., & Pan, Y. (2021). Costs and Effects of School Nursing on Student Attendance and Absenteeism. Poster presented at the Annual Meeting of the AERA Online.
Methods Note: Assessing an economic evaluation against the Standards for the Economic Evaluation of Educational and Social Programs
To help establish the credibility of your analysis for editors, reviewers, and readers, it may be helpful to discuss the extent to which your study meets the Standards for the Economic Evaluation of Educational and Social Programs. CAP Project provides a CASP Standards Template (on the CAP Project Resources page in the section Getting Started with Cost Analysis) to facilitate this for the cost analysis portion of an economic evaluation. It lists everything the standards recommend should be addressed in a high quality cost analysis and allows you to score your work (or someone elseβs) against these standards. In the limitations section of your manuscript or report, you can address how well you met the standards overall and why you were not able to meet some. No one study is likely to meet every standard, often for perfectly justifiable reasons!
You can also use the CASP Standards Template to help you review a manuscript for a journal or a grant proposal. The end-of-module assessment for this module provides you with an opportunity to practice this.
Applying Economic Evaluation to Education Decision-making
Evidence-based decision-making: an introduction to return on investment approaches for school districts
This slide deck, originally prepared for a CAP Project presentation to a large school district, explores the pros and cons of various research methods that have been developed over the past few decades to help decision-makers choose among programs and practices in which to invest: academic return on investment, program value added analysis, cost analysis using the ingredients method, cost-effectiveness analysis, cost-benefit analysis, and cost-utility analysis. We review applications to education agency decision-making and discuss when each method might best be used.
Recommended Readings
The readings for this module include a journal article identifying the top ten resources for CEA in the development field, an online resource from the Health Economics Resource Center which describes how CEA can be used in decision-making and how to report uncertainty of CE ratios, and a journal article demonstrating the application of CEA to reading programs with recommendations for future research to advance the usefulness of CEA.
-
For cost analysts working in the field of development, this journal article identifies the most useful resources available!
Glandon, D., Fishman, S., Tulloch, C., Bhula, R., Morgan, G., Hirji, S., & Brown, L. (2022). The State of Cost-Effectiveness Guidance: Ten Best Resources for CEA in Impact Evaluations. Journal of Development Effectiveness, 1-12. https://www.tandfonline.com/doi/full/10.1080/19439342.2022.2034916
-
This online resource describes CEA and illustrates how it can be used to make decisions about which medical interventions to support.
Health Economics Resource Center (n.d.). Cost-Effectiveness Analysis. U.S. Department of Veterans Affairs.
-
Hollands, F. M., Kieffer, M. J., Shand, R., Pan, Y., Cheng, H., & Levin, H.M. (2016). Cost-effectiveness analysis of early reading programs: a demonstration with recommendations for future research. Journal of Research on Educational Effectiveness, 9(1), 30-53.
Additional Resources
Here are a few more publications that report economic evaluations of educational programs or policies, including two in higher education. The REACH Cost Analysis Report includes numerous tables providing a variety of cost breakdowns. The last journal article in the list tackles an issue raised in Video 3 which you may encounter when producing a CE ratio for interventions with binary outcomes (e.g., graduated/did not graduate). It describes how to convert an odds ratio to a range of plausible relative risks.
-
Belfield, C., Crosta, P., & Jenkins, D. (2014). Can community colleges afford to improve completion? Measuring the cost and efficiency consequences of reform. Educational Evaluation and Policy Analysis, 36(3), 327-345.
-
Hollands, F. & Tirthali, D. (2014). Resource requirements and costs of developing and delivering MOOCs. The International Review of Research in Open and Distance Learning, 15(5), 113-132
-
Shand, R., Muroga, A., Rodriguez, V., & Levin, H. M. (2018). REACH Cost Analysis Report. Center for Benefit-Cost Studies of Education, Teachers College, Columbia University.
-
Grant, R. L. (2014). Converting an odds ratio to a range of plausible relative risks for better communication of research findings. BMJ, 348. doi:10.1136/bmj.f7450.
See also the correction at https://www.bmj.com/content/348/bmj.g2124
Apply Your Learning
At this point you should be able to conduct and report on a high quality cost analysis or CEA, and evaluate those of other cost analysts. Take the End of Module Self-Assessment to test your ability to apply your learning to evaluate a published cost analysis. To complete this assessment, you will need to download the CAP Project Cost Analysis Standards Template.
Your Feedback
How helpful was this module? Do you feel more confident about your ability to report a cost analysis? What can we improve? Please complete the Module 5 Feedback Questionnaire.
CONGRATULATIONS!
We hope that you have developed new skills in designing a cost analysis, collecting and recording the data, analyzing the data to produce cost estimates, and reporting cost and cost-effectiveness metrics. We look forward to reading your contributions to the body of economic evaluations and seeing more data to inform education decision-makers!
Overview
Module 1
Module 2
Module 3
Module 4
You are here β
Module 5
Learning experience and media design by ITβS EDUMENTARY.