Module 4
Analyzing Cost Data
At this stage, with all your cost data documented, it’s time to check for accuracy, consistency, and alignment across ingredient categories; fill in any missing data using justifiable assumptions; adjust prices for inflation and geographical location; spread the costs of durable ingredients over their useful lifetimes; compound or discount costs incurred in different years to account for the time value of money; calculate costs of each ingredient; and conduct sensitivity analysis to assess whether results change substantially when key assumptions are varied.
In This Module…
-
What else do I need to do to be ready for analysis?
How do I standardize prices from different years and locations?
How can I spread the costs of durable items over their useful lifetimes?
How do I standardize costs that are incurred in different years?
What is sensitivity analysis, which assumptions should I test and and how do I do it?
-
Before you begin this module, take the pre-quiz to test whether you know which adjustments to prices and costs are needed when.
-
Module 4: Video 1
Introduction to Module 4Module 4: Video 2
Price Adjustments in Cost Analysis: InflationModule 4: Video 3
Price Adjustments in Cost Analysis: AmortizationModule 4: Video 4
Price Adjustments in Cost Analysis: Discounting/CompoundingModule 4: Video 5
Price Adjustments in Cost Analysis: Geographical AdjustmentsModule 4: Video 6.1-6.4
Adjustments and Sensitivity Analysis in a Cost Analysis Template
Module 4: Video 7
Analyzing Cost Data in a Cost Analysis Template: Distributing Centralized Costs Across Sites -
Bhula, R., Mahoney, M. , & Murphy, K. (N.d.) .Conducting cost-effectiveness analysis (CEA). https://www.povertyactionlab.org/resource/conducting-cost-effectiveness-analysis-cea
Shand, R., & Bowden, A. B. (2022). Empirical support for establishing common assumptions in cost research in education. Journal of Research on Educational Effectiveness, 15(1), 103-129. https://www.tandfonline.com/doi/full/10.1080/19345747.2021.1938315
Leach, S.M., Hollands, F.M., Stone, E. et al. Costs and Effects of School-Based Licensed Practical Nurses on Elementary Student Attendance and Chronic Absenteeism. Prev Sci (2022). https://doi.org/10.1007/s11121-022-01459-0
-
E$timator
A free online tool that facilitates cost analysis.CWIFT Indices
Geographical indices for US states, counties, and local education agencies.CAP Project Demo Cost Analyses
Examples of completed cost analyses in the CAPCATs. -
At the end of the module, we ask you to make a few adjustments to prices for the Stepping Up program (described in earlier module assessments).
The development and production of these materials was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305U200002 to Teachers College, Columbia University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
Methods Note:
Before commencing analysis, check your data for accuracy, completeness, and consistency
Before starting cost data analysis, it is very important to check your data for accuracy, completeness, and consistency. It also helps to have a second analyst review your data entries and assumptions. Completing the list of data checks below will help you improve data accuracy and completion:
-
Compare the details in written or verbal descriptions of the program or in your overview tables to the ingredients you have listed and to the data you have collected on units, quantity, percentage of use, and price, for each resource. The details should align. Here are examples:
Ex. 1) If you noted that one full-time staff person spends half of their time on the program, the unit is a person, the quantity should be 1, and the percentage of time spent on the program should be .5 or 50 percent. The price would be this person’s annual salary and fringe.
Ex. 2) If instead you noted that an hourly aide spent 20 hours per week over 6 weeks on the program, then the unit is an hour of the aide’s time, the quantity is 120, the percentage of use is 100%, and the price should be the aide’s hourly rate.
-
If you are missing data on the units, quantity, percentage of use, or price of any resource, you will either need to go back to your data sources to gather the missing details, or you’ll need to make justifiable assumptions to fill in the missing data. For example, if you cannot find the salary for an assistant principal, you might make an assumption that it is 10-20% lower than a new principal’s salary.
-
For example, if the program requires 30 laptops and the price you obtained is for a 10-pack of laptops, then the unit is a 10-pack and the quantity you need at that price is only 3. The price would be the price for the ten-pack, let’s say $8,000.
Alternatively, you can make the unit a single laptop and the quantity would be 30 but the price would need to be shown as one 10th of the price of the 10-pack: $800.
-
Examples:
Ex. 1) If you entered 20 hours for a counselor’s time to implement a program, do you also include facilities, and materials and equipment used by the counselor for those 20 hours (but not more than 20 hours)?
Ex. 2) If you have records showing that 45 students used a digital tool, do you show an adequate number of digital devices and also internet access if it’s needed?
Ex. 3) If teachers spent time in professional development as part of the program, have you accounted for their training time as well as the time, and any travel costs, for a trainer?
-
Examples
Ex. 1) resources provided by a district office may be used at multiple schools in your study so the costs should be spread among them.
Ex. 2) If equipment is shared with students who are not part of your study, you should only attribute a proportional share of costs to the students who are in the study.
-
When you think you are ready to share data summaries and assumptions, do some “member checking” - that means checking your data and assumptions with the people who served as sources of the data. This may have been staff implementing the program or people receiving services. You can share the overview tables and your assumptions with implementers and participants to check that the ingredients, quantities, percentages of use, and assumptions make sense to them given their practical knowledge of the program.
-
Make sure you annotated your worksheet clearly so your assumptions, data sources, and calculations are explained and can all be replicated by you and others.
Module 4: Video 1
Introduction to Module 4
11:29 minutes
This video provides an overview of activities the analyst should undertake at the analysis stage: checking data collected for completeness, accuracy, and consistency; assigning values to each ingredient; making adjustments to prices and costs where necessary; categorizing costs; distributing centralized costs across sites; and conducting sensitivity analysis.
Key Terms You Should Know from Video 1:
Member checking, inflation adjustment, amortization, discounting, compounding, geographical adjustment, fixed/variable/lumpy costs, sensitivity analysis, base case
Methods Note:
Cost-feasibility analysis. When facing budget constraints, decision-makers may need a cost analysis to take a further step to assess whether they can afford to procure the new resources or can reallocate existing resources needed for the program.
The analyst must compare the resources needed to implement the program to resources that are already available or need to be procured. This requires an inventory of relevant existing ingredients and an assessment of which of these are underutilized or can be fully or partially reallocated without undesirable consequences for other activities. This assessment may require data collection on current personnel time use, a review of school schedules and room availability, and inquiries regarding current use of materials and equipment that could potentially be shared with new activities.
Once the gap between available and necessary resources is determined, the decision-maker needs to assess whether the missing resources can be purchased or otherwise acquired.
Alternatively, they could consider whether the implementation could be modified to fit within available resources without likely loss of effectiveness (Pratt-Williams, Harris, Saucedo, & Mislevy, 2021). For example, it might be possible to partner with other schools or districts to share trainer fees.
Methods Note:
Cost of Facilities Calculator and Amortization. If you used CAP Project’s Cost of Facilities Calculator to estimate costs of a classroom or other educational space, it will have already amortized the costs of the facilities over its useful lifetime. Be sure you do not amortize again if you enter a price from the Cost of Facilities Calculator into CAPCAT!
Module 4: Video 2
Price Adjustments in Cost Analysis: Inflation
4:35 minutes
This video introduces the four most common adjustments you will need to make to prices/costs in a cost analysis. It explains inflation adjustments in detail with a worked example.
Watch Video 2
Key Terms You Should Know from Video 2:
Inflation index, CPI-U
Module 4: Video 3
Price Adjustments in Cost Analysis: Amortization
5:18 minutes
This video explains how to spread the value of an asset over its useful lifetime and where to find interest rates to use in this amortization calculation. A worked example is included.
Watch Video 3
Key Terms You Should Know from Video 3:
Amortization, useful lifetime
Module 4: Video 4
Price Adjustments in Cost Analysis:
Discounting/Compounding
04:23 minutes
This video explains the concept of time value of money and why you need to standardize costs incurred in different years. An example of how to execute this adjustment is included.
Watch Video 4
Key Terms You Should Know from Video 4:
Time value of money, discounting, compounding
Module 4: Video 5
Price Adjustments in Cost Analysis:
Geographical Adjustments
03:47 minutes
This video demonstrates how to adjust prices obtained from one geographical location to reflect price levels in a different location. An example geographical adjustment is included.
Watch Video 5
Key Terms You Should Know from Video 5:
Geographical adjustment, regional price parities
Methods Note:
Sensitivity analysis involves varying one or more assumptions used in your analysis to assess whether and how overall results change and might lead to different conclusions. Practically, you can execute these analyses by making a copy of your cost analysis worksheet and making changes to the copy so you can compare the results with the original or “base case” analysis. If you use E$timator, you can similarly duplicate a project and use the copy for the sensitivity analysis.
Focus your sensitivity analyses on variables about which you had to make assumptions and about which you are most uncertain, especially those that are likely to produce substantial changes in costs. Possible items to vary include:
1) The discount rate, especially for programs implemented over a long period of time
2) The useful lifetimes of key durable assets such as a purpose-built facility for the program
3) The amount of time spent on the program by personnel from whom you have not been able to collect time use data
4) Prices of personnel ingredients in instances where the personnel experience level may vary and salaries increase substantially with additional years of experience. For example, you could explore the impact on your cost results of using salaries for teachers with 25+ years of experience vs. using the much lower salaries for teachers with only 5 years of experience.
Module 4: Video 6.1
Adjustments and Sensitivity Analysis in a Cost Analysis Template: Inflation
5:14 minutes
This video reminds the viewer of the four most common adjustments needed to prices/costs in a cost analysis. It demonstrates how CAPCAT 1.2 executes an inflation adjustment using CPI-U. It also shows you where to change inflation indices if you prefer to use a different inflation index, and where to add inflation indices and interest rates to allow for a prospective cost analysis (i.e., one that estimates costs anticipated in future years).
Watch Video 6.1
Key Terms You Should Know from Video 6.1:
Inflation index, prospective cost analysis, CPI-U, sensitivity analysis
Module 4: Video 6.2
Adjustments and Sensitivity Analysis in a Cost Analysis Template: Amortization
6:37 minutes
This video demonstrates how CAPCAT 1.2 amortizes the value of ingredients that have a useful lifetime of more than a year. It also shows you where to change assumed interest rates and default lifetimes to facilitate sensitivity analyses. Examples help you anticipate how your results should change when you vary these assumptions.
Watch Video 6.2
Key Terms You Should Know from Video 6.2:
Amortization, base case analysis, interest rate, treasury bonds, sensitivity analysis
Module 4: Video 6.3
Adjustments and Sensitivity Analysis in a Cost Analysis Template: Discounting/Compounding
7:05 minutes
This video demonstrates discounting and compounding adjustments in CAPCAT 1.2 Plus. Be sure to watch Module 4 Video 4 first so that you understand the concepts behind these adjustments and do not confuse them with inflation adjustments! Remember: inflation adjustments are needed when prices come from different years; discounting or compounding is needed for multiyear programs in which costs are incurred over multiple years.
Watch Video 6.3
Key Terms You Should Know from Video 6.3:
Discounting, compounding, discount rate, time value of money
Module 4: Video 6.4
Adjustments and Sensitivity Analysis in a Cost Analysis Template: Geographical Adjustments
12.36 minutes
This video shows you how you can use geographical indices to adjust local prices to produce national costs in CAPCAT 1.2 Plus. It also demonstrates how to execute additional geographical adjustments in CAPCAT 1.3 Advanced: converting costs from one locality to another locality, converting national average prices to produce local costs; and making geographical adjustments to individual ingredients.
Watch Video 6.4
Key Terms You Should Know from Video 6.4:
Geographical index, Bureau of Economic Analysis
Methods Note:
Expenditure analysis You can use the expenditure column in the CAPCATs to document expenditures - these are not adjusted in any way but are simply the raw outlays of funds needed to pay for resources that must be acquired specifically for the program being studied. Sometimes, this is all decision-makers say they want to know about because that’s what they have to budget for. Try showing them both economic costs and expenditures and explaining why it’s important to consider all resources used for the program, even if they don’t need to be purchased. Resources used for this program cannot be used for other purposes at the same time so something is being given up somewhere!
Module Note:
Where can I see the rest of the ingredients entered into CAPCAT 1.2 Plus?
You can find the fully-completed CAPCAT 1.2 Plus worksheet with all ingredients for our example cost-effectiveness analysis of Reading Recovery vs. Fast ForWord Reading on the CAP Project templates page. Note that the templates are updated time to time so you may find small differences in what is shown in the Modules and what you see in the latest version available on the templates page.
Module 4: Video 7
Analyzing Cost Data in a Cost Analysis Template: Distributing Centralized Costs Across Sites
09:24 minutes
This video demonstrates how you can use CAPCAT 1.2 Plus to distribute centralized costs, such as those incurred at a district office, across sites such as schools. It also shows where and how site level costs using local and/or national prices are reported in CAPCAT.
Key Terms You Should Know from Video 7:
Centralized costs, sites
Methods Note:
Treatment-on-the-Treated (TOT) vs. Intent-to-Treat (ITT) Cost Estimates In a cost-effectiveness analysis, costs should only be included for the sample for which effects are being reported. For a TOT analysis, you will need to eliminate costs that are incurred at sites that attrited. To help you with this, the CAPCATs include a column that is labeled “Costs Included” which, if you change the default from “Yes” to “No,” allows you to drop a cost from the results without having to delete all the data. You’ll also need to remember to distribute centralized costs only across the remaining sites. Duplicating your worksheet once your analysis is complete allows you to use one copy for the TOT analysis, and one for the ITT analysis.
Recommended Readings
Recommended readings for this module include a description of cost-effectiveness analysis by researchers at J-PAL who conduct a great deal of international work; a second look at an article that includes many suggestions for reasonable assumptions to make when conducting cost analysis; an article describing a CEA of a school nurse program with the completed cost analysis provided in CAPCAT 1.3.
-
Online summary of cost-effectiveness analysis by researchers at J-PAL.
Bhula, R., Mahoney, M. , & Murphy, K. (N.d.) .Conducting cost-effectiveness analysis (CEA). https://www.povertyactionlab.org/resource/conducting-cost-effectiveness-analysis-cea
-
This article includes suggestions for data sources and assumptions to use in your cost analysis.
Shand, R., & Bowden, A. B. (2021). Empirical support for establishing common assumptions in cost research in education. Journal of Research on Educational Effectiveness. DOI: 10.1080/19345747.2021.1938315
-
Cost-effectiveness analysis of school nurse program with accompanying completed CAPCAT 1.3 template.
Leach, S. M., Hollands, F. M., Stone, E. et al. (2022). Costs and Effects of School-Based Licensed Practical Nurses on Elementary Student Attendance and Chronic Absenteeism. Prevention Science. https://doi.org/10.1007/s11121-022-01459-0
Apply Your Learning
At this point you should be able to make adjustments to prices for inflation and geographical location, amortize the value of durable ingredients over their useful lifetimes, discount or compound costs to account for the time value of money, and conduct sensitivity analysis. Take the End of Module Self-Assessment to test your ability to apply your learning.
Additional Resources
Additional resources listed here include:
i) A free, online tool, E$timator, that executes cost analysis with automatic adjustments based on defaults or user input. Includes 4 completed cost analyses as demonstrations. Useful for users who prefer wizard-based software to Excel templates like the CAPCATs.
ii) The CWIFT Indices which allow for geographical price adjustments at the state, county and local education agency levels.
iii) CAP Project demonstration cost analyses on Reading Recovery vs. Fast ForWord Reading; an online program for middle and high schools; and on professional development costs for math programs illustrating geographical adjustments at the ingredient level.
-
Hollands, F. M., Hanisch-Cerda, B., Levin, H. M., Belfield, C.R., Menon, A., Shand, R., Pan, Y., Bakir, I., Cheng, H. & Kushner, A. (2015-22). E$timator. Teachers College, Columbia University. www.costtoolkit.org.
-
CWIFT Indices. Cornman, S. Q., Nixon, L. C., Spence, M. J., Taylor, L. L., & Geverdt, D. E. (2019). Education demographic and geographic estimates (EDGE) program: American Community Survey Comparable Wage Index for Teachers (ACS-CWIFT) (NCES 2018-130). U.S. Department of Education, National Center for Education Statistics.
-
CAP Project provides 3 completed cost analyses as models for cost analysts
Your Feedback
How helpful was this module? Do you feel more confident about your ability to conduct a cost analysis? What can we improve? Please complete the Module 4 Feedback Questionnaire.
Up next…
Module 5: Reporting Cost Analysis Results
Overview
Module 1
Module 2
Module 3
MODULE 4
Module 5
You are here ↓
Learning experience and media design by IT’S EDUMENTARY.