Defensible, Data-Driven Program Evaluation for K-12 Leadership

​

Data-Driven Program Evaluation

Moving beyond anecdotes, publisher claims, and perceived benefits to deliver reliable outcome data, standardized local effect sizes, and defensible proof of student growth.

​

1. Data Aggregation & Cleaning

   I aggregate raw screening, diagnostic, and progress monitoring data into standardized local architectures, controlling for missing variables and cohort noise.

2. Standardized Local Effect Sizes

   Instead of relying on national vendor norms, I calculate explicit, local effect sizes to measure exactly how much growth your students achieve under specific interventions.

3. Board-Ready Impact Reporting

   I deliver clear, executive-level reports and longitudinal cohort trajectory models that provide defensible evidence for school board reviews and budget decisions.

​

This module outlines a realistic, repeatable data framework to measure the explicit health of our Tier 1 systems, track intervention cohorts over time, and utilize local effect sizes to ensure our resources drive meaningful student growth.

​

Disclaimer: This is shared as a hypothetical model for evaluating local interventions and programs using available student data. However, this is not an exhaustive method or the only way to evaluate outcomes. My background is in the area of school psychology and educational administration, not statistics. I am merging data analysis with a pragmatic approach to objectively evaluate student outcomes. 

​

Implementation Note: Building out multi-year data pipelines and running local effect-size analyses requires dedicated time and technical execution that busy school teams don't always have. If your district or leadership team needs an outside partner to build these Excel architectures, audit historical intervention cohorts, or facilitate data-team workshops, I am available for district-level consultation. Feel free to reach out via email.

 

STEP 1: BUILDING THE EVALUATION PIPELINE (EXCEL DATA ARCHITECTURE)

To systematically evaluate programmatic impact, leadership teams must synthesize raw data into functional, comparative spreadsheets. The data set should isolate a Tier 1 Only control group and your specific intervention groups across identical seasonal intervals.

​

Your master tracking spreadsheet should be structured around these core metrics:

Demographic & Subgroup Identifiers: Isolate school, grade, and active service plans (such as 504, Special Education, English Learner) to check for systemic equity gaps.

​

Growth Calculus: Calculate standard fall-to-spring difference scores by subtracting the Fall score from the Spring score. Compare this local growth rate against national growth averages provided by publishers to determine if your groups are truly accelerating.

​

The System Health Metric (Proficiency Check): Track the exact percentage of students achieving Tier 1 targets across each season. This acts as a macro-level check on the stability and strength of your universal instruction.

​

The Warning Trigger Target: Code students falling below a critical warning benchmark (such as the 25th percentile). The primary goal of system evaluation is verifying that this high-risk subpopulation is actively shrinking from fall to spring.

​

STEP 2: QUANTIFYING IMPACT VIA LOCAL EFFECT SIZES

Relying strictly on whether a student met an arbitrary benchmark fails to explain the true magnitude of an intervention's success. By calculating local effect sizes, school psychologists can convert raw assessment growth into a standardized metric to determine if a program is practically meaningful.

​

To calculate a local Cohen's d effect size, find the mean growth of your intervention group, subtract the mean growth of your general grade level, and divide that total by the pooled standard deviation of both groups.

​

Strategic Choices in Variance Calculation (Effect Sizes):

Cohen's d (Pooled Variance): The standard recommendation for local school analysis. By pooling the standard deviations of both the intervention and general education groups, you achieve greater statistical precision and ensure the whole student population is accurately represented.

​

Glass's Delta (Control Variance Only): Utilized exclusively when an intensive intervention produces an exceptionally wide variance in growth scores, which would artificially skew a pooled calculation.

​

Contextual Interpretation Benchmarks:

While historical research frequently relies on fixed, rigid interpretations (such as Cohen's traditional metrics of 0.2 as small, 0.5 as moderate, and 0.8 as large), modern educational practice requires evaluating impact within a relative context.

​

The What Works Clearinghouse (WWC) establishes that a local effect size of 0.25 or greater represents a substantively important, qualified positive impact on student achievement, even if the absolute sample size is small.

 

STEP 3: MANAGING MULTI-YEAR COHORT TRAJECTORIES

A comprehensive program evaluation looks far beyond immediate, single-year growth charts. True programmatic health requires longitudinal tracking to determine if short-term intervention gains hold up over time.

​

When analyzing historical cohorts of intensive programs (such as early literacy or reading interventions), teams must systematically audit long-term outcomes at 1, 2, and 3 years post-intervention:

​

Intervention Recidivism Rates: What exact percentage of exited students continue to trigger on screening assessments or require a return to multi-level tiers in subsequent grades?

​

Special Education Convergence: Track how many students within a specific tier group eventually qualify for an Individualized Education Program (IEP) due to a Specific Learning Disability (SLD).

​

Instructional Alignment Audits: If data reveals that a resource yields small or stagnant effect sizes relative to the intensive time and cost of delivery, it serves as a critical signal to adjust the intervention's internal design. For example, early reading interventions experiencing flat growth pathways frequently require the explicit addition of systematic phonics components, sound-spelling mastery, and structured fidelity checks to break the pattern.

​

RESOURCES & TOOLS

Access the NIRN Hexagon Tool & Guide [Hexagon Link via NIRN] - Direct framework for assessing intervention readiness and context.

​

Download the General Intervention Quality Checklist [Intervention Checklist Link] - A diagnostic checklist to highlight areas of high-quality delivery.

​

SELECTED REFERENCES & EVIDENCE BASE

Framework Foundations & Systems:

Burns, M. K., VanDerHeyden, A. M., & Boice, C. H. (2008). Best practices in the delivery of intensive academic interventions. In A. Thomas & P. Harrison (Eds.), Best practices in school psychology V (pp. 1151-1162). National Association of School Psychologists.

​

Kratochwill, T. R., & Shernoff, E. S. (2004). Evidence-based practice: Promoting evidence-based interventions in school psychology. School Psychology Review, 33(1), 34-48.

​

Nagle, R. J., & Glover-Gagnon, S. (2014). Best practices in designing and conducting needs assessment. In P. Harrison & A. Thomas (Eds.), Best practices in school psychology: System-level services (pp. 263-276). National Association of School Psychologists.

​

Sanetti, L. M. H., & Collier-Meek, M. A. (2020). Supporting implementation of evidence-based practices: A focus on school-based professionals. Guilford Press.

​

Wisconsin Department of Public Instruction. (2013). Wisconsin's Specific Learning Disabilities (SLD) rule: A technical guide for determining the eligibility of students with specific learning disabilities. Author.

 

WSPA Article regarding Program Evaluation (link)

​

Statistical Interpretation & Effect Size Frameworks:

Coe, R. (2002, September 12-14). It's the effect size, stupid: What "effect size" is and why it is important [Paper presentation]. British Educational Research Association Annual Conference, University of Exeter, Exeter, England.

​

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

 

Effect Size Part 1 AND Effect Size Part 2 (Practical Application Guides to Effect Size) - two pdfs attached

 

Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532-538.  

​

Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2-18.  

​

Hattie, J. (2023). Visible learning: The sequel: A synthesis of over 2,100 meta-analyses relating to achievement (1st ed.). Routledge. Video Link 

​

Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., & Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. LINK 

​

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2022). What Works Clearinghouse: Procedures and standards handbook (Version 5.0). Link to WWC Manual 

​

VanDerHeyden, A. M., Codding, R. S., & Martin, R. J. (2022). Using local data to evaluate intervention effects and system health. Journal of School Psychology, 92, 142-159.

​

Volker, M. (2006). Reporting effect size estimates in school psychology research. Psychology in the Schools, 43(6), 653-672.

​

Targeted Intervention Mechanics & Practice Guides: (LINK TO PRACTICE GUIDES- ALL) - download any of interest

​

Foorman, B., Beyler, N., Borradaile, K., Coyne, M., Denton, C. A., Dimino, J., Furgeson, J., Hayes, L., Henke, J., Justice, L., Keating, B., Lewis, W., Sattar, S., Steke, A., Wagner, R., & Wissel, S. (2016). Foundational skills to support reading for understanding in kindergarten through 3rd grade (NCEE 2016-4008). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

 

Gersten, R., Compton, D., Connor, C. M., Dimino, J., Santoro, L., Linan-Thompson, S., & Tilly, W. D. (2008). Assisting students struggling with reading: Response to Intervention and multi-tier intervention for reading in the primary grades. A practice guide (NCEE 2009-4045). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. 

​

Iversen, S., & Tunmer, W. E. (1993). Phonological processing skills and the Reading Recovery program. Journal of Educational Psychology, 85(1), 112-126.

 

​Iversen, S., Tunmer, W. E., & Chapman, J. W. (2005). The effects of varying group size on the Reading Recovery approach to preventive early intervention. Journal of Learning Disabilities, 38(5), 456-472.

​

Morris, D., Tyner, B., & Perney, J. (2000). Early Steps: Replicating the effects of a first-grade reading intervention program. Journal of Educational Psychology, 92(4), 681-693.