Does adopting this program represent a meaningful change from current practice? If the program is not substantively different from what is already in place, then benefits of switching may be small. In that case, strengthening implementation of the current program may lead to better outcomes at a lower cost than switching to a new but similar one.

What resources are required? How much will effective implementation cost, and is it feasible? Consider start-up costs (training, set-up), recurring costs (coaching, data review), and the staff time required to coordinate and troubleshoot. Adopting programs that are not well-aligned to other ongoing initiatives may cause additional coordination work. The time required from teachers, coaches, and administrators, along with materials, training, and ongoing support, should all be counted as part of the full cost.For a practical guide to estimating program costs, see IES’s Cost Analysis: A Starter Kit.

What will the program displace? New programs that require a substantial reallocation of time or money have high opportunity costs: they reduce attention to other priorities and may require cutting back on other educational opportunities. If the time and resources come from other successful initiatives, then overall impact includes both gains from the new program and losses from the reallocation of resources. Name what will be reduced or dropped.

Are the program’s components aligned with existing evidence about effective practice? A program may look promising as a package, but specific activities, strategies, and resources can vary in how much they align with what is known from prior research. Look for whether the instructional practices of the program are consistent with evidence summaries such as relevant IES practice guides (see Appendix A for additional resources).

What components are essential and where is there room for local adaptation? It is natural to adapt a program to fit local needs and conditions. Flexibility can help address community members’ concerns and improve local fit, but adaptations could also inadvertently remove or weaken components integral to the program’s effectiveness. When assessing how a program will be implemented, identify which components are essential and which can be modified. Ask whether the available evidence applies to the version the district would implement. If existing policies or resource constraints make it difficult to implement essential components, consider whether those barriers can realistically be addressed. If not, the program may not be a good choice for the district.

Who has to do what? Which roles are responsible for delivery (teachers, coaches, administrators, vendors), and what is the expected routine (frequency, duration, scheduling)? Studies may point to implementation features, such as regular attendance, ongoing coaching, minimum dosage, staffing qualifications, or time allocated for delivery, that appear important to maintaining effectiveness. Use that information to plan for a strong implementation.

Will district and school leaders, teachers, and students use the program as expected, or resist it? Low participation, uneven use, or burnout can undermine even well-designed programs. If past initiatives have struggled with uptake or sustained use, that is a signal to plan for additional supports. If engagement data are not available, ask what evidence exists about uptake in local settings and what supports were used to achieve it. If educators, or parents, or others involved have substantial reservations about the program, then it may be difficult to implement effectively.

How will program implementation be monitored locally to ensure conditions for success? Describe what “implementation as intended” means for the program and what indicators of implementation quality will be tracked (such as participation rates and usage) and how (logs, observations, brief surveys). Identify who is responsible for reviewing the data and how often.

What would count as evidence that implementation is off-track, and what mid-course corrections would be taken? Plan in advance for how concerns will be escalated and handled if participation is low, delivery is uneven, or supports are not being provided. Do not wait until the end of the year to decide what signals would require attention or how to respond.

Have studies of the program been published or made publicly available? Look for full reports, not only marketing summaries. It is a good sign when program developers are willing to engage in evaluation, describe how the study was carried out, and provide detailed results.

Do studies include detailed information on study context and program implementation? Such detail could include program components, who delivered the program, how often it occurred, resources required, and costs. If implementation is described only in general terms, then it is difficult to know what caused changes in outcomes. Context and implementation information is also necessary for assessing the full costs of implementation (Principle 1: Implementation costs) and for determining whether the study is relevant (Principle 6: Meaningful outcomes and Principle 7: Contextual considerations).

If adaptions of the program exist, do any studies speak to the adapted versions? Ongoing evaluation is important when programs evolve over time. Evidence on an old version of a program tells leaders about that version of the program. While adaptations may have improved the program, it is also possible that the program was altered in ways that make it less effective than the original version (e.g., it was oversimplified to control cost).

Did researchers clearly specify the research questions, methods, and analyses, and document the study plans before conducting analyses? Researchers make many decisions while collecting, cleaning, and analyzing data, and these decisions can subtly shape the findings. Plans written in advance, such as a publicly posted protocol, analysis plan, or pre-registration plan, reduce the risk of incorrectly concluding there was an impact when there was none (“false positives”), which might occur if researchers’ decisions are influenced by whether they generate interesting results. Some journals use Open Science Badges to note when authors have shared data or materials or registered their plans in advance. These are useful markers of transparency, but readers should still examine what was shared in advance and whether it maps to the findings reported.

Do studies include a clear and thoughtful discussion of limitations and caveats? Such a discussion signals a cautious and self-reflective analyst. For example, does the report address any differences between the program and comparison group that may call into question whether the program’s apparent effectiveness might be explained by other factors (see Principle 3: Comparison group)? If outcome measures are narrow or short-term, is this mentioned as a caveat (see Principle 4: Interpreting effect sizes)? If the sample has limited diversity or a narrow range of scores, are findings appropriately described as limited to those students? If results vary across sites, subgroups, or outcomes, is that variation reported? Every study has caveats; failing to discuss limitations indicates a lack of transparency.

Are studies conducted by independent researchers with no professional or financial stake in the success of the programs? Developer studies may selectively highlight favorable findings while minimizing or omitting less favorable results. And indeed, studies conducted or commissioned by program developers find larger positive effects on student outcomes than independent studies (Wolf et al., 2020). Pay close attention to what is reported, what is not reported, and whether results have been reproduced by others.

Does the study have a comparison group? Studies that lack any comparison group (e.g., pre-post studies, or before-and-after assessments of the same students) can provide descriptive evidence of change over time, but do not provide evidence of the program’s causal effect. Scores often improve over time because of normal growth and instruction, even without a new program. Changes in staffing, schedules, other supports, or broader events can also explain before-and-after changes. A comparison group helps separate change that would have happened anyway from change plausibly attributable to the program.

Were students assigned to groups randomly? When random assignment or a lottery is used to determine who is assigned to the program, the left-out participants form a comparison group that is usually quite similar to the group assigned to the program. This creates a clean comparison for assessing the benefits of treatment. By contrast, if people self-select into programs, or educators intentionally assign certain students to programs, the factors that drive those choices can cause the program and comparison groups to be inherently different (this is called “selection bias”). In brief, random assignment into the program avoids the problem of selection bias.

Is the comparison group arguably similar to the group who got the program? When random assignment is not feasible, check for evidence that the comparison group is similar before the program was introduced, especially on prior achievement and other characteristics related to the outcomes. If the groups are not comparable – for example, if higher performing students are more likely to participate, or if educators choose who is offered the program – then differences in outcomes may reflect underlying differences in the groups rather than the effect of the program.Researchers refer to similarity between comparison and treatment groups as baseline equivalence (see the WWC Standards Brief for Baseline Equivalence for more details).

Are certain people excluded from the study, and, if so, does that change interpretation of the findings? Excluding people from the study is fine when based on a pre-existing characteristic that can be measured in both the program and comparison group (such as excluding English learners from the study). In contrast, exclusions based on experiences that occur after the program begins can undermine the similarity of the groups. For example, some studies exclude treated students from the analysis based on number of sessions attended, lessons completed, or other measures of their engagement with the treatment. In the comparison group, we cannot identify which students would have participated at that level had they been offered the program. This creates an unfair comparison if students who remain in the treated sample are systematically different from the students excluded.

How narrow or broad is the outcome? Some outcomes are narrow and closely aligned to the program (such as letter-sound fluency), while others capture broader constructs (such as reading achievement). All else equal, we are more likely to see larger effects on outcomes closely aligned with the program and measured soon after implementation, and smaller effects on broader outcome measures or outcomes measured after more time has passed.

Who is included in the study? First, some studies drop students who did not participate or receive the full dosage from the treatment group sample, which may result in larger effect sizes than you would see if the full sample of students offered the treatment were retained in the study.Dropping students in this way can also make the estimate entirely invalid; treat such studies with caution. Second, a standardized effect size expresses the difference between groups relative to the variation in scores, so the amount of variation in the study sample can affect the reported effect size. In particular, the same difference will appear larger in a more homogenous sample with more similar scores (for example, when a study only includes low-performing students) and smaller in a sample with a wider range of scores (as when a study includes all students).

What is the program compared against? Often the comparison is described as “business-as-usual,” but this can mean very different things. Ask what students in the comparison group received. An effect size is the change relative to this contrast condition. For example, if students in the treatment group are receiving one-on-one tutoring, and students in the comparison group receive small-group tutoring, the contrast is between the types of tutoring—not between tutoring and not being tutored at all. We would not expect to see as large an effect when the contrast is small, even if the program is of high quality.

At what scale and under what conditions was the program studied? Small pilot studies can provide more training, monitoring, technical assistance, or researcher involvement than a district could sustain during routine implementation. These conditions can contribute to larger effects than might be found when a program is implemented across more schools with less support. When interpreting effect size, examine the level of implementation support provided in the study and whether the program has produced similar effects in studies conducted at a larger scale. Ask whether the program could be similarly supported if rolled out in a new context without the original researchers’ involvement.

How precise is the estimate? Sample size does not determine whether a program’s true effect is large or small, but it does affect how much uncertainty surrounds the estimate. Estimates from small studies can vary a great deal and may appear unusually large or small by chance. Look at the confidence interval, not only the reported effect size, and ask whether it includes effects that would lead to different conclusions about the magnitude of the effect.

Do other ways of describing the result tell the same story? If the study translates findings into months of learning or other terms, check how those findings were constructed and whether they are consistent with the reported effect size. Translations may rest on additional assumptions or simplifications that can misrepresent findings (Jackson, 2025; Pane & Baird, 2019). Just as effect sizes are relative to some reference group, translations are indexed by some reference group as well. Be sure to check if translated findings are relative to similar students, at similar grade levels, on similar tests, to your target context.

Is the size of the effect worth the cost? When evaluating whether an effect size is “big enough,” consider if the program is hard or easy to implement. A small effect may be good enough if effort and expenses are low. Even a large effect might not be worth an enormous effort or substantial reallocation of resources from other successful programs. Ask if the expected benefit justifies the expected effort and expense.

Could a large sample be making a very small difference look more important than it is? Be cautious when a report emphasizes statistical significance when the estimated effect is very small. When samples are large, tiny differences can be statistically significant. Consider the practical value of a small effect.

How precise is the estimate? To assess practical significance, especially if the study is small, look for confidence intervals. These are ranges that show how much the true effect could reasonably differ from the study’s estimate. Small studies often produce wide confidence intervals, which indicates more uncertainty about the size of the effect. Consider whether the program would be worth adopting if the true effect is near the low end of the range.

Consider the comparison condition when interpreting the estimated effect. If the study compares the program to a strong business-as-usual, we expect the benefit of the program to be smaller. If the same program is compared to a weak business-as-usual (such as low-quality instruction or limited resources), the benefit of the program might appear larger. Consider whether the business-as-usual in the decision context is relatively weak or strong.

Do the outcome measures align with how the findings about the program are reported? If stated findings are about student learning, the measures should be of student learning. Usage metrics (e.g., minutes used, lessons completed, or in-program mastery) or satisfaction measures (e.g., user ratings) describe participation and experience. They do not, on their own, measure student learning. Measures also need to match how findings are reported in breadth. For example, if the finding is described as the program improves reading achievement, the outcome measure should capture reading achievement broadly rather than focusing on a narrow subdomain, such as phonemic awareness.

Are the outcome measures well-constructed (valid) and consistently measured (reliable)? Look for evidence that outcome measures were carefully developed, ideally independently of the study, and for technical documentation of validity and reliability consistent with the Standards for Educational and Psychological Testing (see Appendix A). When outcome measures are created by the study team or embedded in the product, ask what evidence exists that the measure captures the intended construct and is not overly tailored to the program. Independent measures provide a useful check since estimated effects tend to be larger on measures that are more closely tied to the intervention (Wolf et al., 2020; Wolf & Harbatkin, 2023).

Are the outcomes measured associated with longer-term outcomes of interest? For short-term outcomes (observed immediately after the end of a program), check for evidence of predictive validity: are those outcomes at least associated with other policy-relevant or longer-term (12 or more months) outcomes that matter? For example, if the short-term outcome is an interim assessment, does it correlate with end-of-year assessments, grade retention, course completion, or high school graduation? If the link is unknown or weak, short-term gains are insufficient evidence for longer-term outcomes.

How similar are the students and settings to those in the study or studies that demonstrate effectiveness? Pay attention to who was served and where the program was implemented. If the students in the study differ in important ways from those in the local context to be served – for example in grade level, prior achievement, language status, or special education identification – treat the evidence as less directly applicable.

Is the study’s comparison condition similar to current practice? The benefit of a program depends not only on the program itself, but also on the comparison condition. A program may look more effective when compared to a weak alternative than when compared to a strong existing support. When interpreting evidence from another setting, ask what the program replaced in the study and whether that is like what it would replace in the local context.

What features of the context matter for the program to be successful? Consider contextual features, such as state or local policies, teachers’ union contracts, and qualifications of the staff implementing the program. If these conditions differ from the study setting, implementation may look different, and outcomes may then differ as well.

Would the program be as well-resourced and implemented with the same level of quality as in the evidence examined? If not, how might program implementation be compromised? If the program is scaled back, be explicit about what will change (time, staffing, training, supports). Expect “lighter touch” versions of a program to have smaller effects.

How many studies exist, and how much do the settings differ? Evidence from multiple districts, schools, or years is more informative than multiple studies from the same organization in the same setting. Look for variation in student populations, staffing models, schedules, and the comparison condition.

Are the results consistent in direction, even if the size varies? Some variation is expected because study estimates are imprecise and subject to noise. But differences in results may also reflect systematic differences across studies. Look for whether the results tend to point in the same direction across studies. A single null or slightly negative estimate is less concerning if the sample is small or the estimate is close to zero. More concern is warranted when a large, well-designed study shows clearly negative results that are hard to reconcile with positive findings elsewhere.

Are the studies using outcomes that are relevant and similar enough across studies? If one study uses a narrow measure closely aligned to a program and another uses a broader outcome, differences in effect size may reflect the outcome measures used rather than differences in program effects (see Principle 4: Interpreting effect sizes and Principle 5: Practical significance). All else equal, attend more closely to studies that speak to the intended use of the program in terms of the outcomes of interest and the timeframe over which changes in outcomes will be measured.

Is the comparison condition similar across studies, and similar to the local setting under consideration? A program can look more effective when compared to weak practice and look less effective when compared to strong practice. For each study, ask what the comparison group received, and ascertain which study looks the most similar to the treatment contrast in the local context (see Principle 7: Contextual considerations).

Do the studies describe implementation conditions in enough detail to learn from differences? If multiple studies report different impacts, that could be due to differences in implementation. Look for those details. For example, in the case of tutoring, determine the tutor qualifications, degree of training and coaching, group size, session length and frequency, participation rates, and how tutoring was scheduled within the school day. If such details are missing, it is difficult to know whether differences in results could be due to differences in delivery.

Is the subgroup finding based on the right comparison? Subgroup findings are not simple comparisons between groups. A subgroup finding should estimate either the program’s impact within a subgroup or whether the program’s impact differs across subgroups. An impact finding within a subgroup requires comparing students in that subgroup who were offered the program with similar students in the same subgroup who were not. A finding that a program works better for one subgroup than another requires comparing the impact for one subgroup with the impact for a different subgroup, not comparing average outcomes across the groups. In the latter case, the finding could simply reflect baseline differences rather than differences caused by the program. See Principle 3: Comparison group.

Was the subgroup of interest identified before the results were known? Subgroup findings are more credible when researchers publicly state which subgroups will be examined before they conduct the analysis. This is the idea behind preregistration, which records the planned analyses in advance. This reduces the risk that a pattern found only after looking at the data will be treated as if it were expected all along. Since researchers might check many subgroups, preregistration can increase confidence that a reported finding is not just due to random chance.Looking at subgroups increases the possibility of false discoveries (findings due to chance or noise in the data rather than a true effect) which is one reason why subgroup analyses warrant caution (von Hippel & Schuetze, 2025).

How many subgroup comparisons were examined? Evaluation reports should state which subgroup analyses were conducted and how many comparisons were tested. The more comparisons a study examines, the higher the likelihood of false positives. Look for a statement that the study adjusted for multiple comparisons.

Was the subgroup tied to the program’s theory of action? A subgroup analysis should have a reason. For example, a phonics intervention may have a rationale for examining whether effects differ by students’ baseline decoding skills. Subgroup findings are more credible when they are supported by a clear rationale related to how the program works.

Does the subgroup finding appear in more than one site, cohort, or study? A subgroup finding is more credible when the pattern appears more than once. A finding that appears in one study but not others may reflect measurement error or sample composition rather than a real and stable difference in effects.

Could the subgroup finding reflect differences in implementation or access to program supports? One group may appear to benefit more because those students received more time, stronger delivery, or added support. For example, a reading program may appear to work better for students below grade level because those students in the treated group also received more small-group instruction, more time on the program, or closer monitoring from reading specialists. If effects are smaller for Latine students, leaders might examine whether implementation differed across the schools serving those students and, where relevant, whether the program provided adequate multilingual supports. The subgroup finding alone cannot establish which condition explains the difference.

Who or what produced the evidence? Ask whether the tool was tested with actual students and educators or only with simulated users, AI-generated responses, benchmark tests, synthetic data, or developer-created scenarios. Tests using simulations or AI-generated responses can show the tool performs a function, produces accurate answers, or behaves as intended under selected conditions, but such tests are not evidence that using the tool improves outcomes for students. Evidence of impact requires measuring outcomes for real students who experienced the tool and comparing those outcomes with what would have occurred under a credible alternative.

How was the tool used, and what was it compared to? First investigate how the tool was used: what changed for students or educators when the tool was introduced? Did students use the adaptive math platform during classroom instruction, or for homework? Did teachers use the tool’s recommendations to form groups or assign lessons? Then identify the comparison group and ask what students and educators in that group experienced. The estimated impact reflects the difference between those two conditions, not the effect of the algorithm in isolation.

What could go wrong for students or educators? Personalized and algorithmic tools can create errors or tradeoffs. A tutoring screener may miss students who need help or assign tutoring to students who do not need it. An adaptive platform may place students in content that is too easy, too hard, or disconnected from classroom instruction. A tool may also increase screen time, add teacher burden, or displace essential instruction. Ask whether the evidence examines these risks and whether the benefits are large enough to justify the tradeoffs.

Was the tool tested in a setting like the context in which the evidence is being used? A tool tested in one district may not work the same way elsewhere (see Principle 7: Contextual Considerations). Results may change when students, assessments, data systems, or staffing differ. Some schools may have less capacity to use the tool effectively. When the evidence comes from a setting unlike the district’s own, leaders should consider what factors make it more or less likely that they see similar results.