Measuring Progress

Progress Must Mean More Than Activity

WIN distinguishes between doing things and producing worthwhile results.

Website visits, registrations, meetings, volunteer hours, courses completed, partnerships formed, documents published, and money spent can describe activity. They do not by themselves demonstrate that people learned more, made better decisions, avoided harm, developed greater capability, or produced lasting improvement.

WIN’s measurement approach therefore begins with a basic question:

What meaningful change was the activity intended to produce, and what evidence would show whether that change actually occurred?

This distinction between activity and meaningful outcomes is central to WIN’s What Success Looks Like framework.

Start With the Intended Outcome

Measurement begins before a program is declared successful.

For consequential activities, WIN identifies the intended outcome clearly enough to determine what evidence would support success, what evidence would indicate limited or no effect, and what evidence could reveal unintended harm.

Clear outcomes make it harder to redefine success after results are known.

Inputs, Activities, Outputs, Outcomes, and Impact

WIN distinguishes several levels of measurement because they answer different questions.

Inputs are resources committed to the work, such as time, funding, expertise, technology, facilities, or information.

Activities are actions performed, such as developing material, conducting training, holding meetings, performing reviews, or operating a pilot.

Outputs are immediate products of those activities, such as pages published, courses completed, participants served, assessments administered, or reports produced.

Outcomes are meaningful changes associated with the work, such as improved understanding, better reasoning, stronger skills, fewer errors, safer behavior, or more effective decisions.

Impact concerns broader or longer-term effects that may develop beyond immediate outcomes.

WIN does not treat movement at one level as automatic proof of success at another.

Establish a Baseline

Meaningful change is difficult to evaluate without understanding the starting condition.

When practical, WIN establishes relevant baseline information before a consequential intervention or pilot begins. The appropriate baseline depends on what is being examined and may involve knowledge, behavior, performance, error rates, existing conditions, or other relevant measures.

Baselines make later comparisons more informative and reduce the temptation to interpret every observed result as improvement.

Measure What Matters

Measurement should focus on variables that meaningfully relate to the intended objective rather than selecting indicators merely because they are easy to count.

A convenient number can still be a poor measure.

WIN therefore examines whether an indicator actually represents the outcome of interest, whether important dimensions are being omitted, and whether concentrating on the metric could unintentionally distort behavior.

Leading and Lagging Indicators

Some indicators appear before the ultimate outcome is known. Others can be measured only after sufficient time has passed.

Leading indicators may provide early evidence that a process is moving in an expected direction.

Lagging indicators may show whether the intended result ultimately occurred or endured.

WIN may use both where appropriate while avoiding the assumption that an encouraging early indicator guarantees a successful long-term result.

Quantitative and Qualitative Evidence

Numbers can reveal patterns, frequencies, changes, comparisons, and trends. They can be extremely useful when collected and interpreted appropriately.

Qualitative evidence can reveal information that numbers alone may miss, including misunderstandings, participant experiences, implementation problems, unexpected effects, contextual differences, and reasons a program did or did not function as expected.

WIN may use both quantitative and qualitative evidence when the combination provides a more accurate understanding.

Define Success Before Seeing the Results

For consequential pilots or evaluations, WIN seeks to define important success criteria before examining final results when practical.

This reduces the risk of moving the goalposts after the evidence is known.

Success criteria may include target outcomes, acceptable error or failure rates, safety boundaries, implementation requirements, durability, cost considerations, or other measures relevant to the activity.

Define Failure Too

Responsible evaluation also requires recognizing conditions under which an approach has not worked well enough.

Failure may include little or no meaningful improvement, unacceptable unintended consequences, excessive cost, poor reliability, inability to reproduce results, inadequate safeguards, or other outcomes inconsistent with the purpose of the intervention.

Identifying failure criteria does not assume failure will occur. It makes honest evaluation possible if it does.

Do Not Move the Goalposts

WIN does not redefine success merely because actual results are disappointing.

If an intervention fails to meet important predetermined criteria, that result becomes information for investigation and correction. WIN may revise the intervention and test it again, but the original result is not transformed into success by changing the definition afterward.

Use Comparison Where Appropriate

Observing change does not automatically establish what caused the change.

Where practical and ethically appropriate, WIN may use comparisons across time, groups, conditions, implementations, or other relevant reference points to help distinguish the effects of an intervention from unrelated changes.

The appropriate comparison method will depend on the question, available data, ethical constraints, feasibility, and the strength of the conclusion being considered.

These comparisons must be interpreted according to the strength and limitations of the available evidence, consistent with WIN’s How We Evaluate Evidence approach.

Measure Development, Not Just Completion

Completing a course, attending a program, reading material, or receiving a certificate does not by itself demonstrate development.

Where the objective involves learning or capability, WIN will increasingly examine whether participants can understand, apply, retain, and appropriately transfer what they learned.

Completion may be useful administrative information, but it is not automatically an outcome.

Look for Unintended Consequences

A program can improve one measure while creating a different problem.

WIN therefore examines important unintended effects, including changes in incentives, behavior, workload, privacy, safety, accessibility, trust, fairness, administrative burden, or other conditions relevant to the intervention.

An intended benefit does not automatically outweigh every unintended cost.

Watch for Measurement Distortion

Metrics can change behavior simply because people know they are being measured.

Participants or administrators may optimize for a visible number rather than the underlying objective. A measure can also become less informative when achieving the metric becomes more important than achieving the purpose it was supposed to represent.

WIN will therefore interpret metrics in context and use multiple indicators when a single measure could be misleading.

Pilots Are Opportunities to Learn

WIN uses controlled pilots, when practical, to learn before broad expansion.

A pilot can reveal whether people understand the material, whether procedures work, whether measurements are useful, whether technology is reliable, whether safeguards are adequate, and whether unexpected problems appear.

A pilot that reveals serious weaknesses can still provide valuable information.

A disappointing pilot can therefore be successful as a learning process even when the intervention itself was not successful. This principle is examined further in Learning From Failure.

Measure Implementation Separately From the Idea

Poor results can have different causes.

An underlying idea may be weak, or a potentially useful idea may have been implemented badly. Participants may not have received the intended intervention. Staff may not have followed procedures. Technology may have failed. Conditions may have differed substantially from those assumed in the design.

WIN therefore examines both the result and whether the intervention was actually implemented as intended.

This distinction does not excuse failure. It helps identify what needs to be corrected.

Account for Time

Some effects appear quickly. Others require months or years before they can reasonably be evaluated.

WIN matches measurement periods to the nature of the outcome rather than assuming every worthwhile effect should appear immediately.

Where durability matters, follow-up measurement may be needed to determine whether an initial improvement persists.

Privacy-Conscious Measurement

Measurement does not justify collecting unlimited information about people.

WIN should collect and retain only information reasonably necessary and proportionate to legitimate measurement, educational, administrative, security, or legal purposes.

As future programs become more consequential, measurement practices will incorporate appropriate privacy disclosures, access controls, data minimization, retention practices, security safeguards, and professional review.

Accuracy and Data Quality

Conclusions cannot be stronger than the information supporting them.

WIN considers factors such as missing information, inconsistent definitions, measurement error, incomplete records, sampling limitations, inaccurate entries, unreliable instruments, and other conditions that can distort results.

Material limitations will be considered when interpreting or communicating findings.

Independent Review

The more consequential a claim becomes, the more valuable qualified independent examination can become.

As WIN’s work becomes more consequential, it should seek appropriate external academic, measurement, statistical, technical, legal, ethical, accessibility, or other professional review when the importance and risk of the work justify it.

Independent review does not guarantee correctness, but it can expose assumptions and weaknesses that internal review may miss.

Transparent Reporting

WIN’s reporting will distinguish, where important, among what was attempted, what was measured, what actually happened, what remains uncertain, what limitations affected interpretation, and what changes follow from the evidence.

Reporting will not treat activity statistics as outcome evidence merely because activity numbers are easier to present.

Appropriate privacy, confidentiality, security, and legal restrictions will continue to apply to public reporting.

Learning From Negative Results

Negative or disappointing findings are part of responsible research and development.

When an intervention does not produce the expected result, WIN examines possible explanations rather than concealing the result or automatically blaming participants.

A negative result may reveal a weak theory, poor implementation, an inadequate measurement method, an unexpected contextual factor, an unintended incentive, or another condition requiring investigation.

The objective is not to protect an intervention from failure. It is to learn accurately from whatever the evidence shows. See Learning From Failure.

Measurement Should Improve the System

Measurement has little value if results are collected and then ignored.

WIN uses credible findings to improve educational material, procedures, safeguards, technology, measurement methods, governance, training, or other relevant components.

The purpose of evaluation is not merely to generate reports. It is to make better decisions.

Living With Imperfect Information

Human development and complex social systems rarely provide perfect experimental conditions.

WIN does not require impossible certainty before learning from evidence, but it also does not treat limited evidence as stronger than it is.

Conclusions will be proportionate to the quality of the available evidence, the strength of the research design, the size and relevance of observed effects, competing explanations, and important uncertainties.

What WIN Should Never Count as Proof

The following do not, by themselves, demonstrate that a WIN program or idea works:

Website traffic.

Enrollment.

Course completion.

Certificates issued.

Volunteer hours.

Money raised or spent.

Media attention.

Positive testimonials.

Organizational growth.

Agreement with WIN.

Each may provide useful information for a particular purpose, but none automatically demonstrates meaningful improvement or establishes causation.

A Practical Measurement Cycle

WIN’s measurement process can be summarized as a repeating cycle:

1. Define the intended outcome.

2. Understand the starting condition.

3. Select meaningful indicators and methods.

4. Establish important success and failure criteria where practical.

5. Implement on an appropriate scale.

6. Measure both implementation and results.

7. Examine unintended consequences and alternative explanations.

8. Report material findings and limitations accurately.

9. Correct weaknesses.

10. Test again when warranted.

11. Expand only when the evidence and organizational readiness justify expansion.

Measurement is therefore part of a learning process rather than a final judgment performed after development is finished.

Readiness Before Advancement

Measurement is also one of WIN’s readiness gates.

Before a consequential pilot or program expands, WIN will examine whether the intended outcome is sufficiently clear, whether meaningful measurement is possible, whether important safeguards are functioning, whether results justify further development, and whether significant weaknesses have been addressed.

A predetermined schedule does not override material evidence that a system is not ready.

The Standard WIN Should Apply to Itself

WIN expects its own claims and programs to remain subject to the same demand for evidence that it asks others to accept.

If evidence contradicts an important WIN conclusion, the appropriate response is to examine the evidence, identify the source of the discrepancy, and correct the conclusion when warranted.

Institutional pride is not evidence.

WIN’s Prove Us Wrong principle applies this standard directly to WIN’s own claims and conclusions.

Progress That Deserves Its Name

WIN considers progress meaningful when evidence indicates that people, programs, or systems have improved in ways connected to the intended objective and when those improvements do not depend on ignoring significant costs or unintended harm.

That standard is more demanding than counting activity. It is also more useful.

WIN’s long-term objective is not to become better at reporting impressive numbers. It is to become better at determining what actually works, what does not, why, under what conditions, and what should change as a result.

Continue Exploring

What Success Looks Like
How We Evaluate Evidence
Learning From Failure
Research Approach
Frameworks Explained
Future Learning Platform
Prove Us Wrong