Research Approach
How WIN Examines Complex Questions
Complex human problems rarely yield to one source, one discipline, or one explanation. WIN therefore uses a structured research approach designed to make difficult questions easier to examine, compare, challenge, revise, and preserve.
The approach combines evidence evaluation, competing explanations, systems thinking, causal reasoning, failure analysis, measurement, long-term evaluation, and revision.
It is not a claim that every WIN document constitutes formal academic research. It is a disciplined method for improving the quality of reasoning, exposing important assumptions, distinguishing stronger conclusions from weaker ones, and making important claims easier to test.
WIN’s research process can be summarized in six recurring steps:
1. Define the Question
2. Gather Competing Explanations
3. Map the System
4. Examine Failure Modes
5. Separate Confidence Levels
6. Preserve the Reasoning
These steps are not always performed once in a rigid sequence. New evidence discovered at a later stage can require an earlier question, assumption, definition, or explanation to be reconsidered.
1. Define the Question
Good analysis begins by defining what is actually being examined.
Important terms must be clear enough that different people can understand the question in substantially the same way. Scope also matters: the relevant population, conditions, time horizon, intended outcome, comparison, and known limitations need to be identified before conclusions become fixed.
A poorly framed question can produce confident answers to the wrong problem.
WIN therefore asks what is actually being claimed, what would count as relevant evidence, what boundaries apply, and whether the question contains assumptions that need to be examined separately.
2. Gather Competing Explanations
WIN does not begin by assuming that the first plausible explanation is correct.
Relevant evidence, competing interpretations, contradictory examples, alternative causal models, and explanations from different disciplines may all need consideration.
The purpose is not to manufacture artificial balance between strong and weak claims. It is to reduce the risk of selecting only information that supports an existing belief.
Competing explanations become particularly important when several different mechanisms could produce similar visible outcomes.
WIN’s standards for evaluating the strength and limitations of evidence are explained in How We Evaluate Evidence.
3. Map the System
Many human problems involve interacting causes rather than a single isolated variable.
WIN examines relevant actors, incentives, information flows, constraints, opportunities, institutional rules, feedback loops, delays, dependencies, and unintended consequences that may influence the outcome.
Mapping these relationships helps distinguish a visible symptom from the conditions that repeatedly generate it.
The purpose is not complexity for its own sake. A useful system map makes an explanation easier to examine by identifying relationships that may be strong, weak, incomplete, uncertain, or misleading.
This broader method is developed further in Systems Thinking.
4. Examine Failure Modes
A proposal is evaluated not only by how it might succeed, but also by how it might fail.
WIN asks what assumptions could be wrong, what incentives could be distorted, what costs could be transferred elsewhere, what safeguards could fail, what implementation problems could occur, and what unintended consequences might appear later.
Failure analysis is especially important when an intervention initially appears successful.
A visible improvement can conceal delayed costs, displacement of the problem, dependency, institutional fragility, measurement distortion, or conditions that eventually recreate the original failure.
5. Separate Confidence Levels
Not every conclusion deserves the same level of confidence.
WIN distinguishes among strongly supported conclusions, provisional interpretations, plausible hypotheses, working frameworks, unresolved questions, and future proposals when those distinctions are important.
Confidence increases when independent evidence converges, predictions or expectations survive testing, competing explanations weaken, findings remain reproducible where appropriate, and important contradictory evidence has been addressed.
Confidence decreases when evidence conflicts, assumptions fail, expected results do not occur, important variables were omitted, findings fail to reproduce, or a better explanation emerges.
Making confidence levels visible allows knowledge to improve without requiring false certainty.
6. Preserve the Reasoning
Useful conclusions become more durable when future reviewers can reconstruct how they were reached.
Important assumptions, evidence, definitions, competing explanations, dissenting views, revisions, limitations, confidence levels, and consequential decisions are preserved where practical.
This makes knowledge easier to audit, teach, challenge, update, and transfer when people or technologies change.
Preservation is not intended to freeze a conclusion permanently. It preserves enough of the reasoning for improvement to continue.
Evidence Before Preference
WIN’s research process is designed to prevent a preferred conclusion from determining which evidence is allowed to matter.
Evidence that supports an explanation is considered alongside evidence that weakens it. Relevant contradictory evidence is not excluded merely because it creates difficulty for an existing framework.
The appropriate question is not “How can we prove this idea?”
It is “What explanation best accounts for the relevant evidence, and what evidence could show that explanation to be incomplete or wrong?”
This principle is institutionalized through WIN’s Prove Us Wrong standard.
Multiple Disciplines Can Be Necessary
Complex human questions can cross disciplinary boundaries.
Psychology may help explain behavior. Economics may illuminate incentives. Education research may address learning. History may reveal recurring patterns. Law may define institutional constraints. Statistics may help evaluate evidence. Computer science may affect technical feasibility. Ethics may expose consequences that a purely technical analysis overlooks.
WIN therefore draws on relevant expertise across disciplines rather than assuming that one field can answer every dimension of a complex problem.
Interdisciplinary research does not eliminate the need for disciplinary competence. Questions requiring specialized expertise should receive appropriately qualified review.
Distinguish Observation From Interpretation
What was observed and what the observation means are not always the same thing.
WIN separates, where important, the underlying observation or data from the interpretation placed upon it.
Two researchers can agree that an event occurred while disagreeing about why it occurred, how representative it is, what caused it, or what conclusions it supports.
Keeping observation and interpretation distinct makes disagreement easier to analyze.
Correlation Does Not Automatically Establish Cause
Two conditions can occur together without one causing the other.
They may share another cause, influence one another in both directions, occur together by coincidence, or reflect selection, measurement, or contextual effects.
WIN therefore examines timing, mechanisms, competing explanations, comparison cases, relevant research, and other evidence appropriate to the causal claim being considered.
The strength of causal language remains proportionate to the strength of the evidence.
Search for Real Root Causes
WIN uses the term Real Root Cause for the deeper mechanisms or conditions that meaningfully generate, sustain, or repeatedly recreate a problem.
A visible symptom is not automatically its own cause.
At the same time, a deeper-sounding explanation is not automatically a Real Root Cause. A proposed root cause must still account for relevant evidence, survive comparison with plausible alternatives, explain important observations, and remain open to correction.
Root-cause analysis is therefore an evidence problem, not merely a search for the deepest possible description.
WIN explains this concept more fully in Real Root Cause.
Examine Incentives
Human systems often behave differently from their formal rules because people respond to incentives.
WIN examines what behavior a system rewards, discourages, tolerates, makes easier, makes costly, or allows people to avoid.
This includes unintended incentives. A policy intended to produce one outcome can encourage behavior that undermines the original purpose.
Understanding incentives does not imply that every person responds identically. It identifies pressures that can influence behavior across a system.
Examine Feedback Loops
Causes and effects can reinforce or counteract one another over time.
A reinforcing loop can amplify a pattern. A balancing loop can resist change. Delayed feedback can make an intervention appear successful before important consequences become visible.
WIN examines relevant feedback loops because short-term observations can be misleading when system effects unfold over longer periods.
The role of recurring causal patterns is examined further in Developmental Loops.
Consider Time Horizons
An intervention that looks successful after one week may look different after one year.
WIN distinguishes immediate outputs from longer-term outcomes and examines whether benefits endure, costs accumulate, behavior adapts, incentives change, or new dependencies emerge.
The appropriate time horizon depends on the claim being evaluated.
Long-term thinking does not mean ignoring immediate evidence. It means avoiding the assumption that immediate evidence tells the entire story.
See Long-Horizon Thinking for WIN’s broader treatment of this principle.
Use Historical Evidence Carefully
Historical evidence can reveal recurring patterns, institutional failures, incentive structures, unintended consequences, and long-term effects that short studies cannot easily show.
Historical comparisons also have limitations.
Different periods may involve different populations, technologies, institutions, laws, economic conditions, cultural conditions, and information environments.
WIN therefore uses history as evidence without assuming that superficial similarity proves identical causes or predicts identical outcomes.
Distinguish Anecdotes From General Evidence
Individual experiences and examples can identify possibilities, reveal unexpected problems, expose failure modes, or suggest hypotheses worth investigating.
They usually cannot establish how common an effect is or whether the same explanation applies broadly.
WIN therefore treats anecdotes as potentially useful information while keeping conclusions proportionate to what they can establish.
Measurement Must Match the Question
Measurement is useful only when it meaningfully represents what WIN is trying to understand.
Easily counted activity—such as attendance, website traffic, registrations, course completions, documents produced, volunteer hours, or money spent—does not automatically demonstrate meaningful improvement.
When evaluating consequential programs, WIN distinguishes among inputs, activities, outputs, outcomes, and longer-term impact.
The measurement method must remain connected to the intended outcome.
WIN’s detailed measurement framework is explained in Measuring Progress.
Predictions and Expectations Improve Testability
A research explanation becomes easier to evaluate when it produces expectations that can be compared with later observations.
WIN identifies, where practical, what an explanation would lead us to expect under relevant conditions and what observations would weaken confidence in that explanation.
Failed predictions do not automatically identify why an explanation failed, but they provide information that requires examination.
Look for Unintended Consequences
A proposed improvement can solve one problem while creating another.
WIN examines relevant effects on incentives, privacy, safety, accessibility, workload, cost, fairness, trust, institutional resilience, technical complexity, and other parts of the system that may be affected.
A favorable result on one preferred measure does not automatically establish overall success.
Pilots Are Tests, Not Demonstrations
A pilot is useful when it can reveal weaknesses before expansion.
WIN’s future pilots are intended to test assumptions, implementation, usability, measurement, safeguards, administrative burden, unintended consequences, and whether expected outcomes actually occur.
A pilot that reveals significant problems has produced valuable information.
WIN does not define a pilot as successful merely because it was completed or because participants responded positively.
Compare Expectation With Result
Evaluation becomes stronger when important expectations are identified before the final result is known.
WIN compares what was expected with what actually occurred and examines the difference.
This reduces the temptation to reinterpret every possible result as confirmation of the original idea.
Failure Is Evidence
Failure does not automatically establish which explanation is correct, but it changes what must be investigated.
A failed prediction may expose a weak assumption. A poor outcome may reveal a flawed concept, implementation failure, inadequate measurement, unexpected conditions, or an unintended consequence.
WIN analyzes failure to determine what information it provides rather than treating it solely as embarrassment or blame.
The broader institutional learning process is explained in Learning From Failure.
Success Also Requires Examination
Favorable results deserve analysis too.
An intervention may appear successful because the underlying idea worked, because implementation was unusually strong, because external conditions were favorable, because another variable produced the improvement, or because of ordinary variation.
WIN therefore asks whether successful results are reproducible, durable, causally understandable, and likely to remain useful under the conditions in which expansion is being considered.
This is part of WIN’s broader definition of What Success Looks Like.
Independent Review Becomes More Important as Claims Become More Consequential
Internal confidence is not a substitute for qualified external examination.
As WIN’s research, education, measurement, technology, or public claims become more consequential, appropriate independent academic, technical, statistical, legal, ethical, accessibility, security, or other professional review becomes increasingly important.
External review does not guarantee correctness. It provides additional opportunities to expose assumptions, methodological weaknesses, missing evidence, and competing explanations.
Preserve Disagreement
Not every legitimate disagreement needs to be forced into premature consensus.
When qualified reviewers interpret evidence differently, preserving the substance of the disagreement can be more informative than recording only the conclusion selected by the organization.
Important dissenting reasoning may later become valuable when new evidence appears.
Revise When the Evidence Requires It
WIN’s research conclusions are not protected from revision merely because substantial work has already been invested in them.
New evidence may strengthen a conclusion, narrow it, expand it, change its confidence level, expose a missing variable, reveal a better explanation, or overturn it.
Revision is therefore part of the research process.
The purpose of preserving earlier reasoning is not to prevent change. It is to make later change understandable.
A Research Process Designed to Improve
No method eliminates error.
The practical standard is whether a research process makes error easier to detect, competing explanations easier to compare, uncertainty easier to see, important claims easier to challenge, and correction easier to perform.
For WIN, research is therefore not a one-time search for final answers. It is a disciplined cycle:
Question → Evidence → Competing Explanations → System Analysis → Testing → Measurement → Failure Analysis → Revision → Preserved Knowledge
The cycle can repeat whenever new evidence or better reasoning requires it.
The goal is not to create an institution that never changes its conclusions.
The goal is to create an institution capable of learning why a conclusion should change—and preserving enough evidence and reasoning for future people to understand why.
Continue Exploring
How We Evaluate Evidence
Prove Us Wrong
Measuring Progress
Learning From Failure
Real Root Cause
Systems Thinking
Developmental Loops
Long-Horizon Thinking
What Success Looks Like