Most analysis fails before it starts, not because of flawed methodology or bad data, but because of poorly framed questions.

The analyst pulls data, builds charts, runs regressions, and produces a polished deliverable. But the deliverable doesn’t inform the decision it was supposed to inform. The stakeholder looks at it, nods politely, and proceeds based on intuition anyway. The analysis was technically competent and strategically useless.

This pattern is so common it has a name: the Type III Error, providing the right answer to the wrong problem.1 Industry estimates suggest that 60% to 85% of analytics and AI projects fail to achieve their stated goals, with failure rates exceeding 90% in analytically immature organizations.2 The failure usually isn’t computational; it’s conceptual. Better methodology can’t fix a bad question.

The Question Problem

Questions that drive analysis often fail in predictable ways.

Too vague. “How are we doing on customer satisfaction?” is a topic, not a question. It doesn’t specify what aspect of satisfaction, which customers, what timeframe, or what would constitute a useful answer. Analysis in response to vague questions produces vague deliverables.

Disconnected from decisions. “What were our sales last quarter by region?” is precise, but what decision does it inform? If the answer doesn’t change what the organization does, the analysis is reporting, not decision support.

Predetermined conclusions. “Can you find data to support expanding into the Northeast?” isn’t a question; it’s a request for ammunition. This is confirmation bias embedded in the inquiry itself, guaranteeing that the team will find supporting evidence while missing disconfirming data.

Wrong level of specificity. Questions can be too broad (“Why is revenue down?”) or too narrow (“Why did customer #4,572 churn?”). The right level matches the decision being made and the actions available.

The quality of analysis is bounded by the quality of the question that initiated it. As statistician John Tukey observed, “Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.”3

The Proxy Trap

Even well-framed questions can fail when the metrics chosen to answer them diverge from what actually matters.

This is Goodhart’s Law in action: “When a measure becomes a target, it ceases to be a good measure.”4 We rarely measure the thing itself; we measure a proxy. And proxies can diverge from reality in dangerous ways.

A landmark example comes from healthcare algorithms. Researchers found that an algorithm designed to identify high-risk patients for care management used healthcare costs as a proxy for health needs. The implicit assumption: sicker people cost more money. But due to systemic barriers to access, Black patients at a given level of sickness generated significantly lower costs than white patients with the same conditions. The algorithm systematically deprioritized sicker Black patients because it was optimizing for the wrong variable: cost rather than health.5

The math was correct. The model optimized perfectly. The question was wrong.

Similar dynamics appear in business contexts constantly. A call center measured on calls handled per hour will handle calls quickly, even if that means cutting conversations short and leaving problems unresolved. A sales team measured on deals closed will close deals, even if those deals are unprofitable or poorly fitted to customer needs. The metric improves while the outcome it was supposed to represent degrades.

Characteristics of Good Analytical Questions

Questions that lead to useful analysis share certain characteristics:

Tied to a decision. Good questions are connected to choices the organization faces. “Should we expand our support team or invest in self-service tools?” creates clear analytical direction. The analysis has a job to do: inform that specific decision.

Specific enough to answer. Good questions define scope: which customers, which timeframe, which metrics. This specificity makes analysis tractable and results interpretable.

Answerable with available data. Good questions are grounded in what can actually be measured. This doesn’t mean avoiding hard questions; it means being realistic about what evidence can and cannot establish.

Open to multiple answers. Good questions don’t presuppose conclusions. “What factors are contributing to churn?” invites discovery. “Prove that our new feature reduced churn” does not.

The Art of Question Development

Good analytical questions are developed, not received. The initial request from a stakeholder is usually a starting point, not a final brief.

The CRISP-DM methodology, the most widely used framework for data science projects, places “Business Understanding” as its first phase for exactly this reason.6 Yet this phase is consistently the most neglected. Teams jump straight to data preparation or modeling, assuming they know the goal. This is where the Type III error gets baked in.

Probe for the decision. When someone asks for analysis, ask what decision it will inform. “What will you do differently based on what you learn?” reveals whether there’s a real decision at stake.

Clarify scope and definitions. What exactly do we mean by “customer”? Active customers? All-time customers? Which segments? Ambiguity in definitions creates ambiguity in results.

Identify what would change minds. What evidence would lead to one decision versus another? If a stakeholder would proceed regardless of what the analysis shows, the analysis isn’t needed.

Test feasibility. Before committing to analysis, assess whether the question can be answered with available data and methods. Better to narrow an unanswerable question than to produce unreliable answers.

When Questions Change

Analysis often reveals that the original question was wrong, not in its framing, but in its assumptions.

The question assumed the problem was X, but the data suggests the problem is Y. The question focused on one segment, but another segment turns out to be more relevant. The question asked about causes, but the data can only show correlations.

Good analysts surface these discoveries rather than forcing answers to the original question. The willingness to say “we asked the wrong question, and here’s what we should ask instead” is a mark of analytical maturity.

This requires psychological safety. If analysts fear being blamed for not answering the original question, they’ll force-fit answers rather than redirecting. Organizations that get value from analysis create space for question evolution.

The Question as Foundation

Analysis is only as valuable as the question it answers. Sophisticated methods applied to poorly framed questions produce sophisticated irrelevance.

The best analysts spend significant time on question development: understanding the decision context, clarifying scope and definitions, testing feasibility, and aligning with stakeholders. This investment makes everything that follows more valuable.

Before asking “what does the data show?”, ask “what question are we trying to answer, and is it the right one?”

The question is the foundation. Build on solid ground.

Citations

1 A.W. Kimball, "Errors of the Third Kind in Statistical Consulting," Journal of the American Statistical Association, 1957.

2 Melbourne Business School, "Why Do Analytics and AI Projects Fail?", 2024; Gartner estimates on data governance initiative failures.

3 John Tukey, "The Future of Data Analysis," The Annals of Mathematical Statistics, 1962.

4 Charles Goodhart, 1975; as paraphrased by Marilyn Strathern.

5 Obermeyer et al., "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations," Science, 2019.

6 CRISP-DM (Cross-Industry Standard Process for Data Mining), developed by a consortium including IBM and NCR, 1996.