Jason Cook DesignStart a conversation

How to Tell Whether a Consultant's Diagnosis Is Any Good

How do you know whether you can trust a consultant's report? Six questions, adapted from the standards auditors and intelligence analysts use, that you can ask without being an expert.

diagnosisevidenceprocess
A page from the sample report headed "What we found", in which each finding states a claim, its evidence, a confidence level, and what it means, in front of the page on assumptions, unknowns and checks.
FIG. 1A findings page from the sample report: claim, evidence, confidence, and what it means.

You hired a consultant, and the report arrived. It's well designed, it sounds sure of itself, and it tells you what to do next. How do you know whether you can trust it?

You can't, if all that stands behind it is the consultant's reputation or confidence. A diagnostic is meant to produce an accurate recommendation, and accuracy is something you're entitled to check. A report that asks you to take it on trust hasn't finished the work.

Surprisingly, there's no written standard I'm aware of to judge a consultant's report, so I started with the standards auditors and intelligence analysts use to defend their conclusions to people who weren't in the room, then adapted them for small to medium businesses.

Six questions to ask

Each comes from one of those standards, and none of them requires you to be an expert in the subject.

  1. What was each finding measured against, what was found, and what is the evidence? The US Government Accountability Office builds an audit finding from four parts: the criteria, the condition, the cause, and the effect.1 If a finding is only a conclusion, you can't see any of them.
  2. Is the evidence relevant and reliable, or just plentiful? The same standards say a large volume of evidence doesn't make up for weak evidence. Ask what each source is and how it was checked.
  3. How confident is the consultant in each finding, and why? A good report says so, gives a reason, and uses each label the same way throughout. The US intelligence community's analytic standards require that, and they keep confidence in the evidence separate from the likelihood of an event.2
  4. Where are the assumptions, and what changes if one is wrong? A CIA tradecraft primer calls the habit a key assumptions check: list what the conclusion rests on, rate each item, and revisit the list before you finalize.3
  5. Where does the evidence run out? The audit standards want a limitation stated next to the finding it affects, along with how many cases were examined.
  6. Does the recommendation follow from the findings, state its conditions, and allow for "don't proceed"? Ontario's agriculture ministry guidance on feasibility sets minimum criteria before the work starts and puts the rule plainly: either the project meets them or it's abandoned.4

My first-phase report is built to answer these questions directly. In the sample report on this site, written for a fictional agency, each finding states its claim, its evidence, how confident I am and why, and what it means for the owner. One finding is a hypothesis the evidence ruled out. Assumptions and unknowns have their own section, with how each unknown will be handled. The recommendation lists its conditions and what would change it.

Read the evidence before you read the recommendation. If it isn't there, that tells you what you needed to know.