Chapter 13
A Taxonomy of Verification Obstacles: Six States
13.1 Why a taxonomy is needed
“Cannot be verified” covers states of different kinds. Because the remedies differ, they must be treated separately. This taxonomy is an organization contributed by this text.
When work stalls at stage 3 of the five stages of Chapter 11, availability, one cannot decide whether to spend effort or abandon the attempt without being able to say what exactly is stuck.
13.2 Six categories
| Category | State |
Correct remedy |
| (i) No measurement exists | nobody anywhere measures the quantity |
conduct an original survey; secondary data will never fill the gap |
| (ii) Ill-posed hypothesis | the theory does not fix the sign |
refine the theory; more data does not help |
| (iii) Not identified | the data exist but confounding cannot be broken |
better design; find exogenous variation or a control group |
| (iv) Access constraint | it exists but cannot be obtained |
open a route, or abandon on legal or technical grounds |
| (v) A choice in aggregation | it is surveyed but does not appear in the published tables |
request the tabulation, or apply to use anonymized microdata |
| (vi) The adjustment does not operate | data are ample, but the explanatory variable does not vary over the range the theory assumes |
no route resolves it |
The role of the taxonomy is to separate what effort can move from what it cannot.
(iii), (iv) and (v) can move with effort or ingenuity: (iii) with design, (iv) with a route, (v) with a change in what is published.
(i), (ii) and (vi) do not move. (i) cannot be filled without bringing the measurement into existence; (ii) does not fix a sign however much data is added; and under (vi), so long as the explanatory variable does not vary, no increase in sample size makes a test possible.
Remark 13.1 ((vi) is not a negative result). (vi) is a defect neither of the theory nor of the measurement. It is the fact that the adjustment mechanism the theory assumes does not operate in reality.
That a theoretical prediction fails to appear is itself an observation. In Part IV the statement “fee rates are not adjusted to the risk of the outcome” is such a case (Remark 15.6).
In terms of the layers of Section 6.3, prices being set by custom is a phenomenon of the institutional layer. The picture is one in which a theory premised on optimization fails to operate in the face of custom.
Remark 13.2 (Impossibility of aggregation differs from a choice in aggregation). Under (v) the quantity is merely not tabulated; it is in the sample. By contrast, there are cases where the unit of observation does not correspond to a party, so a quantity per party cannot be recovered. The identifiers of Chapter 10 are such a case (Remark 10.5).
That case is not resolved by a change in what is published. It is closer to (i) than to (v), but differs in that the quantity is not unmeasured; the unit of measurement differs from the unit of the question.
13.3 The taxonomy also serves as a test before starting
The taxonomy organizes, after the fact, why a verification did not complete; it also serves as a test before a survey is begun.
At stage 3 of Chapter 11 it is the criterion for judging whether the quantity can be obtained at the granularity required. The feasibility assessment of Chapter 17 is a worked instance of this use.