Standard-Setting
设定
Paper 1 said that the establishment of a discipline draws divisions, and that divisions leave remainders. This paper says that once fields are divided, every discipline has premises it needs in order to run which its current internal procedures cannot settle without circularity, such as the threshold of "significance" in statistics. These premises must always be held steady, so the position that holds them cannot be empty; what holds them can be standards, instruments, classifications, indicators, training and default parameters (S9). The position that publicly questions these premises, however, can be empty; when the questioning position is empty, the setting position runs all the same, and that is enough to show the two positions are not one (S10). The paper first defines "scope of warrant" (S11), then gives four conditions for occupying the questioning position: jurisdiction, right of response, right of review, independence; the four are conjunctive, and each can fail on its own (S12). Each case is recorded under three items, which are not steps in a progression: whether the position is occupied; whether questioning took place, and whether the answer reached the matter questioned; what the outcome is, and whether it entered everyday procedure (S13). That a discipline's own foundational research and revision working groups touch a premise does not mean they occupy the questioning position; this is decided by the three records (S14). Six propositions in all (S9 to S14), stated at the end of each section and listed together at the end.
Writing Declaration: This paper was co-drafted with Claude (Anthropic). All intellectual decisions, framework design, and final editorial judgments were made by the author.
Introduction
Whether the result of a study counts as a "finding" is, in many disciplines, decided by a threshold: a p value below 0.05. The source of this threshold is often traced to Fisher. In a textbook of statistical methods he said that the point at which a normal distribution deviates from its mean by about two standard deviations corresponds to p = 0.05: "it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not" (Fisher 1925, p.47). The word "convenient" is honest: it does not say that this point was derived from statistics.
From then on the threshold held many things steady: whether a paper can be published, whether a drug can go on to the next phase of trials, whether an effect should be written into the textbooks. It has also been questioned all along. In 2016 the American Statistical Association issued a statement, one of whose principles says: "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold." (Wasserstein & Lazar 2016, Principle 3). In 2018 more than seventy researchers jointly proposed changing the default threshold for new discoveries to 0.005 (Benjamin et al. 2018). Yet where the threshold is set depends on how the cost of wrongly judging that there is an effect compares with the cost of wrongly judging that there is none, and statistics by itself cannot supply that comparison.
There are two positions here. One position holds the threshold steady, so that no paper has to argue again from the beginning about "what counts as a finding"; it is always occupied and always running. The other position asks whether the threshold ought to be here, and on what grounds it is here; this position can be empty, and when someone does ask, the one asking may not be able to require anyone to answer. This paper is about these two positions.
The reading follows the Prequel. This paper concerns the second domain, standards that act on persons (Prequel §5). "Setting cannot be empty" holds for every discipline, and some of the examples borrowed in §1 come from the first domain, measurement and instruments; what this paper registers and checks are only the lines on the normative ledger that act on persons. §1 says that the setting position cannot be empty; §2, that the questioning position can be empty; §3 defines scope of warrant; §4 gives the four conditions for occupying the questioning position; §5 gives the three items to be recorded for each case; §6 says how a discipline's own questioning is to be judged.
§1 Setting Cannot Be Empty
Every discipline, to run, uses some premises that its current internal procedures cannot settle without circularity. A significance threshold is judged by statistical tests, but the threshold cannot judge the threshold. A diagnostic classification is revised through clinical research, but the research must first use the classification to sort patients into groups. The definition of a unit of measurement is tested against data measured in that unit. Once a procedure uses these premises, the same procedure cannot be used to prove them over again from the start.
These premises must always be held steady, or work cannot go on: every paper would first have to argue over the threshold, every diagnosis over the classification. What holds them steady can be a written standard, a calibrated instrument, a classification, an indicator, a course of training, or a default parameter in a piece of software. The position that holds them steady this paper calls the setting position.
These things are very different from one another. A standard has articles, an issuing body, a version number. The calibration of an instrument is hidden in a certificate and a chain of traceability. A classification is revised every so many years, and when it is revised there is a working group. A course of training is what a graduate student learns from a supervisor as "this is how it is done in our field." A default parameter is the 95% confidence interval that common statistical software gives when the user changes nothing. The first three are visible: they have texts and authors, and one can point to them and say "this one." The last two are hard to see: a course of training lives in each person's hands; a default value is written in code or a manual, but why it takes this value may not be stated anywhere separately.
The setting position can be explicit, written in the articles of a standard; or implicit, existing only in training and default values. It can be concentrated in one committee, or distributed across tens of thousands of papers written by convention. It can only run in one way or another; it cannot be empty: if it were, work would stop.
S9 Setting cannot be empty. Premises that a discipline needs in order to run, and that its current procedures cannot settle without circularity, must always be held steady; what holds them can be standards, instruments, classifications, indicators, training and default parameters. The setting position can only run explicitly or implicitly, concentrated or distributed; it cannot be empty.
§2 Questioning Can Be Empty
Toward these premises there can be another kind of work: publicly asking where they come from, what alternatives there are, what consequences they bring, what remainder they leave, and how they could be withdrawn if they were to be. The position that does this work this paper calls the questioning position.
The questioning position can be unoccupied. When it is empty, the setting position runs all the same: the threshold is still used, the classification still sorts. This alone is enough to show that the two positions are not one. To say they are "not one" is not to say that questioning can do without any setting: to question one threshold, one must use other thresholds. Nor is it to say that the two positions are causally unrelated: settings are often changed by questioning, and questioning needs settings to be asked at all. It only says that one running does not guarantee that the other is there.
Psychology offers an example. Methodological criticism of null hypothesis significance testing was published in the field's own journals for decades: the statistical power of studies is generally low (Cohen 1962); significance tests are misread, and the support they give to theories is far weaker than people suppose (Meehl 1978; Cohen 1994). These criticisms reached the premise. In the records this paper examined (Note 1), no verifiable position was found that had the right to require core practitioners to respond publicly, or the right to initiate a review of this premise; in the decades after these criticisms were published, papers went on being written by 0.05. By the four conditions of §4 below, whether the position was occupied in this case could not be established: for the right of response and the right of review, nothing checkable was found in the records this paper examined; jurisdiction and independence were not checked item by item.
This is close to what Kuhn called normal science: within a paradigm one solves puzzles; the paradigm is not questioned, the puzzles are (Kuhn 1962). The setting position is close to the role a paradigm plays in normal science. They part in two places. First, this paper does not say that questioning comes only in times of crisis; it says the questioning position can be standing or can be empty, depending on the four conditions. Second, this paper does not judge whether a premise ought to be replaced; it registers only who can ask, whether anyone asked, and whether the answer reached the matter.
S10 Questioning can be empty. The position that publicly questions the sources, alternatives, consequences, remainder and means of withdrawal of these premises can be unoccupied. When the questioning position is empty, the setting position runs all the same, which is enough to show the two positions are not one. It is not claimed that questioning can do without any setting, nor that the two positions are causally unrelated.
§3 Scope of Warrant
To say whether the questioning position "has someone in the seat," one must first say what it asks about. It asks about a discipline's scope of warrant.
A discipline makes several commitments to those outside it: when its work counts as successful; which results may be cited; which methods carry justificatory force. These three together are its scope of warrant. When a clinical guideline writes "high certainty of evidence," what it commits to is that conclusions rated by its grading method may be cited; when a metrology body gives a recommended value for a constant, what it commits to is that the number may be used within the stated uncertainty. A scope of warrant has an edge: within the edge, the discipline is responsible; beyond it, it is not. What the questioning position asks about is precisely this edge itself: whether it is drawn rightly, and whether there is anything beyond it that the discipline's conclusions have wrongly carried in.
This definition is given before the four conditions: the first of the four conditions needs it.
S11 Scope of warrant. A discipline commits to when its work counts as successful, which results may be cited, and which methods carry justificatory force; these three together are its scope of warrant. This definition is given before the four conditions.
§4 Four Conditions
For a questioning position to count as "having someone in the seat," look at four conditions:
- Jurisdiction: it can put the scope of warrant of the discipline under review itself in question. One that can only find fault within the scope does not count.
- Right of response: it has the right to require core practitioners to respond publicly. One that can only voice opinions, which the other side can ignore, does not count.
- Right of review: it has the right to initiate a review of the premise, a failure test, or an evaluation of alternatives. One that can only request explanations and cannot initiate a review does not count.
- Independence: its continued existence and evaluation, including resources, appointment and removal, credentials and reputation, do not depend on this set of premises continuing to hold.
Why they are conjunctive. Without jurisdiction, one cannot ask about the edge itself, only about matters within it. Without the right of response, one can ask and no one need answer. Without the right of review, an answer can be just a paragraph, and the premise stays as it was. Without independence, questioning relative to this set of premises cannot be assured: the questioner's own livelihood is tied to the premise being questioned. If any one of the four is not met, questioning does not hold on that condition; the others are still recorded each on its own.
The four conditions take over from Institution Theory. Institution Theory requires an institution to keep open a channel for questioning and correction, and that "An institution bears the burden of explanation for what it has itself produced, and must answer the very matter it is questioned about"; independence is meant functionally: "what it guards against is the party under review sealing off review of itself on its own" (Institution Theory Z34). It also says that self-examination can be done, but it "does not replace the position of being questioned independently" (Institution Theory Z35a). This paper brings these sentences down to a discipline's questioning position and divides them into four conditions (own undertaking).
The four can fail one by one. If one is not met, only that one is recorded; it is not said that the whole position fails. The psychology example in §2 was recorded in just this way: for the right of response and the right of review, nothing checkable was found in the records this paper examined; jurisdiction and independence are recorded separately. How independence changes as roles change is the business of Paper 8.
S12 Occupancy requires the four conditions jointly. Jurisdiction: it can put the scope of warrant of the discipline under review itself in question. Right of response: it can require core practitioners to respond publicly. Right of review: it can initiate a review of the premise, a failure test or an evaluation of alternatives. Independence: its continued existence and evaluation, including resources, appointment and removal, credentials and reputation, do not depend on this set of premises continuing to hold. The four can fail one by one; if one is not met, only that one is recorded.
§5 Three Records
Each case is recorded under three items. The three are not steps climbing one above another; they are three columns, each filled in separately.
-
Occupied or not: whether the four conditions hold jointly in this jurisdiction. The record states which premise and which jurisdiction.
-
Whether questioning took place, and whether the answer reached the matter: did anyone ask? Did the answer land on the very matter that was asked?
-
Outcome: retained, rewritten, or withdrawn. The outcome and "change in force" are two columns: whether a rewriting entered everyday procedure is recorded separately. If the outcome is retention, the decision to retain is registered, together with how that decision was carried into procedure; the "change in force" column is recorded as not applicable.
Recorded this way, there are two cells that can be filled, and both must be fillable.
In force but not occupied. A premise was changed, and the change entered everyday procedure, yet from beginning to end there was no position meeting the four conditions. The so-called "credibility revolution" in economics may be a case. Angrist and Pischke used this name to say that empirical microeconomics had, over several decades, come to center more and more on research design (Angrist & Pischke 2010, p.4). The change appears to have come into force; in the records this paper examined, nothing showed that it was driven by a position with the right of response and the right of review. The rules the American Economic Association later set govern the replication of data and code, and the registration of field experiments before submission (AEA, RCT Registry policy); among the rules examined, none governs the identification premises themselves. To place this case in its cell, one would have to trace which premise exactly, who bears it, and how authority changed; here it is recorded only as a possible case.
Occupied and retained. A position meeting the four conditions asked, the answer reached the matter, and the outcome was to retain the original premise. For this cell the paper has not yet found a suitable case; it is registered as empty.
The three examples this paper uses, placed in the three columns, look like this:
| Case | 1. Occupied or not | 2. Questioning undertaken; answer reached the matter | 3. Outcome; change in force |
|---|---|---|---|
| Methodological criticism in psychology (p: significance testing with 0.05 as threshold) | Not established: no checkable right of response or right of review in the records this paper examined; jurisdiction and independence not checked item by item | Criticism undertaken; whether a response to this premise reached the matter, not checked | No record found of a decision to retain, rewrite or withdraw; whether any specific change entered everyday procedure: not checked |
| The research agenda for the psychiatric diagnostic manual and its fifth edition (p: categorical classification) | Not fully checked: evidence of independence to be obtained separately | Undertaken; partly reached the matter | Outcome: main body retained, an alternative model placed in Section III; change in force: Section III did not enter everyday diagnosis |
| The credibility revolution (p: to be traced to a specific identification premise) | Pending | Pending | Pending |
What the outcome column of the first row records is that no record of a decision was found: no position was found that decided to retain, and the threshold goes on being used. Going on being used is not retention; retention is a decision made after questioning. By the account in Paper 5, this may be a case of shelving, depending on whether the using side registers it as unresolved. All three columns of the third row are pending: the specific premise must first be traced.
Both cells show one thing: outcome and occupancy must be recorded separately. Retention is not failure: staying after questioning is exactly the contrast in Paper 1, "a premise retained after sufficient questioning." Nor does a premise's having changed, on that result alone, prove that the questioning was independent: the change may have come from other forces.
S13 Three records. Each case is recorded under three items, not as a progression: 1, occupied or not (this premise, this jurisdiction, whether the four conditions hold jointly); 2, whether questioning took place, and whether the answer reached the very matter asked; 3, outcome (retained, rewritten, withdrawn), in a separate column from whether it entered everyday procedure (change in force); in case of retention, the decision to retain and how it was carried out are registered, and "change in force" is recorded as not applicable. "In force but not occupied" and "occupied and retained" are both registrable cells. Retention is not failure; nor does a premise's having changed, on that result alone, prove that the questioning was independent.
§6 A Discipline's Own Questioning
A discipline also questions its own premises: foundational research, methodological journals, revision working groups. Touching a premise is not the same as occupying the questioning position.
Psychiatry is an example. In 2002 the American Psychiatric Association published a research agenda in preparation for the next edition of its diagnostic manual, which discussed premises of the current classification, for instance whether disorders should be divided into separate categories or seen as continuous dimensions (Kupfer, First & Regier 2002). That is touching. The fifth edition, published in 2013, still classifies by category in its main body; in Section III it includes "Emerging Measures and Models," among them an alternative diagnostic model for personality disorders (APA 2013, Section III). Recorded by §5: questioning took place, and the answer partly reached the matter; the outcome is that the main body was retained and the alternative model placed in Section III, without entering everyday diagnostic procedure. Occupancy has to be checked separately: the right of response and the right of review lie with the revision working group, but its independence depends on whether its continued existence and evaluation depend on this set of premises, which requires separate evidence and cannot be inferred backward from "no rewriting."
So this paper says: a discipline's own questioning is judged by the three records, not by whether the matter has been discussed.
S14 A discipline's own questioning is judged by the record. That a discipline's foundational research and revision working groups touch a premise does not mean they occupy the questioning position; it is judged by the three items of S13, not by whether the matter has been discussed.
Where This Paper Stops
This paper does not prove that these premises ought to be questioned; it does not answer who ought to sit in the questioning position; that is the business of Paper 9.
What it hands to Paper 3: the questioning position often seems to have someone in it, because evidence review is always reviewing. Paper 3 asks what its conclusions are taken to have answered.
The remainder it leaves: the cell "occupied and retained" has no case yet; the credibility revolution case must be traced to a specific premise and its bearers before it can be placed; occupancy in the psychology case was not established, and the search behind "nothing was found" was limited; whether it is shelving is judged separately by Paper 5.
Notes
- For the psychology case, the records this paper actually read are: the three critiques themselves, Cohen 1962, Cohen 1994 and Meehl 1978, and the American Statistical Association's 2016 statement. It did not go through the instructions to authors of psychology journals one by one, nor the statistical reporting standards in the American Psychological Association's publication manual. "Nothing was found" speaks only of this set of records and is not extended to psychology as a whole.
The Six Propositions of This Paper
- S9 Setting cannot be empty. Premises that a discipline needs in order to run, and that its current procedures cannot settle without circularity, must always be held steady; the position that holds them can only run explicitly or implicitly, concentrated or distributed; it cannot be empty. (Conceptual distinction; own undertaking)
- S10 Questioning can be empty. The position that publicly questions these premises can be unoccupied; when it is empty, setting runs all the same, which is enough to show the two positions are not one. It is not claimed that questioning can do without setting, nor that the two are causally unrelated. (Conceptual distinction; tested by cases)
- S11 Scope of warrant. A discipline commits to when its work counts as successful, which results may be cited, and which methods carry justificatory force. (Conceptual distinction: definition)
- S12 Occupancy requires the four conditions jointly. Jurisdiction, right of response, right of review, independence; the four can fail one by one, and if one is not met, only that one is recorded. (Conceptual distinction: the criterion of occupancy; takes over Institution Theory Z34, Z35a, brought down to a discipline's questioning position, own undertaking; its application tested by cases)
- S13 Three records. Occupied or not; whether questioning took place, and whether the answer reached the matter; outcome and change in force in separate columns. Two registrable cells; retention is not failure; a change does not on its result alone prove independence. (Notation)
- S14 A discipline's own questioning is judged by the record. Touching is not occupying; judged by S13. (Conceptual distinction: criterion)
References
I · This system
- Qin, H. (2026). Open to Questioning (SAE Institution Theory, Paper 6) [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23271324
- Qin, H. (2026). What a Discipline Is Not (SAE Theory of Disciplines, Prequel) [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.24177508
- Qin, H. (2026). Parochialism (SAE Theory of Disciplines, Paper 1) [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.24177748
II · Literature and institutional documents
- American Economic Association. RCT Registry policy. aeaweb.org (current page).
- American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental Disorders (5th ed.). Section III.
- Angrist, J. D., & Pischke, J.-S. (2010). The credibility revolution in empirical economics. Journal of Economic Perspectives, 24(2), 3–30.
- Benjamin, D. J., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2, 6–10.
- Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65(3), 145–153.
- Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003.
- Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd.
- Kuhn, T. S. (1962). The Structure of Scientific Revolutions. University of Chicago Press.
- Kupfer, D. J., First, M. B., & Regier, D. A. (Eds.). (2002). A Research Agenda for DSM-V. American Psychiatric Association.
- Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834.
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.
Interfaces
Main upstream: Institution Theory Z34, Z35a; Paper 1, S2, S3.
Also used: Fisher 1925; the ASA statement (Wasserstein & Lazar 2016); Benjamin et al. 2018; Cohen 1962, 1994; Meehl 1978; Angrist & Pischke 2010; the DSM research agenda 2002 and DSM-5 Section III; Kuhn 1962.
This series: S9 to S14.
Handed to Paper 3: evidence review is always reviewing; what are its conclusions taken to have answered.