Defining the Right Answer
A risk assessment answers questions, and a good risk
assessment will answer those questions well.
But, what does “well” mean here?
- True. The obvious response to that question is the “true” answer. But, if the scientifically true answer is, in fact, not known then that definition of “good” just isn’t practical. A different answer is therefore often substituted: Garbage In, Garbage Out.
- Politically Correct. If the true answer is not to be had, then risk assessment can be viewed as a set of standardized procedures and/or expectations. Even though the answer isn’t really true, it is relatively simple to obtain, and that may on some occasions be sufficient to justify the use of default assumptions and the like. Furthermore, as long as “politically correct” is not conflated with “true” then there is little or no harm done. But, it seems to be inevitable that just that very thing happens: Acting upon a particular assumption leads people to believe it. Gospel In, Gospel Out. Reality then apparently become a product of political negotiation. But people aren’t entitled to their own facts, not even in Washington.
- Honest. Which leads to a third definition of “good”, which is “honest”. While that sounds trite, it really isn’t. “Honest” in this case doesn’t necessarily entail admitting to a moral failure. Instead, it involves fessing up to uncertainty: An estimate with a narrower confidence interval isn’t really better unless that confidence is justified. In fact, if moral judgment is an issue at all, it involves giving up the PC objective, where “good” really means “good enough”. Perhaps the main difficulty with the honest uncertain answer is technical. It is easy to say “I don’t know”, but that doesn’t answer the question and it may not be entirely honest either when something, but not everything, is known about the problem. Therefore, the goal becomes to provide all plausible answers, rather than the single true answer. Plausible In, Plausible Out can get complicated.
The Other Probability
In the realm of words, different species of probability have
been recognized and discussed for centuries.
Hume (1739) differentiated between the “probability of chance” from the
“probability of causes”. Even though
Cohen (1977) called the other probability “nonpascalian” since Blaise Pascal is
often credited with developing the mathematics associated with chance and
statistics, Hacking (1975) credited Pascal with devising the probability tree,
using it to give the theoretical probability for the existence of a Catholic
god the same “epistemic standing” as an aleatory probability. Yet, when risks become matters of degree, the
other probability often disappears, and probability seemingly becomes
synonymous with statistics. The
probability that inevitably gets left out when this happens is the theoretical
sort.
However, there are many analysts who do find it useful to
distinguish concepts of probability.
For, example Kaplan (1997) gave a keynote address at a Society of Risk
Analysis meeting that gave outlined three concepts of probability. One can be uncertain about variability, and
one can entertain different theories that may account for a statistical
reality. Furthermore, the importance of
model uncertainty in making many risk estimates is widely recognized.
The simpler part of solving the model uncertainty problem of
is to employ a different formal representation, namely the probability tree,
for a theoretical probability than for the continuous probability distribution
that is typically used for statistical uncertainties. But, the other thing that needs to happen is
that the responsibility map for the division of professional labor needs to be
redrawn. When uncertainties arise, it is
generally thought that the solution is to send in a statistician. In fact, many statisticians do indeed
themselves consider the Other Probability to be their responsibility, and as a
result, have devised various and sundry Bayesian schemes intended to stuff the
recalcitrant prior probabilities into an aleatory mold. But, that’s not really what is needed. Probability trees are the domain of
multi-handed scientists (David, 1975), so if you want to assign a probability
to a theory, ask them what to do. Even
if they can’t identify the correct theory, they should be able to say which are
more likely and why: Maybe
that’s a good way to find out who the better scientists really are.
Examining Assumptions
When assumptions are justified by tradition or regulatory
policy, then the process of providing estimates do not provide an occasion for
questioning the validity of the premises.
Not very scientific: Policy In, Policy Out. Maybe the PC answer is within the realm of
plausible interpretation, but who would know?
On the other hand, if an estimate is purported to be valid, then the
premises are bound to be questioned, especially when there is health or money
at stake. That will call for a
supporting argument.
If one premise can be proven to the exclusion of all others,
then Truth In Truth Out. In some cases,
a sensitivity analysis may demonstrate that the alternative assumptions yield
about the same answer, thereby making the issue moot. But, with low-dose risk
estimation, that doesn’t happen; the choice of model used to make the risk
estimate matters a lot (NRC, 1994). So,
you have a probability tree. Yet, which
alternative theories make up the tree still matters, and scientific arguments
still need to made for either including them or not. An uncertainty analysis may make it easier to
come up with a risk assessment that is scientifically credible, but that can
only happen if the set of alternative assumptions can be defended as being at
least plausible, and the full set encompasses the entire range of assumptions
that are distinctly possible: Probable
In Probable Out.
Bayesian solutions to model uncertainty often allow
probabilistic judgments regarding theories.
However, the general inclination is to relegate such judgements as
“priors” that are subsequently modified based on the available data. There are a couple of problems with
that. First, judging theories does not
happen in a data vacuum; data has a lot to do with it. Second, and perhaps more importantly, new
data may to revision of the scientific judgments of alternative theories, or
maybe even result in the introduction of a new theory entirely. If that happens, Bayes theorem will have
nothing to do with it.
References
Cohen, L.J. (1977). The Probable and the Provable. Oxford: Clarendon Press.
Hacking, I (1975). The
Great Decision. In: The Emergence of Probability.
Cambridge University Press.
Kaplan, S (1997). The
Words of Risk Analysis. Risk Anal 17:407-411.
Official Post Soundtrack
Post Notes
Thesis Post #51, Part of the solutions thread.
No comments:
Post a Comment