To Bayes or Not to Bayes, That Is the Question

The Responsibility Gap

There is no reason to assign probabilities to competing theories unless there is an actual decision that must be made.  Pascal knew that, and that’s one of the things you would learn from chapter 6 (Hacking, 1975).  It obviously can’t be a flawless process; otherwise the correct theory would be assigned a probability of 1 every time.

So, I figure the next best thing is to have probability assignments that are scientifically defensible.  At least that’s what I tried to do when working for a regulatory agency as a scientific advisor because a) I figured it was my job, b) I liked the job, and c) I was allowed to do it.  I always thought that would start a conversation about what the probability assignments should be.  But that didn’t happen for several related reasons:

1) It may be easier to not make a decision at all.  It’s sort of an FDA tradition to declare emergency and then at a later date declare victory without doing anything at all.

2) If a decision absolutely must be made then it is much easier to with a formula, e.g. safety/uncertainty factors that doesn’t need to scientifically defensible.

3) If a risk estimate is necessary then it’s much easier to use a default assumption (linear extrapolation from high doses

All of those techniques insulate the expert from the decision, which obviates the need to assign probabilities to alternative hypotheses.  

Bayesian metholodogy also insulates experts from the decision, but to a lesser degree.  Since it does lay out the competing theories that underly a decision, I think it is far preferable to the backroom methodologies outlined above.  But perhaps the best about Bayesian methodology is that the practitioners actually want the job.

Help Wanted

I started assigning probabilities to alternative hypotheses because I figured it was my job and for many years I was allowed to do it.  I was also allow to participate in the writing of my own job description and I always made sure it said I was supposed to “convey uncertainty about potential health effects arising from contaminants  in food to decision makers”.  But that didn’t last; after a reorganization my job description changed to something like “support a decision that has already been made”.   That’s when I pretty much figured I’d rather write a blog than work at the USFDA.

I don’t really want my old job back.  I’m too old for that.  But I still figure someone else should have it.  There’s been another reorganization, but there is still a contaminants branch facing the same old problems.  Seems like a lot of people in the EPA and WHO should know their way around a probability tree as well.  

But it isn’t really just a government issue; it’s primarily a science problem.  The Bayesians shouldn’t need to identify plausible theories after a study has already been conducted.  That should have happened before the effort to design experiments and/or collect observational data began.  But of course, it is entirely possible that all of the hypotheses considered at the outset of a study are disproven by the new data.  That makes it time for a new theory.  Neither a frequentist or Bayesian analysis can help with that.  But a probability tree can.

Model Shopping

Perhaps the scientists who need probability trees the most are epidemiologists, especially the ones doing multivariate analysis with multiple putative causal influences.  I’ve been over some of it before from a historical perspective (Neyman was right, but Fisher sold more textbooks); null hypothesis testing doesn’t necessarily test the hypotheses that really matter.  That can easily set up a model shopping exercise that is solely interested in generating statistical significance, possibly by using a model that isn’t plausible in the first place.  I’ll also add the general point made in my last post that trying to turn hypothesis testing into a statistical exercise that treats observations as instances in stochastic probability theory rather than evidence for or against a theory even when the underlying theory isn’t stochastic at all. 

Statistical significance testing isn’t crazy when the number of alternative theories is exactly two, and the number of observations is small.  But otherwise, it’s nuts.  Model shopping isn’t such a bad idea when you are shopping for plausible theories.  However, it needs to happen as part of an open discussion that even someone working for the government can take part in.  That means the set recorded observations  used in published studies needs to be shared.  Furthermore, the search for more plausible theories doesn't stop just because a paper has already been published.  

Reference

Hacking, I (1975).  The Great Decision.  In: The Emergence of Probability.  Cambridge University Press, pp. 63-72.

Official Sound Track

Talking Heads (1977).  Don't Worry About the Government.  In: Talking Heads: 77, Track 8.