Dear Journal Editors

Rejected

Thank you very much for reviewing my manuscript entitled “Plausible In, Plausible Out: A Bootstrap Methodology to Characterize the Uncertainty Associated With Dose-Response Modeling”.  I am disappointed in the result, of course.  However, as it met the same fate as every other paper on model uncertainty that I have sent to the various and sundry editors that the Journal has had over the last 25 years, I am not terribly surprised.

I think I will not attempt to rewrite the paper to make it more acceptable.  The paper says what I want it to say and I don’t think any of the major suggestions made in the review will make it any better.  A similar example to that discussed in the manuscript is also in the FDA (2016) assessment on arsenic in rice released two weeks ago (see section 9.4), and I suppose my purposes will be better served by working the rest of the text into my other writing projects.  Nonetheless, for the benefit of my colleagues who are interested in model uncertainty in general and this paper in particular, I would like to address some of the comments made in the course of the review.

A Methodology Paper

The paper I submitted is a discussion of two methodological developments used in USFDA assessments for arsenic in apple juice (Carrington et al, 2013) and rice (USFDA, 2016).  The first method involves the use of a parametric bootstrap simulation to propagate the uncertainty associated with dose estimation into the characterization of the dose-response relationship.  Putting error bars on the doses is a novel technique and as just about everyone I know thinks it is a pretty good idea, this was the impetus for writing the paper in the first place.   Although the reviewers didn’t seem to think this technique was remarkable, at least they didn’t object.  I guess it really is a pretty obvious thing to do once you’ve thought of it.  The second reviewer did suggest that additional details be provided about the input distributions for the dose estimates.  I decided not to do that when I wrote the paper because I thought getting into specifics would be a distraction from the more general idea of allowing dosimetric uncertainties to be represented in a dose-response analysis.   If I get around to reworking the analysis for some other purpose, I will heed those suggestions to the extent that I can; many of the issues raised by the second reviewer resulted from the necessity of working with published summary data rather than observations from individual subjects.  However, for a methodological presentation where no importance is attached to the actual results at all, I don't think any of that matters.

Against the wishes of potential FDA coauthors, I also chose to include a discussion of model uncertainty.  Since this was and is the hot button political topic for any risk assessment involving arsenic and any other chemical hazard worthy of attention I thought it would be a serious omission to not include it, especially since there may often be an interaction between dosimetry and empirical weighting of alternative models.  As near as I can tell, the reviewers haven’t raised any serious objections to anything I said about model uncertainty, but it is very clear that they really don’t like the way I said it.  This may be partly due to the fact that the reviewers didn't take the discussion in the methodological context that was intended, but since this seems to be the reaction I always get when I try to talk about model uncertainty, I think there is more to it than that.  I think I understand the nature of this editorial issue far better than I used to, so I will take this opportunity to explain.

Unfinished Science

Model uncertainty is subjective.  Some people have it and others don’t.  I think the two basic causative factors are as follows:

• The model has to be thought of as a theory.  Even if it is only approximately correct, the mathematical model has to convey some truth that is not evident from isolated observations.
• There has to be more than one model-theory.  If only one model is under consideration, then there is no uncertainty. 

Taken together, these two criteria basically mean that if you are afflicted with model uncertainty then you are thinking like a scientist.  But here’s the thing; if you have it then you can’t really talk or write about model uncertainty in the objective third person writing style generally preferred by governments and journal editors.  You can describe a model uncertainty as a psychological phenomenon as I just have, but that is pretty much the end of the third person road.  If you think it is just me who can’t write properly, go back and read the most widely cited paper on model uncertainty ever written (Hill, 1966): It is written almost entirely in the first or second person.  For example, the problem the paper sets out to solve is stated as follows:

Our observations reveal an association between two variables, perfectly clear-cut and beyond what we would care to attribute to the play of chance. What aspects of that association should we especially consider before deciding that the most likely interpretation of it is causation?

Part of the problem is that scientists don’t write their papers in the same manner as they converse in private.  When papers are written, it is often because model uncertainties have been resolved and what were once just theories are reported to be objective realities.   But risk analysts can’t do that.  There are many model uncertainties that have not been resolved and they may never be, which leaves us with probability trees and subjective weight-of-the-evidence evaluations.

My Problem

The apple juice and rice risk assessments both used probability trees to depoliticize arguments over which dose-response model “should” be used to characterize the causal relationship between inorganic arsenic and cancer.  I am happy to report that this strategy worked as I hoped that it would.  The two assessments employed somewhat different strategies for assigning probabilities to alternative models.  Because I think it is more consistent with how scientists actually think, I prefer the strategy used in the rice assessment that largely relies on expert opinion. 

My only reservation is that the probabilities used for the rice assessment relied only on my opinion.  As an expert, assigning probabilities to theories was implicitly part my job description at the FDA, and I’m not complaining about having to do what I was paid to do.  I have a PhD in Pharmacology and long experience in modeling dose-response relationships, so I don’t feel unqualified.   However, my primary area of expertise is neurotoxicology rather than cancer biology, and I am far more familiar with the literature on lead and methylmercury than arsenic.  So, it would have been nice to have other experts involved, especially if the stakes are raised from just setting guidance values for apple juice and infant cereal that have relatively little economic impact to suggesting that consumers modify their rice intake.  But in order move from the subjective “I” to the intersubjective “We”, I think that those of us who are afflicted with model uncertainty need to be permitted to write in the same way as we converse among ourselves, and we can’t do that if we are forced to pretend to objectivity that we really don’t have.

References

Carrington CD (unpublished).  Plausible In, Plausible Out: A Bootstrap Methodology to Characterize the Uncertainty Associated With Dose-Response Modeling.

Carrington CD, Murray C, and Tao, S. (2013). A Quantitative Assessment of Inorganic Arsenic in Apple Juice. 

Hill, Sir Arthur Bradford (1965).  The Environment and Disease: Association or Causation?  Proc Royal Soc Med 58:295-300.

U.S. Food and Drug Administration (2016).  Arsenic in Rice and Rice Products Risk Assessment.

Official Post Soundtrack

Green Day (1997).  Reject.  In: Nimrod, Track 14.

Post Notes

Thesis Post #64.  Even though I will probably group it in the arsenic series, this post is really about academic politics.