Biological Problems

The Perils of Multivariate Linear Regression

As is often the case in the epidemiological literature on environmental influences on neurobehavioral development, Bowers and Beck (2006) noted that a paper by Lanphear et al (2005) “has suggested the existence of a supra-linear dose–response relationship between environmental measures such as blood lead concentrations and IQ”.  They then produced an analysis that indicates that the apparent supralinearity is an artifact resulting from the way the data were analyzed.  They stated their conclusion as follows:

Results of the analyses show that a supra-linear slope is a required outcome of correlations between data distributions where one is lognormally distributed and the other is normally distributed. 

While their mathematical analysis was indubitably correct, the way Bowers and Beck reported the results left something to be desired.  How the data are distributed is not really the issue at all.  Instead, the mathematical artifact they found results from conducting linear regression analyses with log transformed data.  If data from a normal distribution, or any other distribution, were log transformed prior to the regression analysis, then the same result would be obtained.  Furthermore, as demonstrated by Jusko et al (2006), a linear regression without log transformation with data drawn from a lognormal distribution does not result in a supralinear dose-response relationship.

To their credit, Hortung et al (2006) also understood that the real issue is the shape of the dose response relationship rather than the distributions that either the dependent or independent variables follow.  They therefore protested that Lanphear et al (2005) had considered the likely shape of the curve before conducting the regression analysis:

The shape of the exposure–response relationship was determined to be nonlinear insofar as the quadratic and cubic terms for concurrent blood lead were statistically significant (p < 0.001 and p = and 0.003, respectively).  Because the restrictive cubic spline indicated that a log-linear model provided a good fit to the data, we used the log of concurrent blood lead in all subsequent analyses of the pooled data.

But, there are many problems with this justification.  First, it is not at all clear how a spline analysis specifically supports a log-linear model, as opposed to other potential nonlinear models (e.g. a Hill function).  Second, there was no consideration of biological plausibility.  Like Bowers and Beck (2006), Lanphear et al (2005) seem to think establishing a causal relationship is a mathematical problem rather than a biological one.  Third, they did not consider the possibility that other covariates might explain the apparent nonlinearity.   Yet, off they went, and a dose-response model that predicts infinite large effects as the dose approaches zero was the inevitable result.  For all practical purposes, Bowers and Beck (2006) were entirely correct.  

Besides the fact that a loglinear dose-response model is a very poor theory, there is a more general lesson to be learned:  A multivariate regression with assumed quantitative relationships between the variables being modeled is highly prone to error.  While a loglinear relationship is obviously wrong, a linear relationship isn’t necessarily right either.  Correlations between variables may result in attribution of mismodeled causal effects to a variable that has no causal effect at all.  For example, if the relationship between socioeconomic status (i.e. the HOME score) and IQ is nonlinear with bigger impacts with low scores and negatively correlated with exposure to an environmental chemical, the some of the socioeconomic effect will erroneously appear to be a low dose effect attributable to the environmental chemical.  There are many other possible explanations as well, all of whicih are more probable than a dose response model than predicts incremental effects to get bigger as the dose gets smaller.

Process vs. Theory

Biological complexity often makes the pronunciation of definitive truths doe Medicine and Public Health practically impossible.  While relying on expert opinion is a common solution to that problem, that solution does not work well when opinion is divided.  As a means of coping with that problem, institutional decision making processes often employ structured evaluation systems to sort through what can often be a voluminous set of scientific literature.  The Safety Assessment methodology that is typically used for premarket approval evaluations is an example.   This description of Evidence-Based Medicine conveys the general ethos of such efforts:

Whether applied to medical education, decisions about individuals, guidelines and policies applied to populations, or administration of health services in general, evidence-based medicine advocates that to the greatest extent possible, decisions and policies should be based on evidence, not just the beliefs of practitioners, experts, or administrators. It thus tries to assure that a clinician's opinion, which may be limited by knowledge gaps or biases, is supplemented with all available knowledge from the scientific literature so that best practice can be determined and applied. It promotes the use of formal, explicit methods to analyze evidence and makes it available to decision makers.

There are two key concepts at work here.  First, the “beliefs of practitioners, experts, or administrators” are getting kicked to the curb in favor of “evidence”.  If you thought the beliefs of experts were based on scientific evidence, then you were misinformed, apparently.  Secondly, there is an emphasis on the “use of formal, explicit methods”, which also serve to limit subjective influences on the evaluation process. 

Experts are not always trustworthy, so the desire for a transparent process is entirely understandable.  But, getting a trustworthy process to replace the experts is easier said than done.  The process has to be designed by somebody, and that usually means experts.  There is also apt to be a negotiation process involved in getting the process to be accepted, so subjectivity isn’t really completely avoided.   But perhaps the bigger problem is that trying to deal with complex biological issues with a formula may often be rather stupid.  If all the studies show the same result, then it really isn’t going to matter whether the decision making process is expert-based or evidence-based.  If the results are different, then the systematic review may succeed at identifying the higher quality studies and grading the general result.  But, it won’t explain why the results are different.  It won’t figure out why a treatment may works sometimes, but not others.  That will take biological theories, and like the biases the evidence-based systems strive to avoid, those are subjective.   There are likely to be different theories, of course, and then the experts will inevitably get into a debate over which are more likely.  But, guess what, that’s the way science works: Trying to eliminate all potential bias with formulae will also eliminate scientific progress.

By all means, more transparency is needed.  In particular, let’s not trust authors to have the last word on how the data they have collected are analyzed and published.  Medical researchers and epidemiologists are notorious for not sharing original data involving human subjects, even when they are legally required to do so (Panhuis etal, 2014; Longo and Drazen, 2016).  That will allow better theories to flourish, and poor theories to flounder.

References

Bowers TS and Beck BD (2006).  What is the meaning of non-linear dose-response relationships between blood lead concentrations and IQ?  Neurotoxicology 27:520-4.

Hornung R, Lanphear B, Dietrich K. (2006).  Response to: “What is the meaning of non-linear dose–response relationships between blood lead concentrations and IQ?”.  Neurotoxicology 27:635

Jusko TA, Lockhart DW, Sampson PD, Henderson CR Jr., and Canfield RL (2006).  Response to: “What is the meaning of non-linear dose–response relationships between blood lead concentrations and IQ?”.  Neurotoxicology 27:1123–1125.

Lanphear BP, Hornung R, Khoury J, Yolton K, Baghurst P, Bellinger DC, Canfield RL , Dietrich KN, Bornschein R, Greene T, Rothenberg SJ,8, Needleman HL, Schnaas L, Wasserman G, Graziano J,13 and  Roberts R. (2005).   Low-Level Environmental Lead Exposure and Children’s Intellectual Function: An International Pooled Analysis.  Environ Health Perspect. 113: 894–899.

Longo DL and Drazen JM (2016).  Data Sharing.  N Engl J Med 374:276-277.

Panhuis WG van, Paul P, Emerson C, Grefenstette J, Wilder R, Herbst AJ, Heymann D, and Burke DS (2014).  A systematic review of barriers to data sharing in public health.  BMC Public Health 14:1144.

Official Post Soundtrack

Jackson, J (1980).  Biology.  In: Beat Crazy, Track 9.

Post Notes

Thesis Post #65.  This covers some of the same ground as Toxicology Meets Epidemiology, but with a more philosophical overview. 

Mixed Probability Calculations

Probability is the Guide of Life

For personal decisions, theoretical uncertainty is the far more familiar form of probability.  If two different sources of information lead to different courses of action, then you have to either decide who and what to trust or hedge your bets.  However, the probability of chance that is amenable to a mathematical treatment and is the main form found in academic discourse can be important too.   The relative importance of the two probabilities can vary with the problem.  Sometimes one or the other will dominate, while in other instances both are important.  Recreational betting games serve as an example:

• Roulette.  Betting on a roulette wheel is purely a game of chance.  The odds and a long term expected return can be calculated very accurately.  Well, unless the game is fixed.
• Horse Racing.  In theory, some horses are faster than others – chance has very little to do with who wins.  Sure, historical records are important, but that’s mainly because they indicate which horses are fast and which ones are not.
• Poker.  The odds that a certain card or cards will turn up can be calculated, and the game of poker can be simply played as a game of chance.  But good poker players also take the mannerisms of their opponents into account when they bet, which turns poker into a mixed probability game.

It’s the last category of problems that make risk analysis interesting.

 

Betting on the Single Instance

If you are betting on a single instance (i.e. what to do now), then boiling down theoretical probability and statistical probability into a single judgment or number is essential.  A simple equation will suffice to represent this notion:

pTotal = pTheory * pChance

If the roulette wheel is fair, then pTheory =1, and therefore pTotal is dominated by calculating the odds.  If the fastest horse wins, then pChance is 1, and pTotal is dominated by pTheory.  When gamblers bet on a horse, converting horse theory to a numerical value is exactly what they do.  Poker players have a tougher calculation – not only do they have to know the odds of a card turning up, they also have to assign a probability to the notion that bluffing will work, or that their opponent is bluffing.

Betting on the Series

But once the bet becomes about the long run, or about public health instead of an individual, then the calculation is quite different.  It’s a two dimensional problem where the primary goal is to predict the frequency of a result or different results, and there will also be uncertainty about estimated frequencies.  The probability calculation isn’t the same any more.  The probability of chance is often a statistical frequency instead.  In fact, it may or may not be a theoretical frequency.  For example, there can be a range of statistical estimates that range from purely empirical to purely theoretical.  An historical record with a large number of observations may justify a frequency estimate with no theoretical uncertainty.  On the other hand, a fewer number of observations may serve to support a statistical theory instead, which begets theoretical uncertainty.  The frequency calculation is now a function instead of a single number, so the relationship between theoretical probability and the frequency of occurrence is now something far more complicated:

p(Frequency) = pTheory(pChance)

Empirical observations may also be used to disprove a theory too.  For example, a large number of observations may show a particular die to be unfair.  The again, there may only be enough data the favor one theory over another without being able to conclusively decide that one is indubitably correct.  That means you are going to need a probability tree. 

Quantifying Theoretical Probabilities

Frequentist probability schemes tend to acknowledge theoretical uncertainty (e.g. as “systematic error”), but then go on to ignore it.  On the other hand, Bayesian probability schemes typically treat theoretical and statistical probabilities interchangeably.  If you are betting on a single instance, that works reasonably well.  Updating a theoretical prior with data can gradually transform the probability into one of chance – the more data there are, the less the theory matters.  But it isn’t really very scientific.  If they were used to discriminate among alternative theories, the data might be put to better use.  That problem is even more critical for the estimation of long run frequencies.  Updating the parameter estimates for a model that has been proven to be wrong doesn’t make much sense.

Since it really is more consistent with how scientific knowledge is developed, explicitly assigning probabilities to theories is a better strategy for long-term issues where knowledge may be expected to progress.  Since theoretical probabilities are inherently subjective, it is hard to improve upon convening a panel of experts to weigh the scientific evidence.  Even if the experts don’t get it quite right, or they aren’t the right sort of experts, the process of assigning probabilities to competing theories creates an occasion for scientific discussion.   As long as no one thinks that probabilities assigned to theories are the gospel truth, it’s all good in my book.

As a recent example, Trasande et al (2015) provided an overview of the efforts to characterize the theoretical probabilities for causal theories involving potential health effects of Endocrine Disrupting Chemicals (EDCs):

We now describe the general methods used to attribute disease and disability to EDCs, to weigh the probability of causation based upon the available evidence, and to translate attributable disease burden into costs. During a 2-day workshop in April 2014, five expert panels identified conditions where the evidence is strongest for causation and developed ranges for fractions of disease burden that can be attributed to EDCs.

I have more than a few quibbles with exactly what they did, ranging from how the problems were characterized in the first place (i.e. by presuming independent attributable risks), the use of implausible dose-response models, the lack of serious consideration of other (i.e. non-EDC) causal factors, and the relationship between association and causation is all-or-none.   Also, because the probability assignments are subjective, a two-day workshop of experts with similar interests is not really sufficient for a decision involving the economic impacts that are alleged, so I don’t recommend taking these estimate as the last word. However, praise for the process is well deserved.  Nonetheless, as it pertains to the present topic of discussion, there is one error in how the theoretical probability was employed after it was arrived at that must not go unnoticed:

Finally, recognizing that attributable cost estimates were accompanied by a probability, we performed a series of Monte Carlo simulations to produce ranges of probable costs across all the exposure-outcome relationships, assuming independence of each probabilistic event. Separate random number generation events were used to assign 1) causation or not causation, and 2) cost given causation, using the base case estimate as well as the range of sensitivity analytic inputs produced by the expert panel. To illustrate with an example, for an exposure-outcome relationship with an 80% probability of causation, random values between 0 and 1 in each simulation led to the first step, which either assigned no costs (random value ≤ 0.2) and costs (random value > 0.2).

If the problem required the combination of both theoretical and statistical probabilities, the use of the probability tree in a Monte-Carlo simulation would be appropriate.  However, there is a problem in implementation that arises from the fact that a causal probability is NOT a probability of chance: A theory is either true all the time or false all the time, and the entire cost estimate is dependent (so, no you can’t assume independence) on the truth of the theory.  So, using a causal probability to calculate the probability of an event is inappropriate.  Instead, the logic should go like this: Since all of the causal probabilities have a probability of less than 95%, the lower bound cost estimate of all of the end points should be zero (see table four).  For those endpoints with a causal probability of less than 50%, the central estimate should be zero as well.   

Reference

Trasande L, Zoeller RT, Hass U, Kortenkamp A, Grandjean P, Myers JP, DiGangi J, Bellanger M, Hauser R, Legler J, Skakkebaek NE, and Heindel JJ (2015).   Estimating Burden and Disease Costs of Exposure to Endocrine-Disrupting Chemicals in the European Union.  J Clin Endocrinol Metab 100: 1245–1255.

Official Post Soundtrack

Cars, The (1978).  All Mixed Up.  In: The Cars, Track 9.

Post Notes

Thesis Post #64.  If someone can figure out a way to short their bet on all those IQ points, I'm all in.