Plausible In Plausible Out

Computers are very logical and very obedient.  They will calculate the conclusion for whatever premises and assumptions they are given.  The one shortcoming, of course, is that the conclusions are never any more reliable than the premises.  Although using a computer to provide risk estimates often does hide shaky premises inside the proverbial black box, that doesn’t have to happen.  But, opening up the black box requires some effort on the part of both the programmers and the users.  In particular, there are three main things that need to happen:

Defining the Right Answer 

A risk assessment answers questions, and a good risk assessment will answer those questions well.    But, what does “well” mean here?

• True.  The obvious response to that question is the “true” answer.  But, if the scientifically true answer is, in fact, not known then that definition of “good” just isn’t practical.  A different answer is therefore often substituted: Garbage In, Garbage Out.
• Politically Correct.  If the true answer is not to be had, then risk assessment can be viewed as a set of standardized procedures and/or expectations.  Even though the answer isn’t really true, it is relatively simple to obtain, and that may on some occasions be sufficient to justify the use of default assumptions and the like.  Furthermore, as long as “politically correct” is not conflated with “true” then there is little or no harm done.  But, it seems to be inevitable that just that very thing happens: Acting upon a particular assumption leads people to believe it.  Gospel In, Gospel Out.  Reality then apparently become a product of political negotiation.  But people aren’t entitled to their own facts, not even in Washington.
• Honest.  Which leads to a third definition of “good”, which is “honest”.  While that sounds trite, it really isn’t.  “Honest” in this case doesn’t necessarily entail admitting to a moral failure.  Instead, it involves fessing up to uncertainty: An estimate with a narrower confidence interval isn’t really better unless that confidence is justified.  In fact, if moral judgment is an issue at all, it involves giving up the PC objective, where “good” really means “good enough”.  Perhaps the main difficulty with the honest uncertain answer is technical.  It is easy to say “I don’t know”, but that doesn’t answer the question and it may not be entirely honest either when something, but not everything, is known about the problem.  Therefore, the goal becomes to provide all plausible answers, rather than the single true answer.  Plausible In, Plausible Out can get complicated. 

The Other Probability

In the realm of words, different species of probability have been recognized and discussed for centuries.  Hume (1739) differentiated between the “probability of chance” from the “probability of causes”.  Even though Cohen (1977) called the other probability “nonpascalian” since Blaise Pascal is often credited with developing the mathematics associated with chance and statistics, Hacking (1975) credited Pascal with devising the probability tree, using it to give the theoretical probability for the existence of a Catholic god the same “epistemic standing” as an aleatory probability.  Yet, when risks become matters of degree, the other probability often disappears, and probability seemingly becomes synonymous with statistics.  The probability that inevitably gets left out when this happens is the theoretical sort.

However, there are many analysts who do find it useful to distinguish concepts of probability.  For, example Kaplan (1997) gave a keynote address at a Society of Risk Analysis meeting that gave outlined three concepts of probability.  One can be uncertain about variability, and one can entertain different theories that may account for a statistical reality.  Furthermore, the importance of model uncertainty in making many risk estimates is widely recognized.

The simpler part of solving the model uncertainty problem of is to employ a different formal representation, namely the probability tree, for a theoretical probability than for the continuous probability distribution that is typically used for statistical uncertainties.  But, the other thing that needs to happen is that the responsibility map for the division of professional labor needs to be redrawn.  When uncertainties arise, it is generally thought that the solution is to send in a statistician.  In fact, many statisticians do indeed themselves consider the Other Probability to be their responsibility, and as a result, have devised various and sundry Bayesian schemes intended to stuff the recalcitrant prior probabilities into an aleatory mold.  But, that’s not really what is needed.  Probability trees are the domain of multi-handed scientists (David, 1975), so if you want to assign a probability to a theory, ask them what to do.  Even if they can’t identify the correct theory, they should be able to say which are more likely and why: Maybe that’s a good way to find out who the better scientists really are.

Examining Assumptions

When assumptions are justified by tradition or regulatory policy, then the process of providing estimates do not provide an occasion for questioning the validity of the premises.  Not very scientific: Policy In, Policy Out.  Maybe the PC answer is within the realm of plausible interpretation, but who would know?  On the other hand, if an estimate is purported to be valid, then the premises are bound to be questioned, especially when there is health or money at stake.  That will call for a supporting argument. 

If one premise can be proven to the exclusion of all others, then Truth In Truth Out.  In some cases, a sensitivity analysis may demonstrate that the alternative assumptions yield about the same answer, thereby making the issue moot. But, with low-dose risk estimation, that doesn’t happen; the choice of model used to make the risk estimate matters a lot (NRC, 1994).  So, you have a probability tree.  Yet, which alternative theories make up the tree still matters, and scientific arguments still need to made for either including them or not.  An uncertainty analysis may make it easier to come up with a risk assessment that is scientifically credible, but that can only happen if the set of alternative assumptions can be defended as being at least plausible, and the full set encompasses the entire range of assumptions that are distinctly possible:  Probable In Probable Out.

Bayesian solutions to model uncertainty often allow probabilistic judgments regarding theories.  However, the general inclination is to relegate such judgements as “priors” that are subsequently modified based on the available data.  There are a couple of problems with that.  First, judging theories does not happen in a data vacuum; data has a lot to do with it.  Second, and perhaps more importantly, new data may to revision of the scientific judgments of alternative theories, or maybe even result in the introduction of a new theory entirely.  If that happens, Bayes theorem will have nothing to do with it.

References

Cohen, L.J. (1977).  The Probable and the Provable.  Oxford: Clarendon Press.

David, EE (1975).  OneArmed Scientists?  Science 176: 679

Hacking, I (1975).  The Great Decision.  In: The Emergence of Probability.  Cambridge University Press.

Hume. D. (1739). A Treatise of Human Nature, EC Mossner (ed.), London: Penguin Books. 

Kaplan, S (1997).  The Words of Risk Analysis.  Risk Anal 17:407-411.

Official Post Soundtrack

Clan of Xymox (1989).  Imagination.  In: Twist of Shadows, Track 8.

Post Notes

Thesis Post #51, Part of the solutions thread.

Dietary Supplements

No Approval Necessary

From a legal standpoint, there is a third class of food chemicals in the United States that are neither food additives nor contaminants.  Like food additives, these chemicals have an intentional use; when consumers buy them, they expect or hope that it will have some desirable consequence.   Yet, unlike food additives, they do not require FDA approval before they can be sold.  This is pretty much true by definition: If a manufacturer wants to get FDA approval, they can.  But if they decide that is too difficult or not possible, they can call it a Dietary Supplement and sell it anyway.

There are some restrictions.  If a manufacturers are going to sell a new supplement, they are supposed to notify the FDA and send in whatever information they have about the supplement.  The FDA may decide to object to the sale, or not.  In particular, dietary supplements are subject to the same legal standard as unintentional contaminants, so if the product is hazardous enough to meet the “may render injurious” standard, then the FDA can prevent sale of the product. 

Yet outside the commonality of “no FDA Approval”, the range of products that are sold as dietary supplements is tremendous, as are the reasons for NOT seeking FDA approval.  Some may be sold as supplements because the process for getting approval is too expensive for small companies, which is probably the most legitimate reason.  Some may be sold as supplements because they couldn’t meet the rather strict safety standards required for food additives.  The supplement route can also be used to avoid the approval process required for drugs.   And finally, products that really have no use at all can rather easily be sold as dietary supplements.

Nutritional Supplements

If a product is sold as a dietary supplement, nutrition is what most people have in mind.  There are nutrients like calcium, iron, and vitamins that some people don’t get enough of in their diet, and therefore they need supplements to meet their nutritional requirements.  So, supplement manufacturers put nutrients in pills, and sell then to consumers that may need them. Furthermore, the amount of a particular nutrient in a supplement generally follows recommendations made by the Federal government.  Since nutrients can also be toxic, dietary guidelines consider both how the amount of the nutrient is that is necessary and how much may be excessive  (Institute of Medicine, 2013).  That all sounds good, and it is. 

However, that doesn’t mean supplement purveyors won’t encourage consumers to buy supplements they don’t really need.  Free enterprise, yay.  For example, you can buy a manganese supplement.  Yes, manganese is an essential nutrient – that has been demonstrated in animal studies with controlled artificial diets.  However, there is no evidence that anyone has ever suffered from manganese deficiency.  On the other hand, there is evidence from human studies that indicate that manganese is neurotoxic with symptoms that closely resemble Parkinson’s disease.

Drugs

Many of the substances sold as dietary supplements are really drugs.  While nutrients are essential to normal structure and function, drugs are not.  Before the Dietary Supplement Health and Education Act of 1994, selling a drug as a supplement was illegal, but now it is not.   Like the drugs that are regulated by the FDA as drugs, the drugs sold as dietary supplements have a wide range of effects.  Many of them can be used to treat various diseases.  There are also many psychoactive substances sold as supplements that alter mood or behavioral performance.  The best known example is caffeine, which is also used as a food additive.

Perhaps the best example of the drug subcategory of dietary supplements is ephedra, a plant that contains ephedrine.   Ephedrine is a sympathomimetic compound that has many of the same pharmacological properties as adrenaline and other substances that occur naturally in the human body.  It is also very similar to many of the drugs (e.g. pseudoephedrine or Sudafed) found in over-the-counter cold medicines that can be obtained in the U.S. and elsewhere without a prescription.  In China, the herb is known as Ma Huang, which like many other products sold as supplements, has a long history of use in Chinese medicine.  It can indeed be a useful treatment for colds.  Ephedra has also been marketed as a stimulant (i.e. in “energy” supplements) and for weight loss.   It works for that too.  But, like many other drugs, it can have side effects, and an overdose can be lethal.  In fact, many people have died from ephedra overdoses.   As a result, and because the FDA can regulate supplements when there is demonstrable harm, ephedra has been banned.  However, you can still grow ephedra yourself, or order synthetic ephedrine from Canada and have it delivered to the United States.

The drugs sold as dietary supplements may or may not be better than drugs approved by the FDA as drugs.  Leaving that issue aside, there is another problem:  Products that are sold as natural remedies in other countries are sold as dietary supplements in the United States, which one might expect to lead some people to think of those products as foods, rather than medicine.  Since people don’t expect to OD on food, they are more likely to do so with a dietary supplement like ephedra.  That’s a problem.

Placebos

The third category of dietary supplements are comprised of substances that pretty much don’t do a damn thing.  You can put dirt in a capsule and sell it as a dietary supplement, and guess what, people do.  Giving people the hope of a cure can, in fact, be a cure; so maybe they are worth something.  A good exemplar here is Laetrile, otherwise known as amygdalin, which is found in bitter almonds, apricot pits, and cassava.  It is called a cyanogenic glycoside because it releases cyanide when ingested.   Even though there is no scientific evidence to support claims that laetrile or amygdalin can treat cancer or any other illness, it has been marketed and used as an anticancer agent.  It was especially prominent in the US during the 1970’s, when it came to called “Vitamin B17” even though it isn’t a vitamin at all.   Although the evidence of harm at recommended levels of use isn’t especially strong, the cyanogenic properties of amygdalin led to an FDA ban in 1971.

The biggest downside of placebos is that they may be used instead of a treatment that would work.  Nonetheless, the main thing the FDA does with regard to placebos is to regulate health claims; a supplement manufacturer cannot claim on the label or packaging that a supplement will produce a desirable health effect without getting FDA approval first.  However, supplement manufacturers can put whatever claim they want on the internet, and they do.  For example, they will tell you that Laetrile works: Google it for yourself.

Reference

Institute of Medicine (2013).  Dietary Reference Intakes Tables and Application.  National Academy of Science, Washington DC.

Official Post Soundtrack

Pere Ubu (1977).  Final Solution.  In: Terminal Tower, Track 3.

Post Note

Thesis Post #50.  Part of the Regulatory Toxicology series.