The Literacy the Marketing Fears
Every claim in this series, from the psilocybin trials to the CBD marketing to the nitazene alerts, ultimately rests on studies, and the reader who can read a study is the reader the marketing cannot fool: the difference between a randomized controlled trial and an observational survey, between a large effect and a statistically-significant-but-clinically-trivial one, between a preprint and a peer-reviewed finding, between an outcome the researchers measured and one the press release invented, is the difference between the informed and the marketed. This post is the literacy guide: the study-design hierarchy, the critical-reading questions, the statistics that matter, and the specific failure modes (the p-hacking, the surrogate endpoints, the publication bias, the press-release amplification) that this series' skeptical coverage has invoked throughout, here assembled into the toolkit the reader can apply to any claim, in this field or any other.
The Hierarchy of Evidence
The evidence pyramid deserves its plain-language statement, because everything else follows from it:
Systematic reviews and meta-analyses (the studies of studies: the systematic literature searches with the pooled statistical analyses, the highest tier when done well and the garbage-in-garbage-out amplifier when their inputs are weak); randomized controlled trials (the participants randomly assigned to the intervention or the control, the design that distributes the confounders evenly and isolates the intervention's effect, the gold standard when the blinding holds and the series' placebo coverage has documented how rarely it fully does); cohort studies (the observational followings of defined groups, the smokers versus the non-smokers tracked for decades, powerful for the questions trials cannot answer and confounded by the differences between the groups that no adjustment fully removes); case-control studies (the retrospective comparisons of the cases and the controls, efficient for the rare outcomes and vulnerable to the recall and selection biases); case series and reports (the careful documentations of individual cases, the hypothesis-generators and the safety-signal detectors, the lowest tier and the irreplaceable one for the rare events); expert opinion and mechanistic reasoning (the bench and the theory, valuable for the plausibility and incapable of establishing the effect).
The series' coverage has invoked the hierarchy constantly: the psilocybin trials' effect sizes meaningful because they came from randomized designs, the survey literature's (the kratom, the migraine, the cannabis-use surveys) signal limited by their observational tier, the case reports' (the CHS, the HPPD, the B12-neuropathy) detection of rare harms that no trial's sample would capture.
The Critical Questions
The reading protocol, compressed into the questions the informed reader asks of any study:
Who funded it and who conducted it? The conflict-of-interest question, not disqualifying (industry funds much valuable research) but always informative: the supplement-industry study of the supplement, the company trial of the company's drug, the advocacy-funded survey of the advocacy's population, each wearing its conclusion's costume. The registration requirement (the clinical-trial registrations that the journals now demand, the checkable record of what the researchers planned to measure before the data arrived) is the reader's protection, and the unregistered study's surprise findings deserve the discount.
Was there a control group, and was it blinded? The placebo post's lesson: the no-treatment comparison and the unblinded design cannot separate the intervention from the expectation, and this series' field (the maximally-expectation-sensitive psychedelics) is the extreme case of the general principle.
What was the outcome measure, and was it what matters? The surrogate-endpoint problem (the proxy that the researchers measured instead of the outcome the patient cares about: the lab value instead of the lived function, the depression-scale score instead of the life, the biomarker instead of the mortality) and the outcome-switching problem (the endpoint the trial registered versus the one the paper reports, the registration check that catches it).
How big was the effect, in absolute terms? The relative-versus-absolute distinction (the 50-percent risk reduction that is the one-in-two-thousand absolute difference, the framing that the marketing exploits and the reader must correct) and the clinical-significance question (the statistically significant difference too small to matter, the p-value's silence about the magnitude).
Who was studied, and does it include me? The generalizability question (the trial population's differences from the real-world patient, the exclusion criteria's shaping of the sample, the series' women's-health and elderly and adolescent coverage's demonstration that the studied population is rarely the needing one).
Where was it published, and in what form? The peer-review's filter (real but imperfect), the preprint's speed-and-risk (the unreviewed posting that the COVID era normalized, the findings that the preprints carried and the journals later reversed), the journal-tier's signal (the specialty journal's lower bar, the predatory journals' no-bar), and the press-release's transformation (the university-media-office amplification that the series' gold-rush coverage documented, the headline's divergence from the abstract's caution).
The Statistical Literacy
The minimal statistics deserve the plain-English treatment: the p-value (the probability of the observed result under the null hypothesis, the arbitrary 0.05 threshold's reign and the growing reform movement's critique of its tyranny), the confidence interval (the range of plausible effect sizes, the width that tells the precision story the p-value hides), the effect size (the magnitude, the thing that matters, the Cohen's d and the odds ratio translated into plain differences), the absolute-versus-relative risk (the framing vulnerability), the number needed to treat (the inverse of the absolute benefit, the trial translated into the clinic's real question), and the multiplicity problem (the many-comparisons trap that the unregistered fishing expeditions exploit, the correction requirements that the rigorous journals enforce and the predatory ones do not).
The Field's Specific Failure Modes
The series' recurring critiques, assembled as the field-specific checklist: the expectancy-confounded psychedelic trials (the blinding's impossibility, the placebo post's demand for active placebos and expectancy analyses); the survey-literature overreach (the self-selected samples, the expectancy-saturated reports, the observational-tier ceiling); the preclinical-overclaim (the animal findings translated to human promises, the BDNF and the critical-period coverage's discipline about the translation gap); the pharmacokinetic-naive dosing (the fixed-dose trials in the pharmacogenetically-variable population of the pharmacogenetics post); the publication bias (the positive-results bias that the trial registries were designed to catch, the file-drawer problem that the meta-analyses must model); and the press-release amplification (the institutional-media arms race that the gold-rush post documented, the stock-price reporting that the public-company era added).
The Bottom Line
The study-reading literacy is the series' skeptical framework in portable form: the hierarchy of evidence (the design that answers the question), the critical questions (the funding, the blinding, the outcomes, the magnitude, the population, the venue), the minimal statistics (the p-value's threshold, the confidence interval's width, the effect size's magnitude, the absolute risk's honesty), and the field's specific failure modes (the expectancy, the surveys, the animals, the press releases). The reader who carries the toolkit is the reader the marketing cannot reach: the supplement label's claim sorted into its evidence tier in seconds, the press-release headline weighed against the abstract's caution, the exciting new finding placed on the pyramid where it belongs. This series has tried to model the literacy throughout. This post makes it explicit, which is the only education that lasts: not the conclusions, but the method, ready for the studies the next decades will publish and the claims the next markets will make.
