A Flawed Minimum Wage Study Shows How Bad Stats Get Turned Into Policy Gospel

The minimum wage is on the rise in the U.S. Thirty states, the District of Columbia, and at least 68 localities have set hourly compensation above the federal floor of $7.25. New York City, Long Island, and Westchester are at $17. In California, fast-food establishments are required to pay their workers $20 per hour.

Advocates for these laws point to academic studies that claim higher minimum wages don’t cost jobs. Many of them use statistical techniques that generate misleading results or are misinterpreted by the media. A 2024 paper published in The Review of Economic Studies titled “Minimum Wage Employment Effects and Labor Market Concentration” didn’t get as much press attention as some studies in this canon, but the paper is still worth examining because its statistical missteps are typical of academic research published in prestige journals.

Co-authored by five economists, the paper found that minimum wage increases haven’t led to job losses at big-box stores, concluding that these “monopsonistic” establishments therefore must systematically underpay their workers. The University of Pennsylvania, where one of the co-authors works, promoted the study in an article titled, “Increasing minimum wage has positive effects on employment.”

The authors reached their conclusion by dividing American counties into those with retail sectors dominated by a few large employers (call these “the Big-Box counties”) and those with more diverse retail employers (“the Mom-and-Pop counties”). Next, they looked at how employment of store clerks, order fillers, retail salespeople, and cashiers in the general merchandise sector responded to minimum wage increases relative to overall county employment.

They found that in Big-Box counties, a 10 percent higher minimum wage was associated with 1.12 percentage points lower job losses among store clerks than among county workers overall. In Mom-and-Pop counties, the same wage increase was associated with 1.79 percent larger job losses among store clerks than among overall county workers. In other words, the relative loss of low-level retail jobs was smaller in the Big-Box counties than in the Mom-and-Pops.

The authors presented this finding as evidence that workers were being underpaid in areas that experienced smaller job losses. Big-Box employers have “more wage-setting power” and thus must “tend to pay workers less,” as the University of Pennsylvania summarized it. The Washington Center for Equitable Growth cited this paper in an essay arguing for expanded antitrust enforcement and higher minimum wages. The study also appeared (in its preprint form) in a roundup by Noah Smith of recent scholarship demonstrating how economists have become “unlikely crusaders” against monopolistic power.

The study took a bizarre logical leap. A reasonable interpretation of why Big-Box counties experienced lower unemployment after a minimum wage increase is that store clerks at companies like Walmart already earn more than clerks at Mom-and-Pop establishments, so a minimum wage increase is less likely to artificially boost their earnings above the marginal revenue product of their labor. The study also didn’t prove that minimum wage increases cause employment increases, as the University of Pennsylvania claimed; it only showed that average county workers get laid off at higher rates than Walmart clerks.

There’s another problem with this paper that’s common in academic research: its overreliance on “p-value,” which indicates the statistical significance of its findings. A p-value is not enough to tell us whether a finding is the result of pure chance. For that, we also need to know two more p‘s—prior probability and the power of the test. This study, like many, leaves out the two extra p‘s.

To illustrate what I mean, consider going to a baseball game and seeing one player get four hits in four at-bats. That’s a good day for the player, but four-for-four games happen about twice a week during the Major League Baseball (MLB) season. Nevertheless, our single observation could support a paper claiming that the player is the best hitter in baseball history, “significant at the 5 percent level.”

What exactly does “significant at the 5 percent level” mean? Let’s say that to be the best hitter in baseball, the player has to at least match Tetelo Vargas’ batting average of 0.471 in 1943. The chance of a 0.471 hitter going four-for-four is 4.92 percent. Since that’s less than 5 percent, we say we reject the null hypothesis that the hitter we just saw had a batting average of 0.471 or lower at the 5 percent level. So we can publish the claim that we just saw the best hitter in baseball history. But remember, going four-for-four happens about twice a week in MLB, and every player who achieves this feat has a lower batting average than Vargas.

The problem is that we over-relied on the p-value and ignored prior probability and power. There have been about 25,000 hitters in MLB history. If we test all of them at the 5 percent level, we expect 1,250 of them to be falsely identified as the best hitter in the history of baseball. That’s why prior probability matters.

Meanwhile, if we did happen to catch Tetelo Vargas during the 1943 season, he went four-for-four in only one of 30 games. That makes this a low-power test; we catch only one out of 30 true results. So we get 1,250 false results and one-thirtieth of a true one.

Back to the minimum wage paper. Its p-value is around 2 percent, which is below the required 5 percent threshold for publication, supporting the claim that a 10 percent higher minimum wage is associated with 1.12 percentage points lower job losses among store clerks than among overall workers in Big-Box counties.

The authors committed the same fallacy as claiming that a player is the best hitter in history based on going four-for-four in one game. The test is too low-powered. There are only three counties in the U.S. with county-wide minimum wage regulations higher than state and federal rules: Howard County, Prince George’s County, and Montgomery County, all in Maryland. All are Big-Box counties. Their minimums range from $15.30 to $17.65. There’s not much ability to notice a 1.12 percent difference in job losses from that sample.

I’m simplifying the analysis; the authors do make use of less direct comparisons. But given all the noise and complexities, it seems unlikely a 1.12 percent difference would stand out. I think I’m being generous in saying there’s 20 percent power to their test

Let’s say we run this test on 1,000 similar hypotheses. The hypothesis that minimum wage increases affect average county workers more than Walmart clerks has a low prior probability. So let’s say 5 percent (50) of the 1,000 are true. At our assumed 20 percent power, we flag 10 to publish as claims. Of the 1,000 hypotheses, 950 are false, and at a 2 percent p-value, 19 of them pass statistical muster. So, it’s 10 out of 29, or a 34 percent chance, that this claim is true. No reasonable combination of prior and power assumptions makes the study’s result more likely true than false.

Researchers do not report priors and power because they are not required to. Journals don’t require priors and power because there is no consensus on how to estimate them, and because requiring them would force authors to make a much weaker claim than “statistically significant at the 5 percent level.” Press offices love p-values because the word significant sounds scientific. Journalists rarely look past the press release. Activists and politicians cite the resulting headlines as if they were settled facts. Each link in the chain has a small incentive to keep the misunderstanding going, and almost nobody has an incentive to break it.

I picked on the minimum wage paper because it’s an unusually clean example of overemphasizing p-values, but this practice is a common distortion in academic studies.

The argument against minimum wage increases must be made on its own terms. I’m taking aim at this trick of taking a marginal academic finding, stripping out its uncertainty, and restating it as a sweeping policy claim.

The same trick is run on every spending program, every regulation, every tax, and every policy of any constituency that wants to dress something up as a scientific finding. A study reports a 1.12 percent effect with a 2 percent p-value, and, somewhere downstream of the press release, a politician quotes it as proof that a favored policy is the wise course of action. The reader who has learned to ask for the prior and the power is, structurally, a much harder reader to fool. Run the arithmetic on the next study you see cited as an established fact, with whatever priors you find honest. Most of the time, the implied probability that the claimed result is correct will land somewhere between a quarter and a half.

The federal minimum wage has not moved since 2009. The state laboratories of democracy have, in the meantime, run an enormous natural experiment on the question of what happens when you raise it. But the science has been badly weaponized for a policy fight, exploiting a popular view that a p-value below 5 percent is enough to draw a meaningful conclusion.

This article is excerpted from the book Wrong Number: How to Extract Truth From a Blizzard of Quantitative Disinformation by permission of Wiley. The book is based on Aaron Brown’s video series of the same title with Reason.