When big data falls short

The  OSUN Economic Democracy Initiative  research team looks at the reliability of the widely used Employment data series from the  Opportunity Insights Economic Tracker . We find significant anomalies that cast doubt over its usefulness for policy or research purposes.

Please visit our main  When Big Data Falls Short page  to view the overview on our analysis of Economic Tracker employment series first published in early April 2021.

According to the Economic Tracker, private-sector employment in Florida is at record highs — up over 16% relative to January 2020. The latest BLS data, however, reports that Florida’s private-sector employment remains well below pre-pandemic levels.

The same in California — the Economic Tracker reports private-sector employment in mid-March 2021 was up over 7% relative to January 2020 while BLS data reported private-sector employment down by about 8% over the same period.

More surprising, the latest Opportunity Insights Economic Tracker employment data reports that private-sector employment in mid-March 2021 for the U.S. was down 7.1% relative to January 2020. Yet in the Economic Tracker state-level data series, only three states (plus Washington D.C.) have employment below their January 2020 level.

And of these three, employment in two states — Hawaii and New Mexico — is reported to be almost back to pre-pandemic levels. Hawaii's employment comes in at only -0.7% relative to its January 2020 level while New Mexico is even closer at -0.6%. The only state in the Economic Tracker employment data still widely below January 2020 levels is West Virginia, down 4.7%.

In other words, the Economic Tracker reports that private-sector employment is down 7.1% even though only one state is significantly below its pre-pandemic level — and that state, West Virginia, is reportedly down ‘only’ 4.7%.

How is this possible? There seems to be a similar aggregation issue when comparing the county and state-level data as well.

For example, look at the county-level employment data for New York state. The Economic Tracker does not list a single county in the state of New York as having private-sector employment above their pre-pandemic levels (employment data for three countries — Cattaraugus, Schohaire, and Schulyer — are not available).

Yet the Economic Tracker reports that for the state of New York as a whole, private-sector employment is up 4.7% relative to January 2020. BLS data however reports that private-sector employment in New York state is still below pre-pandemic levels.