A Report Anomaly or a Data Error? 9 Real Cases and a One-Hour Checklist

How an “organic surge” turned out to be a budget error

One morning, marketing brought a report: in the first week of the month the number of new accounts had jumped, and they were organic. Nobody celebrated. The first guess was bots, and the analysts were asked to find where the bot script ends.

They found no bots. They found an incorrectly filled budget document: because of it, other data in the report was double-counted, including organic traffic. Once fixed, the number nearly halved and the “surge” was gone.

For many years I led analytics and data teams in online services and retail, and stories like this happened again and again. The checks were done by analysts on my teams, conclusions were reviewed by leads and accepted by project teams. My job was to make sure the right number reached the decision. Absolute amounts are removed below, but the error mechanics and proportions are real.

Why a report error is dangerous

A wrong number costs nothing by itself. What is expensive is the decision made on it:

• cutting an ad channel that actually works, or moving budget into a channel that only looks good;
• spending weeks hunting for bots that do not exist and tightening filters that also block real customers;
• scaling a promotion or email campaign that earns less than the report shows;
• sending a partner or the accountants numbers you later have to explain.

Checking a report takes an hour; a decision based on a wrong number lives for months. So for each case below: the symptom, the business’s first guess, the check and what it actually was.

Four types of data errors

1. Errors in source documents

Double counting from one document. This is the story from the start of the article. A manual entry error in one document went through the chain of calculations and affected metrics not directly related to the budget.

What to do: check manual documents that feed calculations as strictly as code.

Sales that never happened. In retail, some receipts are parked at the till without payment. They should be deleted at the end of the shift and never reach the books. Sometimes, for technical reasons, they stayed longer than a day, the data warehouse picked them up and they appeared in sales statistics as real. Such documents are now filtered out on load.

What to do: take document status in the accounting system into account. It is data too, and without it the report shows sales that never happened.

2. Errors in calculations or filters

Traffic dropped, although nothing changed in the campaigns. A report showed a sharp traffic drop in one area. Marketing checked the campaigns: no changes. The cause was in the warehouse: the procedure that builds the metric had been changed with a bug. It was rolled back and the history recalculated.

What to do: if a metric breaks while nothing changed in the business, first find out what changed in the calculations.

Inactive users suddenly grew. Over two weeks, the number of “inactive” users in one product jumped. First guess: low-quality traffic. We broke the losses down by channel: organic lost 34% of users, paid channels 42%. The shares are close, there is no single “bad” channel, so traffic is not the cause. The cause was in the data export: a filter that excludes people active in the company’s other products had not been selected. The export instructions were updated afterwards.

What to do: if an “anomaly” is evenly spread across all channels and segments, look for an error in how it is counted, not in a single source.

3. Errors in metric rules and definitions

“Organic” purchase growth. A new product unexpectedly showed purchase growth without advertising. The business decided a placement with an external partner had worked, but it was never published. The cause was the attribution rule: if a person came to the new product and had been active in another company product in the last 30 days, that other product is counted as the source. In the same weeks, a joint promotion of the two products was running, and its participants produced all the “organic” growth.

What to do: before drawing conclusions about organic growth, open the attribution rules and the promotion calendar. If such stories keep happening, it is time for an attribution and marketing data audit.

The email report attributes extra money. Email campaigns were evaluated with a report on revenue from returning customers. The check found three problems: money was not cleared of taxes and fees, there was some double counting, and a customer’s return was counted by a purchase in one product while money was counted across the whole account. As a result, for some customers the report showed more money than they had brought the company at all.

What to do: write down the definition of every “money” metric: which money, over what period, at what level. And measure the effect of emails with a control group. How this works is described in the honest CRM evaluation service.

4. Incomplete data

A month with 30 days instead of 31. Every month the company sends a partner a report on daily activity. One month the export had 30 days out of 31, the next month 29. A year earlier the same had happened. The cause was partial data loss for some days, and the instructions for fixing such failures contained an error too. The data was restored and the instructions corrected.

All days present, but the numbers don’t match. After a failure, the partner report differed from the control table by 3–4%, although all days of the month were there. Part of the events within days was missing. The data was reloaded from backups and everything matched.

What to do: check not only that all dates are present, but also the volume of data within each date. This is easy to automate: compare the number of incoming events with a typical day every day.

Why the accounting system, BI and the books show different revenue

The most common dispute in companies: which revenue is right. The accounting system shows one number, BI another, the accountants a third. Usually all three are right in their own way; they simply count different things from different documents.

Accounting system vs BI. In a retail business with an online store, the weekly report from the accounting system did not match BI. A one-year reconciliation showed a noticeable total discrepancy. The causes:

• some online orders were marked as offline, and vice versa;
• some sales had no store specified;
• different accounting databases had no shared product and store reference data;
• some accounting reports were built from summary documents that close late, while BI was built from source documents: receipts, sales, returns;
• documents were edited retroactively in closed periods, and report history kept changing.

After cleanup, the online share in most segments rose to 20–30%. So the online channel had been underestimated, and decisions about it were made on an understated number.

Another reconciliation does not fix this. The team agreed that the accounting system would become the single source for all reports and added mandatory fields to order documents: channel, store type, fulfilment method. The new report matched BI by 95% on average. There were pitfalls: the new fields are only recorded from the go-live date, history had to be backfilled, and the same fields were filled in differently in different databases. We do this kind of work as part of the unified reporting reconciled with accounting service.

Commercial database vs the books. Finance brought a monthly reconciliation by counterparty. The causes were mundane: the accountants post adjustment documents to a different account, so they drop out when reconciling by one account; negative marketplace reports were moved to current dates; one-rouble differences came from rounding, probably on VAT.

The most interesting part is the proportions. In one month, adjustments explained about 30% of the difference. In another, their sum was 15 times larger than the final difference: errors with opposite signs almost completely cancelled each other out, and the total looked almost perfect.

What to do: don’t trust a small final difference; reconcile line by line.

Checklist: report anomaly or data error

These seven questions cover most of the cases above. Save them or forward them to your team.

1. Is it visible in another source? Compare the report with raw events, a neighbouring report or the books. If the spike is in only one place, the problem is almost certainly there.
2. What changed on that date? Not in the business, but in the data: calculations, reference data, manual documents, attribution rules, export settings.
3. Is the data complete? Are all days present, and is the event volume within each day normal?
4. How is the metric defined? Which money (before or after taxes and fees), at what level (purchase, customer, whole account), which filters and exclusions the report applies.
5. What else happened in that period? Promotions, joint campaigns, launches, outages: anything that could have moved the number besides your hypothesis.
6. Is the anomaly even? Break it down by channel and segment. The same everywhere — the counting is probably broken. Concentrated in one channel — investigate that channel.
7. The reconciliation matches? That’s not all. A small final difference can hide large errors with opposite signs. Check at least a sample line by line.

How to reduce data errors: five rules

The checklist helps once an anomaly has happened. To make them rarer, five rules are enough:

• Every key metric has an owner and a written definition: which money, at what level, for what period, with which filters. Then the “which number is right” dispute takes a minute. This is where data governance and a metrics catalog start.
• There is a log of data changes: what changed and when in calculations, reference data and attribution rules. Several cases above would have been solved with one look at such a log.
• A machine checks data completeness: number of days, events per day against the usual level, matching totals between sources. The alert arrives before the report goes to a partner. This is how anomaly monitoring works.
• One source of truth and mandatory fields in documents. Edits to closed periods only with a trail: who, when and why.
• Manual documents that feed calculations are checked by a second person. Or at least by a checksum.

Together, this is data management. It sounds dull, but it is exactly what separates a report you can allocate a budget on from one you have to double-check every time. If it is unclear where to start, a maturity audit will help: it shows the main gaps and a 90/180/365-day plan.

Instead of a conclusion

In all nine cases, the cause was not the market but documents, calculations, filters and rules. So when a metric jumps, I first ask what changed in the data, and only then what changed in the business.

If you have a number you don’t trust, send it to me on Telegram @OlegTsarkovv or book a free 30-minute call. I’ll take a look and tell you where to start checking.