Three traps in ecommerce data, and how to find them

Every time you look at a comparison between two periods there is a temptation to stop at the percentage. It is the number that goes in the slide, the one said out loud in the meeting. It is also where bad decisions get made, because a percentage never tells you on its own whether it is true.

We met all three of these traps working on real data from real stores. They are not textbook cases: they are the concrete ways a correct number tells a false story.

First trap: the month that was not there

On one store, June of one year showed 8 orders and just over 900 euro, against 34,000 euro in the same month the following year. An increase of several thousand per cent.

Except that June was not a bad month: it was a broken one. A tracking problem, a configuration change, something that stopped what was happening from being recorded. The sales had happened, the data had not.

Any comparison including that month produces meaningless percentages, and the trouble is that they look excellent. Nobody questions a +3,000%: they take it to the meeting.

How to find it: before calculating any change, look at the series month by month. A month worth a tenth of its neighbours is not a weak month, it is a month to check.

Second trap: unobserved does not mean zero

This is the subtlest, and it catches anyone who connected a tool partway through.

If an ad platform connector was switched on in March, January and February show as empty in your dashboard. Empty does not mean nothing happened: it means nobody was watching. Those campaigns may well have been running, with their spend and their revenue.

Treating that gap as a zero produces spectacular and entirely invented growth. It is the same error as the broken month, but worse, because here the data is missing for a technical reason nobody remembers a year later.

How to find it: always ask when the data starts, not when the company started. If the tool declares a history start date, that is the date before which you cannot compare anything. And check per platform, because each may have been connected at a different moment.

Third trap: the blend that is a mix effect

The most elegant of the three, because no individual figure is wrong.

On one store the blended cost per click showed a fall of 18.8%, with click-through rate up 57.9%. Isolating the single platform, the real values were -3.6% and +29.4%.

Neither calculation was incorrect. What had changed was not performance: it was the weight of each platform in the total. One of them, with a structurally very low cost per click, weighed far more in the recent window. The total shifted towards it, and the blended number recorded that shift as if it were an improvement.

It is the same mechanism by which a class's average grade can rise without any student improving: it is enough for the class to change.

How to find it: whenever a ratio (cost per click, click-through rate, conversion rate, average order value) comes from summing different sources, recalculate it per source. If the two versions diverge a lot, the difference is mix, not merit.

Why these three are enough to ruin an analysis

They share one property: none of them produces a visible error. There is no red box, no warning, the spreadsheet does not complain. The result is a plausible, often flattering number that holds right up until somebody with the data asks a question.

Which is why they are worth checking before showing a percentage to a client, an investor or your own team: not for academic rigour, but because the worst moment to discover it is after you have already told the story.

Frequently asked questions

How do I tell a broken month from a merely bad one?

Look at several metrics together. A genuinely bad month has low orders, low sessions and consistent spend. A broken month has inconsistencies: plenty of sessions and almost no orders, or normal spend and zero conversions. The signal is the inconsistency between metrics, not the low value.

If history is missing, what do I compare?

Only the period actually observed on both sides. If you have data from March, the comparison starts in March, and you say so. It is less spectacular and has the advantage of being true.

Does this apply to conversion rate too?

Yes, and it is one of the cases where the mix effect bites hardest. Overall conversion rate changes when traffic composition changes, even if no single source improved. It has to be read per source, always.

How far back should I look?

Far enough to see a full season, because nearly every ecommerce has a seasonality that on its own explains more than half the variation. Comparisons over short windows say more about the calendar than about the work done.

How do you present a figure with a known limitation?

By stating it next to the number, not in a footnote. A percentage with its caveat stays credible even when somebody checks it. A percentage without one, when they check it, takes all the others down with it.