Email marketing software measurement mistakes

Email marketing software measurement mistakes usually begin before calculation: an event is vague, a denominator shifts or an attribution rule is mistaken for causality. The remedy is a reproducible definition and an honest evidence label.

This non-ranked list was compiled on 5 September 2026 for organisations operating in England. An item qualified when it could change a metric's numerator, denominator, interpretation or decision. The review excluded product defects, supplier rankings, performance rates and invented examples. No platform was tested, and no supplier funded the work.

These issues apply across email marketing software such as Mailchimp, Klaviyo, HubSpot, Salesforce Marketing Cloud and Campaign Monitor. Report labels and filters can differ by product, so verify each export's definitions rather than assuming a shared standard.

What to take away

Define events and denominators before calculating, or two analysts will produce different answers from the same export.
Provider acceptance does not prove inbox receipt or human attention, so keep counts at each stage.
A last-touch rule allocates recorded value, while causal impact asks what would have happened otherwise.
Report missing data as missing, not zero, and record the expected arrival, observed arrival and repair owner.
A metric that cannot change a decision is clutter, so define the warning, trigger and owner.

1. Starting with the dashboard label

"Open" and "click" do not define events. Record whether the numerator counts requests, events, messages or deduplicated recipients, then identify the eligible denominator. Without this, two analysts can produce different answers from the same export.

For example, report provider-accepted messages ÷ attempted sends as an acceptance rate. Report opens and clicks against the same explicitly defined delivered-message base; if delivery cannot be observed separately, say the denominator is provider-accepted messages, not inbox receipts.

2. Mixing attempted, accepted and received

A send instruction is an internal event. Provider acceptance is supplier-reported. Neither proves inbox receipt or human attention. Keep counts at each stage and reconcile differences rather than renaming them delivery.

Classify hard bounces (permanent delivery failures) separately from soft bounces (temporary failures). Monitor spam complaints against Gmail and Yahoo's bulk-sender requirements and their complaint-rate ceiling; do not treat an external benchmark as a safe target.

3. Ignoring tracking effects

Pixels, redirects, automated activity and privacy controls affect what can be observed. The ICO's storage and access guidance requires analysis of the particular technology and purpose. Analytical convenience is not PECR permission.

Apple Mail Privacy Protection (MPP) can inflate recorded opens by fetching remote content without a person opening the message. Security scanners, click bots and link prefetch can also register clicks without human action, so document whether the ESP filters or flags these events and avoid treating raw opens or clicks as human attention.

For deliverability context, Google Postmaster Tools provides Gmail-domain reputation and spam-rate signals, while Microsoft Smart Network Data Services (SNDS) provides data about mail sent to Outlook.com users. Seed-list inbox placement testing can supplement these aggregate signals, but does not prove placement for every recipient.

4. Changing exclusions after seeing the result

Late decisions to remove test records, inactive recipients or inconvenient outcomes can bias a comparison. Save the population rule, query version and exclusions before extraction. Treat later corrections as a new version with an audit trail.

5. Hiding missing data as zero

A failed source load is not evidence that no event occurred. Report missing fields, unmatched identities, late records and broken joins. The government's Data Quality Framework treats quality as active lifecycle work tied to intended use.

Keep the expected source arrival, observed arrival, affected period and repair owner in the report. Readers can then see whether the value is incomplete or genuinely zero.

6. Calling attributed revenue incremental

A last-touch rule allocates recorded value. Causal impact asks what would have happened otherwise. The Magenta Book separates monitoring from impact evaluation in its government context. Use a credible comparison and proportionate design before making a causal claim.

A practical incrementality method is a randomised holdout: assign eligible recipients to receive the campaign or to a no-campaign control before sending, then compare the same outcome over the same period and analyse by assignment. This estimates the campaign's incremental effect for the tested population; last-touch attribution remains an allocation rule.

7. Combining incompatible periods

Timezone, send window, late outcomes, refunds and attribution windows can place related events in different periods. Publish the clock start and stop, cut-off and restatement policy. Preserve prior values when a late-data rule changes.

8. Copying a benchmark without its population

An external percentage is unusable when its geography, message types, provider mix, sample, denominator or privacy settings differ materially. ONS guidance on quality in official statistics is scoped to official statistics, but its relevance and comparability questions are useful here.

Retain the rejected source and reason in the research log so the same incompatible figure is not reintroduced later.

9. Removing uncertainty from the headline

Precise decimals do not repair incomplete identity matching or automated observations. State the source of uncertainty, likely direction and whether it could change the action. The Analysis Function provides guidance on communicating quality and uncertainty for public analysis.

10. Reporting without an action owner

A metric that cannot change a decision is clutter. Define the warning, review trigger, stop condition, recovery evidence and owner. Privacy, PECR, accessibility, claims and finance gates remain separate even when the dashboard is numerically healthy.

Before you act

Record whether the numerator counts requests, events, messages or deduplicated recipients.
Save the population rule, query version and exclusions before extraction.
Report missing fields, unmatched identities, late records and broken joins.
Publish the clock start and stop, cut-off and restatement policy.
Retain the rejected source and reason in the research log.
Define the warning, review trigger, stop condition, recovery evidence and owner.
Record whether Apple Mail Privacy Protection (MPP) and automated-click filtering affect open and click totals.
Classify hard and soft bounces separately and monitor complaint rates against Gmail and Yahoo's bulk-sender requirements.
Review Google Postmaster Tools, Microsoft SNDS and seed-list inbox placement tests as complementary deliverability evidence.

Common questions

Why is provider acceptance not the same as delivery?

A send instruction is an internal event, and provider acceptance is supplier-reported. Neither proves inbox receipt or human attention. The article says to keep counts at each stage and reconcile differences rather than renaming them delivery, so the stages stay distinct.

What makes an external benchmark unusable?

An external percentage is unusable when its geography, message types, provider mix, sample, denominator or privacy settings differ materially. The article says to retain the rejected source and reason in the research log so the same incompatible figure is not reintroduced later.

How should a team treat late corrections to a comparison?

Late decisions to remove test records, inactive recipients or inconvenient outcomes can bias a comparison. The article says to save the population rule, query version and exclusions before extraction, and treat later corrections as a new version with an audit trail.