How to judge email marketing software benchmark research

Email marketing software benchmark research should reject a number when its population or metric cannot be matched to the decision. A familiar percentage is not useful merely because it has an industry label.

This guide applies that test to email measurement and deliverability reporting. It sets out four stages: capture the source record, test statistical comparability, test measurement comparability, and classify each figure before anyone acts on it. It also names the UK rules that shape what email data may be collected, and the platform and vendor reports that supply most published email benchmarks.

What to take away

Write the comparison question

State the decision, geography, business population, recipient population, message class, channel, period and exact metric. Distinguish a baseline from a benchmark and both from a threshold. A baseline describes the organisation's own defined history. A benchmark is an external comparison. A threshold is a prospective action rule set by accountable owners.

Capture the source record

For every candidate figure record publisher, title, release date, fieldwork or observation period, geography, target population, achieved sample, recruitment or inclusion route, weighting, provider mix and funding. Then copy the metric's event, numerator, denominator, unit, timezone, deduplication, exclusions and uncertainty.

Most published email benchmarks come from two kinds of source: aggregate reporting by email service providers such as Mailchimp, Klaviyo and Campaign Monitor, and deliverability research from vendors such as Validity and Litmus. Each describes the senders on its own platform or panel. A number from any of them is a candidate figure, not a conclusion.

The Code of Practice for Statistics governs official statistics, not marketing reports, so it cannot certify an email benchmark. Its trustworthiness, quality and value questions are still a sound prompt for provenance, and a supplier's report remains a supplier's report.

Test statistical comparability

Ask whether the sample represents the UK audience you sell to, whether message purposes and recipient categories match, and whether small subgroups are disclosed. Preserve missing data, selection effects and changes in collection.

Sample quality has the same problem in email. A benchmark built from senders who opted into a platform's reporting panel, or from a single client's customer base, describes that group rather than the market. Ask whether the sample represents the senders you compare yourself with, whether small segments are disclosed, and whether the collection method changed during the period.

Where a source provides a range or uncertainty measure, retain it. If it does not explain sampling or coverage, mark uncertainty unresolved. Do not calculate a confidence interval from missing design information.

Test measurement comparability

Open rate, click rate, bounce rate, spam-complaint rate and inbox placement hide different events behind similar labels. Compare tracking state, treatment of machine opens and prefetching by privacy features such as Apple's Mail Privacy Protection, identity matching, observation window, provider status mapping and denominator. Deliverability also depends on sender authentication (SPF, DKIM and DMARC), sending reputation and list hygiene, so two groups of senders with different authentication and reputation practice are not directly comparable. The ICO's storage and access technology guidance means privacy configuration can affect both lawful collection and observed coverage.

Data minimisation can also remove identifiers or shorten retention. The ICO's data-protection principles guide and its direct marketing guidance, alongside PECR and the UK GDPR, set the regulatory context for what may be collected and kept. A richer benchmark is not automatically preferable if it requires incompatible or unjustified data.

Decide whether to use the figure

Classify each candidate as comparable, usable with a stated adjustment, contextual only or excluded. Name the reviewer and decision. Do not average excluded figures or fill a UK gap with a global result.

For causal or financial decisions, a descriptive benchmark is especially weak. An external average of open or click rates cannot show what your own campaign caused; that needs a controlled test with a holdout group and a defined measurement window. A benchmark tells you where you sit, not what happens if you change something.

The practical output may be "no usable benchmark". In that case, build a versioned internal baseline from stable definitions, report its selection limits and set action thresholds from risk and tested capacity rather than pretending the baseline is an industry norm. Schedule a new search when sources, provider definitions or privacy conditions change.

Before you act

Common questions

Why is a familiar email marketing percentage not automatically useful?

A familiar percentage is not useful merely because it has an industry label. Its population, metric definition and privacy basis must be matched to the decision you face. If you cannot describe the senders, the period and the denominator behind it, reject it.

What is the difference between a baseline, a benchmark and a threshold?

A baseline describes the organisation's own defined history. A benchmark is an external comparison. A threshold is a prospective action rule set by accountable owners. The article says to distinguish all three before using any figure for decisions.

How should you handle uncertainty when a source does not explain sampling?

Where a source provides a range or uncertainty measure, retain it. If it does not explain sampling or coverage, mark uncertainty unresolved. Do not calculate a confidence interval from missing design information. This prevents false precision in comparisons.