Reviews
Part of Comparing email software when the verdict is a process rather than a winner
Email marketing software review methodology: gates first
Email marketing software review methodology: fixed criteria, evidence ladder, non-compensating gates, deliverability checks and dated rechecks.
An email marketing software review methodology should make it possible for another reviewer to reach the same evidence record, even if that person reaches a different judgement. It begins with the buyer's workflow, not a catalogue of attractive features.
An editorial protocol for reviewing email marketing software should be reproducible and transparent. This method was prepared on 5 September 2026 for organisations in England. Research was desk-only, with no account, demonstration, support request, import, send or accessibility test.
No supplier funded the work, supplied access or has an affiliate relationship with the publisher. Products without enough public evidence were not evaluated, and customer review scores were excluded.
This is a review protocol, not a product review: it names no product and offers no comparison, and it produced neither a rating nor a buyer outcome.
What to take away
- A review protocol should start with the buyer's workflow, not a catalogue of attractive features.
- Evidence must be classified separately and never merged across different classes of proof.
- Mandatory gates on privacy, security, accessibility, contract and deliverability are non-compensating and cannot be traded off.
- No overall score should be calculated while any mandatory gate remains unknown.
- Every important record needs a recheck trigger rather than one annual review date.
Freeze the question
Create a review protocol before naming candidates. Record the message purpose, sender, recipient and operator populations, acquisition route, peak workload, data classes, integrations, approval path and failure impact. Then list the options, including the existing process and a no-send alternative.
Freeze the review question
- Message purpose, sender, recipient, operator
- Acquisition route and peak workload
- Data classes and integrations
- Approval path and failure impact
- Options plus no-send alternative
- Exact identityedition, region, date
Give every candidate an exact identity: product, edition, region, account type, proposed counterparty and quotation date. A feature demonstrated on another tier is not evidence for the edition under review.
Once the gates are defined, list the product features the workflow requires: automation depth, segmentation logic, A/B testing capability, template-editor accessibility, ecommerce integrations and reporting granularity. These become preference criteria scored only after every mandatory gate shows a pass.
Use an evidence ladder
For each claim, preserve the URL or document, publisher, page date, access date, applicable edition and an excerpt or screenshot. Classify it as signed contract, independently scoped assurance, first-party technical record, supplier demonstration, controlled buyer test, third-party account or unsupported statement. Do not merge these classes.
Evidence ladder classes
- Signed contract
- Independently scoped assurance
- First-party technical record
- Supplier demonstration
- Controlled buyer test
- Third-party account
- Unsupported statement
A worked evidence-index row (illustrative, no real supplier named) records one claim in one class:
| Claim | Class | Source | Page date | Access date | Edition | Evidence recorded |
|---|---|---|---|---|---|---|
| Suppression list applies before every send. | First-party technical record | Supplier help centre, suppression article | Not stated | 5 September 2026 | Edition under review, UK region | Screenshot of setting and audit-log entry |
The NCSC advises buyers to choose a cloud-assessment depth according to the sensitivity of the data and the impact of failure in its cloud-provider guidance. This supports a risk-based evidence plan, not a security verdict on a named platform.
Apply gates before preferences
Define pass evidence for privacy and PECR, security, accessibility, advertising claims, contract, finance, operational recovery and deliverability. Mark each pass, fail, unknown or not applicable. These are non-compensating gates: an accessible editor cannot cancel an unacceptable contract, and strong authentication cannot create permission to send marketing. The deliverability gate requires evidence of SPF, DKIM and DMARC configuration on the sending domain, BIMI readiness where applicable, a custom sending domain rather than a shared default, and clarity on whether the plan uses a dedicated or shared IP pool.
An illustrative gate table keeps the status and owner visible:
| Gate | Pass evidence required | Status | Owner | Recheck trigger |
|---|---|---|---|---|
| Privacy and PECR | Lawful basis, consent records, processor terms | Pass | Data protection lead | Sub-processor change |
| Security | Current assurance scope, encryption, incident terms | Unknown | Security lead | Assurance expiry |
| Accessibility | WCAG 2.2 AA evidence for editor and templates | Fail | Accessibility lead | Edition change |
Advertising claims: pass evidence is a dated substantiation note per CAP guidance, showing that every objective claim in a template or campaign is backed by a first-party technical record or signed contract term. Finance: pass evidence is a quotation with an expiry date, itemised per edition and region, plus a written statement of any committed volume or overage charge. Operational recovery: pass evidence is a documented rollback and data-export procedure, tested within the bounded exercise below, with named owners and a maximum tolerable downtime.
Non-compensating gate decision
Does the gate pass?
proceed to next gate
fail, stop candidate
The ICO's controller and processor contract guidance lists matters including instructions, confidentiality, security, sub-processors, assistance, return or deletion and audits. It is under review after the Data (Use and Access) Act, so the current legal position and the actual roles need qualified checking.
Run bounded tests
Only candidates that pass documentary prequalification should enter a synthetic-data exercise. Write expected states for import, deduplication, suppression, role approval, tracking choice, accessible production, logging, export, deletion and rollback. Capture the configuration, time, operator and evidence for every observation.
A successful exercise is not a performance result. It covers one version and test path. Live deliverability, support quality and recipient response remain untested unless a separately approved study measures them.
Test deliverability before scoring
Seed-list testing should cover inbox placement, spam-folder rate and bounce behaviour across major UK mailbox providers. Monitor the sending domain against Spamhaus blocklists, review domain reputation in Google Postmaster Tools and check Microsoft SNDS for sender-level signals. Where the platform offers a dedicated IP, verify the warming schedule and volume-ramp limits before the first live send; on a shared pool, confirm the provider's abuse-monitoring and reputation-recovery process. Record each check with the date, domain and IP address tested. A pass here does not substitute for a pass on the privacy or contract gates.
Record the decision honestly
Do not calculate an overall score while a mandatory gate is unknown. If weighting is later used for preferences, disclose the scale, weights, missing-data rule and sensitivity. CAP's substantiation note is a reminder that objective promotional claims need documentary evidence. A review should therefore show why words such as "best" were withheld.
The output is an evidence index, exceptions log and signed decision with an expiry date. Reopen it when a plan, contract, sub-processor, integration, law, population or workflow changes. That is more useful than a permanent star rating.
For email software, recheck when a sub-processor changes, when the edition or region changes, or when sending-domain authentication changes.
Contract terms expire when a replacement agreement is offered; test evidence expires after a material configuration change. The owner must refresh the item, narrow the claim or return the gate to unknown. This stops an old screenshot quietly supporting a new buying decision.
Before you act
- Record the message purpose, sender, recipient and operator populations.
- Give every candidate an exact product, edition, region and quotation date.
- Preserve the URL, publisher, page date and access date for each claim.
- Mark each gate pass, fail, unknown or not applicable.
- Verify SPF, DKIM, DMARC and BIMI configuration on the sending domain.
- Check inbox placement via seed lists; monitor Spamhaus, Google Postmaster Tools and Microsoft SNDS.
- Confirm IP warming schedule or shared-pool reputation controls before the first live send.
- Run bounded tests only after documentary prequalification passes.
- Assign a recheck trigger and owner to every important record.
Common questions
What should be recorded before naming any candidate products?
The protocol should record the message purpose, sender, recipient and operator populations. It should also cover the acquisition route, peak workload, data classes and integrations. The approval path and failure impact belong there too. It should then list the options, including the existing process and a no-send alternative.
Why are the gates non-compensating?
The gates are non-compensating, so a strength in one area cannot offset a failure in another. Each gate must be marked pass, fail, unknown or not applicable on its own.
What does a successful bounded test actually prove?
A successful exercise is not a performance result. It covers one version and one test path only. Live deliverability, support quality and recipient response remain untested unless a separately approved study measures them. Configuration, time, operator and evidence should be captured for every observation.
