Halfway through a campaign, the email provider stops answering. It doesn't refuse requests, it just goes quiet. Every send waits out its full timeout, gets no status code, and is handed back to the queue as a temporary failure. The queue does what queues do and tries again.
Some of those emails had already gone out. The provider accepted them; only its reply got lost. So every retry is a chance to send the same newsletter to the same person a second time, and for a product whose customers send to their own users, a duplicate isn't a cosmetic bug. It's their brand in someone's inbox twice, and a reason to unsubscribe.
OnchainSuite is a retention and CRM platform for Web3 teams: campaigns, automations and on-chain triggers that all end in an email. This post is how the team made each marketing email go out at most once, on top of a queue that is built to run jobs more than once.
It's worth being exact about that promise, because the rest of the post depends on it. The design guarantees at most one automatic submission to the email provider, per recipient, per send, as long as the database holding the claims is reachable. It doesn't guarantee one delivered email: once the provider accepts a message, delivery is the provider's business. And it doesn't guarantee every email goes out; the design chooses, on purpose, to risk a missed email rather than a duplicate.
The code below is simplified from the OnchainSuite backend (NestJS, BullMQ on Redis, Prisma on Postgres): names shortened, logging and most error handling trimmed, the shape kept.
Why queues send things twice
A job queue like BullMQ promises at-least-once delivery. If a job fails, or the worker running it dies, the job runs again. That's the right promise for most work: a job that might run twice is far better than a job that might silently never run.
It's only safe when running a job twice has the same effect as running it once. Writing a row with a fixed id is like that. Sending an email isn't: the provider has no idea the second request is a retry, and delivers it.
So the email path needs something the queue can't give it: an at-most-once side effect inside an at-least-once job. The queue keeps retrying; the send underneath refuses to happen twice.
A timeout tells you nothing
The hard part is the timeout. When a call to the provider succeeds, you know the email went. When it fails with a clear error, a rejected address or a bad request, you know it didn't. When it times out, you know neither.
The request may never have reached the provider. It may have reached it and failed. Or the provider may have accepted it, queued it for delivery, and lost the connection before telling you. From the sender's side, all three look the same. It's a small, everyday version of the Two Generals' Problem: no amount of waiting turns a missing reply into a yes or a no.
Most code treats a timeout as a failure, because that's what a thrown error usually means. For email, that's the assumption that sends duplicates.
Why "check, then send" doesn't work
The first idea is to look before sending: query whether this recipient already got this campaign, and skip if so.
That has the same gap as any check that happens before the write it guards. Two workers can both look, both see nothing, and both send. And it doesn't help with a timeout at all: if the first attempt's send is in doubt, there's nothing recorded to find.
Approach | What happens on a retry | What goes wrong |
|---|---|---|
Query "already sent?" first | Skips if a sent row exists | Two workers can both see nothing; a timed-out send left no row |
Record the send after the provider call | Skips if the record exists | A crash or timeout between the call and the record leaves no trace, so the retry sends again |
Claim before the call, release on any error | Skips while a claim exists | A timeout is treated as a failure, the claim is released, and the retry sends again |
Claim before the call, keep it on a timeout | Skips if the send might have happened | A send that failed by timeout, or a worker that crashed right after claiming, is never retried. This is what OnchainSuite does |
The last row gives something up, on purpose, and the rest of the post is why that's the right trade for marketing email.
Claim the send before calling the provider
Before a worker calls the provider, it writes a send claim: a row in Postgres that says "this campaign run is sending to this recipient". The table has a unique index on the organisation, the campaign run and the recipient, so a second claim for the same send can't be inserted. The database refuses it.
async claim(key: SendClaimKey): Promise<boolean> {
try {
await prisma.sendClaim.create({ data: { ...key, state: "claimed" } });
return true;
} catch (err) {
if (isUniqueViolation(err)) return false; // someone already claimed this send
return true; // the claim store is down: send anyway (see below)
}
}isUniqueViolation checks for Prisma's P2002 error, which is how Prisma reports a unique constraint failure. A worker that loses the claim doesn't send, and it records that it skipped, so a dropped duplicate shows up in the data instead of disappearing.
Order is the whole point. The code's own comment says writing the row after the provider call "would reproduce the exact bug": a crash or a timeout between the call and the write would leave nothing behind, and the retry would send again. Written first, the claim exists before anything can go wrong.
Writing it first has a cost, and it isn't only timeouts. If a worker takes the claim and then crashes before it calls the provider, the claim stays behind in its claimed state. The retry finds it, assumes the email may have gone, and skips. That email is never sent. Nothing currently sweeps up claims left in that state; a job that marked old, unsettled claims as ambiguous, so a person sees them, would close the gap.
Why not let the provider deduplicate instead? Some can: Resend accepts an idempotency key and remembers it for 24 hours. But OnchainSuite sends marketing mail through SendGrid and Amazon SES, and neither SendGrid's send API nor SES's SendEmail documents one. Where a provider offers a key, use it as well. A claim in your own database works the same whichever provider you're on.
The recipient isn't stored in plain text either. The claim holds an HMAC of the address, keyed per organisation, so the table can enforce "once per recipient" without becoming another list of everyone's email addresses.
Sent, released or held
After the call, the claim is settled one of three ways, and this is where a timeout stops being treated as a failure.
const claimed = await claims.claim(key);
if (!claimed) return { skippedDuplicate: true };
try {
await provider.send(email);
await claims.settleSent(key);
} catch (err) {
if (isTimeout(err)) await claims.settleAmbiguous(key); // keep it: it may have gone
else await claims.release(key); // any other error: let the retry send
throw err; // the queue still sees the failure
}Outcome | Claim becomes | What a retry does |
|---|---|---|
The provider accepted it | | Skips |
Any other error | deleted | Claims again and sends |
The call timed out | | Skips |
send-email.ts (simplified)
if (!(await claims.claim(key))) return skip(); // unique indextry { await provider.send(email); await claims.settleSent(key);} catch (err) { if (isTimeout(err)) await claims.settleAmbiguous(key); // keep it else await claims.release(key); // safe to try again throw err;}
Un worker toma un trabajo de envío de la cola.
Leer los 5 pasos
- Un worker toma un trabajo de envío de la cola.
- Antes de llamar al proveedor, escribe una reserva para este destinatario.
- El proveedor acepta el correo, pero su respuesta nunca llega. La llamada agota el tiempo.
- Agotar el tiempo no es fallar. Nadie sabe si se envió, así que la reserva se conserva.
- La cola reintenta el trabajo. Su reserva choca con el índice único, así que el reintento se omite.
Releasing is careful too. It only deletes a claim that's still in the claimed state, so a late error from one attempt can't reopen a claim another attempt already settled as sent.
This is a deliberate choice of failure. Holding the claim on a timeout means some emails that didn't actually go out are never retried. For marketing mail the team decided that's the right side to fail on: a missed newsletter is a small loss nobody notices, a duplicate is a visible mistake that costs the customer trust. The ambiguous claims are kept, so someone can look at them later; there isn't a screen for that yet, only a query.
That middle row is broader than it should be, and it's worth seeing why. The line that matters runs between a definite outcome and an ambiguous one, and timeouts are only one kind of ambiguous. A documented validation rejection is definite: the provider refused the message, so sending again is safe. But a 5xx from the provider, or a connection that drops mid-request, can be just as uncertain as a timeout, because the provider may have accepted the message before the error. Today OnchainSuite holds the claim only for errors it recognises as timeouts, and releases it for everything else. Classifying errors by each provider's documented error types, rather than by status family or message text, is the next step, and it's on the list at the end.
Some errors aren't worth retrying at all. A 4xx from the provider, other than a timeout (408) or a rate limit (429), means the request itself is wrong, a malformed address for example, and sending it again won't change that. Those jobs are marked unrecoverable so BullMQ doesn't waste attempts on them.
Holding a claim leaves a question: who decides what happened? An ambiguous claim records the error, not the provider's message id; that lives on the delivery's own record. Today a person has to look and decide, and there's no screen for it yet. The safe way to resolve one is to check the provider's logs for that delivery, and if it really didn't go, send it again as a deliberate new send, not by deleting the claim and hoping.
Keep the timeout shorter than the lock
There's a second way a job runs twice, and it doesn't need any error at all.
While a BullMQ worker runs a job, it holds a lock on it and renews that lock periodically to say it's still working. A job doesn't lose its lock just for running long. It loses it when the renewal fails: the worker crashed, its event loop was blocked, or it couldn't reach Redis. BullMQ then treats the job as stalled and moves it back so another worker can run it. If the first worker was only stuck, not dead, two workers can now be on the same email.
The team's rule is to keep the provider call's timeout below the lock's duration, 30 seconds by BullMQ's default, so a hung provider produces a timeout, which the claim handles, well before anything else goes wrong. That's a safety margin, not a guarantee: the provider call is only part of a job's time, and a stall can still happen for other reasons.
The real protection is the claim. If BullMQ does hand a stalled job to a second worker, that worker tries to take the same claim, the unique index refuses it, and it skips. The lock keeps duplicates rare; the database keeps them out.
One honest caveat: the timeout rule lives in configuration and a code comment, not a check, and the providers' defaults don't all follow it. It's on the list below.
When Redis can't save the retry
At-least-once has a quiet assumption underneath it: that the queue can write down what it's doing. BullMQ keeps every job, every retry and every delay in Redis. If Redis can't take a write, a job that fails can't be scheduled to run again.
That happened once, documented in the team's own incident notes. Each queued email carried its full rendered HTML, tens to over a hundred kilobytes per job, and a large send filled Redis to its memory limit. Redis is configured to refuse writes when full rather than evict data, which is the right setting for a queue: evicting means deleting jobs. But refusing writes meant failed sends couldn't be put back on the queue. As the note puts it, the retry mechanism didn't fail to run, it failed to persist. Some recipients were never retried, and nothing alerted anyone.
The fix that shipped compresses email bodies above 2 KB before they go into the job. The better fix, which the note names and which isn't done yet, is to put only an identifier in the job and render the email in the worker. A queue should carry pointers, not payloads.
Claim the right thing
A claim is only as good as its key. "Once per campaign run, per recipient" is right for a campaign. For automations it was briefly wrong in an instructive way.
Automations send email as one step in a flow: someone signs up, waits a day, gets a welcome email. The first version keyed those claims on the automation. That meant "once per automation, per recipient, ever". When the same person went through the flow a second time, legitimately, their email was silently skipped as a duplicate, and the flow still showed as completed. The key now uses the most specific id available: the campaign, then the person's run through the flow, then the automation as a last resort.
The automation runtime has its own claim one level up. Before a step with a side effect runs, it inserts a row unique on the run and the step:
Claim finds | Meaning | What happens |
|---|---|---|
No row | Nobody has run this step | Run it |
A completed row | It already ran | Skip |
A running row, less than 15 min old | Another worker is on it | Skip |
A running row, older than 15 min | That worker probably died | Take it over |
If the step fails, its claim is deleted, so the retry runs it again. A completed step means "queued for sending", not "delivered", which is why the email layer keeps its own claim underneath.
The last row of that table is the weak spot, and it's a classic one. "Older than 15 minutes" can mean the worker died, or that it's still running something slow. The takeover is a read followed by an update, not one conditional write, and the original worker has no way of knowing it's been replaced: there's no lease it renews and no fencing token that would stop it finishing. So a step that runs past 15 minutes can end up running twice. For email steps, the send claim underneath still stops a second email. For any other side effect, that needs a conditional takeover and a lease the worker renews while it runs, and it's on the list.
When the safety net itself breaks
The claim lives in Postgres. If Postgres can't be reached, the code sends anyway, unclaimed. That's failing open, and it's a choice, not an accident: refusing to send whenever the bookkeeping is down would turn a degraded dependency into a delivery outage. A short window of possible duplicates is better than no email at all.
So the guarantee has two modes, and it's worth naming them. In normal operation, preventing duplicates wins. While the claim store is unavailable, delivery wins, and duplicates are accepted. There's a subtle case between the two: a connection that drops during the insert leaves the code unsure whether the claim was written. It treats that like any other database error and sends.
Transactional mail, password resets and one-time codes, makes the opposite trade. It isn't claimed at all, and it fails over to a second provider on a timeout. Failover resends the very same message, so a duplicate carries the same code, and the user can use either. The cost being weighed is the missing email: a user who never gets their login code can't log in.
Duplicates aren't free everywhere, though. If a user asks for a code twice and each request mints a new one, the first email becomes useless the moment the second is sent. Reusing the active code until it expires, rather than issuing a new one per request, keeps repeated sends harmless.
Guarantee | Why | |
|---|---|---|
Marketing and automations | At most once per recipient | A duplicate costs trust; a miss costs little |
Transactional (codes, resets) | At least once | A missing code locks someone out; a duplicate of the same message is harmless |
How it's tested
The claim logic is covered by focused unit tests: the first claim wins, the second is refused, a timeout holds the claim instead of releasing it, a release can't reopen a settled claim, and a broken claim store sends anyway. The automation tests run two workers against the same step at once and assert the email is dispatched exactly once:
it("dispatches once when two workers run the same step", async () => {
await Promise.all([run(), run()]); // two workers, the same send step, at once
expect(sendSpy).toHaveBeenCalledTimes(1);
});These run against an in-memory store that raises the same unique-violation error Postgres would, not against a real database. They prove the logic around the constraint; the guarantee itself comes from the unique index in Postgres. The next test worth adding fires concurrent claims at a real Postgres and checks exactly one succeeds, because that's the line everything else rests on.
What's still open
- Only recognised timeouts hold a claim. A 5xx or a dropped connection releases it, though either can be ambiguous. Each provider reports these in its own way, so the decision should use the provider's documented error types, not status families or message text.
- A claim left behind by a crash is never revisited. Old, unsettled claims should become ambiguous, so a person sees them.
- Ambiguous sends need a person, and there's no screen yet. Storing the provider's message id on the claim would make each one quick to check.
- The timeout rule lives in configuration, and not every provider's default follows it. A startup check would enforce it.
- The automation takeover isn't fenced. A conditional takeover and a renewed lease would stop a slow step running twice.
- Jobs still carry payloads. Compression bought headroom; sending identifiers instead is the lasting fix.
The rule
A queue that retries is a promise that work will happen at least once. Anything that mustn't happen twice needs a second promise underneath it, recorded before the side effect, enforced by the database, and honest about the cases where nobody knows what happened.
When you can't tell whether something happened, decide in advance which mistake you'd rather make, and write that decision into the code.
Further reading
- BullMQ: retrying failed jobs, stalled jobs, how workers renew locks, job ids
- Email providers on idempotency: Resend's idempotency keys, SendGrid mail send, SES
SendEmail - PostgreSQL: unique constraints
- Prisma: error reference, P2002
- Redis: key eviction and
noeviction - The Two Generals' Problem