A customer pays. Their bank approves it, the payment provider approves it, and the debit alert lands on their phone. Then your app does nothing, because the one request that was supposed to tell it about the payment never got processed.
From the customer's side this is the worst bug you can ship. They don't file a bug report. They email support in capitals, open a dispute with their bank, or quietly decide not to buy from you again. Each of those can cost you more than the original order.
I've built payment flows on three products: a fashion marketplace at Asoebi Africa and a business school's online campus, both on Paystack, and HOFWAMS, an event platform on Stripe Connect. Different stacks, different teams, and the same lesson every time. The webhook is the most important request your backend receives, and most of us write it like a contact form.
The code below is simplified from those projects: names changed, error handling trimmed, the shape kept. Where the projects differ, I'll say which does what, including the parts I'd still improve.
What a webhook actually promises
Read a provider's docs closely and the promise is smaller than you'd think. Stripe is unusually direct about it. Your endpoint "might occasionally receive the same event more than once", and Stripe "doesn't guarantee the delivery of events in the order" they were created. If your endpoint doesn't answer with a 2xx in time, Stripe treats the delivery as failed and keeps retrying, in live mode for up to three days.
So a webhook can let you down in three ways, and each one has a business cost before it has a technical one.
What happens | Why it happens | What it costs | The fix |
|---|---|---|---|
It arrives twice | Your handler was slow or errored, so the provider retried | Two enrollments, two shipments, two receipts | Decide what "once" means, and let the database enforce it |
It arrives late or out of order | Retries back off, and events race each other | A status that moves backwards, a refund before its payment | Guard each transition, and check the provider's real state before moving money |
It never arrives | Your server was down past the retry window, or a deploy broke the route | The customer paid and got nothing | Ask the provider on a timer |
The rest of this post is how I deal with each of those, in the order I learned them.
The handler everyone writes first
When I joined Asoebi Africa, the work that follows a payment ran synchronously, inside the webhook request: mark the order paid, then do everything a paid order needs. It's the obvious way to write a handler, which is why it's so common. It also fails in a way you won't see in development.
webhook.ts (the naive version)
app.post("/webhooks/payments", async (req, res) => { const event = verifySignature(req); await enroll(event.customerId, event.courseId); await sendReceipt(event.customerId); await renderCertificate(event.customerId); res.sendStatus(200);});
O provedor informa um pagamento aprovado.
Ler os 6 passos
- O provedor informa um pagamento aprovado.
- O handler matricula o cliente na hora.
- Depois envia o recibo e gera um PDF, ainda dentro da requisição.
- O provedor para de esperar. Para ele, tempo esgotado é uma entrega com falha.
- Então entrega o mesmo evento de novo.
- O handler matricula o cliente pela segunda vez. Pago uma vez, entregue duas.
Walk through it slowly. The handler enrolls the customer, then keeps the provider waiting while it sends an email and renders a PDF. The provider doesn't know or care that you're busy. It sees no answer, counts the delivery as failed, and sends the same event again. Your handler, which has no memory of the first attempt, enrolls the customer a second time.
The opposite failure is quieter and worse. If the process crashes halfway through, nothing records that the event ever arrived. And if you answered 200 first and then crashed, the provider thinks you're done and will never send it again. Paid once, fulfilled never.
Both bugs come from the same mistake: the request is doing two jobs. It's acknowledging a message and it's running your business logic, and those have very different time budgets.
Rule 1: make the delivery durable before you answer
The fix is to make the webhook handler boring. It verifies the signature, puts the event somewhere that survives a crash, and answers. The slow work happens after, somewhere with its own retries. "Somewhere that survives a crash" has two common shapes, and I've used both.
Queue it. The online campus checks Paystack's signature, an HMAC of the raw body, and hands the whole verified payload to a BullMQ job before it answers. Nothing is written to the database first; the queue is the durable record.
@Post("webhooks/paystack")
async receive(@Req() req: RawBodyRequest, @Headers("x-paystack-signature") signature: string) {
this.paystack.verify(req.rawBody, signature); // HMAC-SHA512 of the raw body, or 401
await this.queue.add("process-paystack-webhook", { payload: req.body });
return { received: true }; // only after the job is safely queued
}If adding the job fails, the handler throws, Paystack gets an error instead of a 200, and Paystack tries again. There's no moment where the event is acknowledged but not recorded.
Store it. HOFWAMS saves every Stripe event to an events table before doing anything with it, keyed by Stripe's event id, with a processed flag, the last error and a retry count. It then processes the event inside the same request, because the work that follows is one short database transaction. Rule 4 is about why that's safe there and not everywhere.
At Asoebi, the change I made was the first move: the webhook stores the Paystack event, and the post-payment work moved onto a background queue, triggered through NestJS's event emitter. The request got short, and a retry stopped being something to worry about.
There's a gap worth naming if you store first and queue second, which is a very common design. The event is saved and committed, then the process dies before adding the job. The event is in your database with nothing scheduled to process it. Two well-known ways to close it: a sweep that finds saved events nobody has processed and queues them again, or a transactional outbox, where the event and the intent to process it are written in one transaction and a relay does the queueing.
And whichever shape you use, don't let the queue be your only guard against duplicates. BullMQ won't add a job whose id already exists, but only while that job is still kept; once completed jobs are cleaned up, the same id can be added again. The online campus doesn't set a job id at all, so a repeated delivery simply becomes a second job. That's fine, because the decision about "once" lives in the database.
Rule 2: know what you're deduplicating
Before you can process an event once, you need to know what "the same event" means, and providers don't agree.
Stripe gives every event an id, evt_..., and a redelivery carries the same one. It identifies the delivery, so it's the natural key for an events table.
Paystack's webhook payload has an event name and a data object, but no top-level event id. The online campus uses the transaction's data.id, falling back to the event name plus the transaction reference when there's no id. That identifies the payment the event is about, not one particular delivery of it.
A delivery id answers "have I seen this message?". A business key answers "have I already done this?". Two different events about the same payment, say a successful charge and a later refund, are different messages about one resource, and they must not be collapsed into one. So the event key decides what you record, and a separate rule decides what you act on: which is the next rule.
Rule 3: let the database decide what "once" means
A queue gives you at-least-once delivery, a polite way of saying "sometimes twice". Stripe and Paystack do the same. So the code that acts on a payment has to be safe to run twice on the same input.
On the online campus, fulfilment runs inside one transaction. It records the event in an append-only ledger, checks whether the payment already has an enrollment, and creates the enrollment and links it to the payment, all together:
await prisma.$transaction(async (tx) => {
const payment = await tx.payment.findUnique({ where: { paystackReference: reference } });
await ledger.append(tx, payment.id, { type: "WEBHOOK_RECEIVED", providerEventId });
if (payment.enrollmentId) return; // already fulfilled: a retry, or a second delivery
assertMatches(payment, providerResult); // amount and currency: see Rule 5
const enrollment = await tx.enrollment.create({ data: { userId: payment.userId, courseId: payment.courseId } });
await tx.payment.update({ where: { id: payment.id }, data: { enrollmentId: enrollment.id, status: "SUCCESS" } });
});Under that sits a unique constraint on the enrollment table: one enrollment per student per course. The enrollmentId check is what normally stops a second fulfilment; the constraint is what stops two concurrent ones that both passed the check. A conflict there means the student is already enrolled, through this payment or another one, and the code logs it as a payment that may need refunding rather than enrolling twice.
That distinction is the heart of it. A check in your code is a snapshot of the past. A constraint in the database is enforced at the moment of the write, which is the only moment that counts.
payment-fulfillment.service.ts (simplified)
await prisma.$transaction(async (tx) => { await ledger.append(tx, { providerEventId }); // a repeat is already there if (payment.enrollmentId) return; // already fulfilled const enrollment = await tx.enrollment.create({ data: { userId, courseId } }); await tx.payment.update({ where: { id: payment.id }, data: { enrollmentId: enrollment.id } });});// every 5 minutes: ask Paystack about anything still pending
O Paystack informa uma cobrança aprovada.
Ler os 5 passos
- O Paystack informa uma cobrança aprovada.
- O webhook confere a assinatura, enfileira um job e responde na hora.
- O worker registra o id do evento e matricula o aluno em uma única transação.
- O Paystack envia o mesmo evento de novo. Ele já está registrado, então nada acontece.
- Se um webhook nunca chega, o job de reconciliação encontra o pagamento mesmo assim.
Duplicates are only half the problem. On HOFWAMS, a credit purchase completes behind a PENDING guard: the update that marks it paid only matches while it's still pending, and the credits, the payment record and that status change are written in one commit. A late or repeated event finds nothing to update and changes nothing. State moves forward from a state you name, never from whatever the last event said.
Effectively once, not exactly once
It's worth being precise about what this buys. Nothing here makes a message arrive exactly once; that isn't on offer from any provider or queue. What the database gives you is an effectively-once business outcome: however many times the work runs, one payment produces one enrollment.
Side effects outside the database don't get that for free. Both projects send the confirmation email only after the transaction commits, so a retry that finds the work already done sends nothing. But the email itself has its own failure modes. If an email job fails after the provider accepted the message, retrying it can send a second copy. If the process dies between the commit and queueing the email, it's never sent. For anything that leaves your system, use the provider's own idempotency key when it has one, or an outbox, and decide which mistake you'd rather make.
Rule 4: decide who does the retrying
Once delivery is durable, there's a real design choice that most posts skip: when processing fails, do you let the provider retry, or do you retry yourself?
Question | Let the provider retry | Retry in your own queue |
|---|---|---|
How | Process in the request, return 500 on failure | Queue the verified event, return 200, process in a worker |
Who holds the event | The provider, plus your events table | Your queue, then your database |
Retry schedule | Theirs. Stripe: up to three days in live mode | Yours: backoff, limits, alerts |
Where you look when it breaks | The provider's dashboard and your events table | Your queue's failed jobs and your logs |
The risk | A slow handler times out and creates duplicates | A worker bug retries forever unless you cap it |
Where I used it | HOFWAMS credit purchases | Online campus enrollments, Asoebi post-payment work |
HOFWAMS takes the first route for credit purchases, and it only works because of how a redelivery is handled. When an event arrives, the handler looks it up by its Stripe id first:
const existing = await prisma.stripeWebhook.findUnique({ where: { stripeEventId: event.id } });
if (existing?.processed) return; // already handled: acknowledge, do nothing
const record = existing ?? (await prisma.stripeWebhook.create({ data: { stripeEventId: event.id, processed: false } }));
try {
await route(event);
await markProcessed(record);
} catch (err) {
await recordFailure(record, err); // last error, retry count
throw err; // a 500 for credit purchases, so Stripe delivers it again
}The important line is the first if. A naive version would insert the event, hit the unique constraint on a redelivery, and answer 200, acknowledging a retry without ever doing the work that failed. Here, an event that's already processed is acknowledged and skipped, and one that's saved but not processed, because it failed or never finished, is processed again. That's what lets Stripe's retries do their job.
And it's only safe because the work is one short transaction behind the PENDING guard, so a retry can never grant credits twice. Other Stripe events on HOFWAMS keep the older behaviour of answering 200 even when processing fails. The failure is recorded on the event's row, and an admin can replay it, but Stripe won't retry it on its own.
stripe-webhooks.service.ts (simplified)
const event = verifySignature(rawBody, signature);const saved = (await findEvent(event.id)) ?? (await saveEvent(event.id)); // unique idif (saved.processed) return; // retry of a handled eventtry { await route(event); await markProcessed(saved); } catch (err) { await recordFailure(saved, err); throw err; // 500: Stripe retries the credit purchase}
A Stripe informa uma compra de créditos concluída.
Ler os 5 passos
- A Stripe informa uma compra de créditos concluída.
- Primeiro ele é salvo, com o id do evento como chave.
- O processamento dá erro, então a API responde 500 de propósito.
- O Stripe tenta de novo. Mesmo id, então é processado exatamente uma vez.
- Uma tarefa de hora em hora reverifica qualquer compra ainda travada, consultando a Stripe sobre o estado real.
My rule of thumb: let the provider retry only when the work is small, fast and idempotent. Anything slow, or anything that talks to a third service, goes on your own queue, where you control the schedule and can see what's failing.
Rule 5: verify the payment, not just the signature
A valid signature proves the message came from your provider. It doesn't prove the payment matches the thing you're about to hand over. Before granting value, check that:
- the payment is the one you expected, found by its reference or checkout id, not by anything the payload merely claims;
- the amount is what you charged;
- the currency is what you charged in;
- the status is actually paid, not pending or failed;
- on a marketplace, it belongs to the right account.
The online campus checks the amount, the currency and the status before it enrolls anyone, and finds the payment by its unique Paystack reference. It compares against the amount the course cost, not the fee-inclusive total, so a customer who paid the card fee doesn't trip the check. HOFWAMS finds the purchase by its checkout session, requires Stripe's paid status, and checks the amount Stripe reports against the stored price before any credits move; a mismatch marks the purchase failed. Its credit purchases are platform revenue, so there's no connected account to check. Currency is the check I'd add next there.
Rule 6: assume some will never come
Everything so far handles webhooks that arrive. Some don't. Your server is down during a deploy, a DNS change goes wrong, someone edits the endpoint URL in the dashboard, or the provider has an incident of its own. From the customer's side it's the same story: they paid, and nothing happened.
The online campus runs a reconciliation job every five minutes. It finds payments still pending after 15 minutes, asks Paystack what really happened through its verify endpoint, and pushes confirmed ones through the same fulfilment function the webhook uses:
@Cron("*/5 * * * *")
async reconcile() {
const stale = await this.payments.findMany({
where: { status: "PENDING", createdAt: { lt: minutesAgo(15) } },
});
for (const payment of stale) {
const verified = await this.paystack.verifyTransaction(payment.paystackReference);
// The same fulfilment as the webhook: if the webhook turns up later, it finds the enrollment and stops.
await this.fulfilment.fulfil(verified);
}
}Reconciliation doesn't have its own way of enrolling anyone. It feeds the same idempotent path the webhook uses, so if the webhook does arrive late, the two can't both enroll the customer. You end up with two independent ways for a payment to be noticed and one way for it to be acted on.
HOFWAMS's safety net is narrower, and it's worth seeing why. An hourly job looks for credit-purchase events that were saved but never finished, fetches the checkout session from Stripe instead of trusting the stored copy, and settles it. That catches events that arrived and failed. It doesn't go looking for purchases whose webhook never arrived at all, which is the gap the campus job closes.
The webhook becomes the fast path, and the sweep is the safety net. Neither has to be perfect.
Rule 7: idempotency isn't the same as getting the state right
Everything so far stops the same thing happening twice. It doesn't, on its own, make your record of a payment match the provider's. Events arrive out of order, and some orders are dangerous.
Take a refund that arrives before the payment it refunds. If your refund handler ignores it because it has no paid payment to attach it to yet, the success event that arrives a minute later marks the payment paid, and fulfils it. You've delivered something you've already given the money back for.
A PENDING guard doesn't solve this. It's excellent at stopping a second grant, but it has no idea a refund exists. Two things do help:
- Record every financial event, and derive the state from the record. The online campus keeps an append-only ledger of payment events and computes the payment's status from it, which is the right foundation.
- Ask the provider before moving money. For a transition with money attached, like a refund or a reversal, fetching the payment's current state from the provider beats trusting the order your webhooks happened to arrive in.
This is the part of payment handling I'd still improve in both systems. Neither handles every out-of-order refund yet, and both are better than they were because the ledger and the guards are there to build on.
Rule 8: never drop a payment on the floor
The last case is the one teams forget. The money arrived, the event is valid, and you still can't deliver: the course was archived, the last ticket sold a second ago, or the customer is already enrolled through an earlier payment.
The tempting move is to log an error and return. Then the payment sits in your provider's dashboard with no order next to it until the customer complains. On the online campus, these cases are logged as errors that say a refund may be required, for someone to act on. That's the minimum. A log line still depends on someone reading it.
What I'd build instead is a state the payment moves into, so it shows up on a screen and in a count:
PENDING ──paid──▶ PAID ──fulfilled──────▶ FULFILLED
│
└──can't fulfil──▶ NEEDS_REFUND ──▶ REFUND_PENDING ──▶ REFUNDEDA refund is its own process with its own failures. It can be requested and never complete, or fail outright, so it needs the same treatment as the payment: a status, and reconciliation against the provider until it's confirmed.
And be careful what you tell the customer. "We couldn't deliver what you paid for, so we've started your refund" is true the moment you request it. "You've been refunded" is only true once the provider confirms it. Getting that sentence right turns "we lost your money" into a conversation a customer will forgive.
What it adds up to
None of this is clever. Put together, a payment flows like this:
- The webhook verifies the signature, makes the event durable in a queue or a table, and answers fast.
- You know what you're deduplicating: a delivery id for messages, a business key for actions.
- The work runs in a transaction, and the database's constraints decide what "once" means.
- A redelivery redoes failed work and skips finished work.
- The payment is checked against what you charged before anything is granted.
- A reconciliation job finds the payments the webhook never told you about.
- Money-moving transitions check the provider's real state, not the order events arrived in.
- A payment you can't fulfil becomes a refund someone sees through to the end, not an error log.
The webhook handler ends up as the shortest file in the payments module. And the aim of everything around it is simple to say: the customer who pays gets one fulfilment, whether the provider tells you once, three times, or never.
Further reading
- Stripe: receiving webhooks, handling duplicate events
- Paystack: webhooks, verifying a transaction
- BullMQ: job ids, retrying failed jobs
- PostgreSQL: unique constraints
- The transactional outbox pattern