Never Send It Twice — Idempotency and Operations for Notifications

No duplicate emails, and missed sends get noticed: reliability design for the notification layer

idempotencyduplicate send preventioncronhealth monitoringerror notifications
7 min read

Introduction

If two booking confirmation emails arrive, the guest starts worrying that they've been double-booked. If none arrives, they have no idea whether the booking worked at all.

This article covers the mechanics of delivering notifications neither too many nor too few. It's unglamorous work, but when it breaks, trust disappears in an instant.

Preventing Duplicates Without a Send Log

Choosing Not to Record Sent Emails

The common approach is to build a table of sent emails and check it before sending. This time, we deliberately didn't build that.

Instead, we made sure the processing that leads up to the email only ever runs once. If the process runs once, only one email comes out.

Approaches to preventing duplicates
BEFORE
Check against a send log

If writing the log fails, you duplicate. And the record of recipients becomes another place personal data lives

AFTER
Limit the process itself to one run

The email is a by-product of the process. If the process runs once, only one goes out — no check needed

As a side benefit, that's one fewer place storing email addresses. The less personal data you hold, the better.

Conditional Updates Make It First-Come, First-Served

The booking confirmation process can be triggered from two directions: a notification from the payment service, and the guest opening the completion page.

So at the entrance to the process, we do a conditional database update that says "set this to confirmed only if it isn't confirmed yet". Only the side that succeeded in updating continues on and sends the email. The side that didn't simply stops.

Don't Repeat the Same Warning

When a sync with the reservation management system fails, we send the operator a warning email. But if the hourly scheduled job keeps detecting the same failure, that same warning arrives every single hour.

So we put a condition on the update that accompanies the warning too: "set the error state only if it isn't already in an error state". If it's already flagged, no update happens and no email goes out.

We use the same approach for detecting cancellations on already-paid bookings, marking them as handled to prevent repeat notifications.

Pitfalls of Scheduled Jobs

Searching for "Today's Batch" Misses Things

When looking for bookings due to be charged, searching for "bookings whose charge date equals today" is dangerous. If the scheduled job fails to run for a day for any reason, that day's batch is stranded forever.

So we search for "bookings whose charge date is today or earlier and that haven't been charged yet" instead. Even if it stops for a day, everything gets picked up together the next day.

Design principles for scheduled jobs
Search by range

Not "today" but "today or earlier". Skip a run and the next one catches up

Cap the number of items

Limit how many get processed per run to avoid timeouts

Always return success

Don't bail out on an error; log the individual failure and move on to the next

Guarding the Entrance

Scheduled jobs are triggered by hitting a URL, which means anyone who learns the URL could run them. So we put a shared-secret check in front.

The crucial part is that if the secret isn't configured, execution is refused. A design where "forgetting the configuration means anyone can run it" is an accident waiting to happen. Tip it toward not running at all when the configuration is missing.

Reporting That "Nothing Happened"

The Email We Send Even at Zero

The daily report for automatic charges is always sent, even when there are zero items. It looks like a waste, but there's a clear purpose behind it.

If an email saying "0 items today" arrives every day, you notice something is off the moment it stops. The email itself doubles as a health check.

And the Emails We Don't Send

Meanwhile, the job that releases temporary holds every ten minutes sends no email at all. If "released 0 items" showed up 144 times a day, the genuinely important notifications would be buried.

The hourly sync check likewise only sends when it detects a change.

The more often something runs, the more selective the notifications. Getting that balance right is what keeps notification fatigue away.

Don't Let a Failed Email Stop the Business

A Send Failure Doesn't Roll Back the Business Logic

What should happen if the payment succeeded but the email send threw an error? You certainly shouldn't reverse the payment.

So every email send runs in a "keep going even if this fails" mode. When several emails go out together, one failure doesn't stop the others.

Keep the money-moving work separate from the notification work. Email matters, but it isn't the substance of the business.

No Sending in Development

In environments without email service configuration, we don't attempt to send — we just log it and move on. Real emails never fly out during development, and no feature breaks because the configuration is missing.

What Goes in the Logs

When an email fails to send, all we log is the booking number and the type of error. Not the recipient's address, not the body of the email, not the response from the delivery service.

Logs exist to help you investigate, but they can just as easily become a leak path. We pick what to keep with "the minimum needed for investigation" in mind.

Wrapping Up

Three things anchored the reliability design for the notification layer.

  1. Prevent duplicates by limiting the process to one run — don't rely on a send log; use conditional updates for first-come, first-served
  2. Report that "nothing happened" — daily jobs send even at zero, so you notice when they go silent
  3. Don't let a failed email stop the business — keep payment and booking processing separate from notification processing