n8n: Idempotency and Webhook Deduplication for Duplicate-Free Workflows

An operation is idempotent when running it once or ten times produces the same result. In n8n workflows it is the property that prevents invoices issued twice, duplicate CRM contacts or emails sent twice when a webhook is delivered more than once. You get it with three elements: a unique key for each event, a check before acting and upsert-style writes instead of plain inserts.

Why Duplicates Happen

Most systems that send webhooks guarantee "at least once" delivery, not "exactly once". If your endpoint responds late or with an error, the sender retries. A network timeout, an n8n restart or a manually retried execution can also run the same event twice. Duplicates are not an anomaly to eliminate but a condition to handle.

The same applies to scheduled runs: if a workflow that syncs orders every hour overlaps the previous run, it may process the same records twice.

Choosing an Idempotency Key

The key uniquely identifies the event. Many providers include an identifier in the payload or a header, such as the event or order id. If there is none, build a stable key by combining the fields that define the operation, such as event type, object id and modification date.

The key must stay the same across retries of the same event and differ for different events. A key based on arrival time does not work: it would change with every delivery.

Check Before Acting

The simplest pattern: at the start of the workflow, look up the key in a store, such as a database table, Redis or n8n's data table. If it exists, the workflow ends without doing anything; if not, it records it and continues. In n8n the Remove Duplicates node can filter duplicates across executions, provided you choose the comparison field carefully.

Checking and recording the key must be a single atomic operation. If two deliveries arrive at the same instant, both could pass the check before either writes. A unique constraint in the database, which makes the second insert fail, removes this race condition.

Upsert Instead of Insert

When the destination allows it, write with an upsert: update the record if it exists, create it otherwise, using the external identifier as the key. Many CRMs and databases offer this operation. Repeating an upsert with the same data leaves the state unchanged, so the workflow becomes idempotent without a separate key store.

For calls to third-party APIs that create resources, such as a payment, use the idempotency header if the API supports it: the vendor recognizes the key and returns the result of the first request instead of running a second.

Respond Immediately and Work in the Background

Many senders retry if they get no response within a few seconds. A long workflow that responds only at the end invites retries. Return a 200 straight away from the Webhook node ("respond immediately" mode), queue the work and process it in a second step, with the idempotency check at the entrance.

Handling Retries and Errors

If a step fails midway, re-running must not repeat effects already produced. With idempotent operations a retry is safe; without them you must record the progress of each step. A dedicated error workflow notifies the team and keeps the payload for controlled reprocessing.

• Assume every webhook can arrive more than once.
• Use the provider's event id or build a stable key from the operation's fields.
• Make key check and recording atomic, with a unique constraint.
• Prefer upsert over insert when the destination allows it.
• Use the idempotency header of APIs that create resources, such as payments.
• Respond to the webhook immediately and process in the background, with a dedicated error workflow.

Testing for Duplicates

A simple test: send the same payload two or three times, even in parallel, and check that the destination shows a single record or a single effect. Add the case where the workflow is interrupted midway and re-run. These tests cost a few minutes and prevent incidents that cost far more in production.

Key takeaway: webhooks repeat, so workflows must tolerate it: an idempotency key, an atomic check, upserts and an immediate response with background processing make n8n automations reliable and duplicate-free.