How to Design an Idempotent Webhook Ingestion Layer for n8n, Microsoft

Idempotent webhook processing starts by assuming a delivery can arrive more than once. A dependable ingestion layer verifies the sender, records the event durably with a unique key, acknowledges the provider promptly, and moves slower business work to an asynchronous worker. This design prevents a delivery retry from becoming a second email, ticket, calendar update, or database change.

This matters when Microsoft 365, Google Workspace, or another SaaS provider triggers an internal automation. A provider may retry because it did not receive a successful HTTP response, even when your endpoint began work. A process can also fail after accepting an event but before it hands that work to a workflow engine. The solution is not to expect exactly-once delivery from the provider. It is to make repeated delivery safe within your own system.

Design for at-least-once delivery

The useful default is at-least-once delivery: an event may be delayed, repeated, or retried. The idempotent consumer pattern addresses this by keeping durable information about messages already consumed. When the same logical event arrives again, the consumer recognizes it and turns the duplicate into a no-op instead of repeating the business action. Microservices.io describes this as the Idempotent Consumer pattern.

A durable database constraint is the core protection. An in-memory set disappears on restart. A workflow execution history can be helpful during investigation, but it is not necessarily an atomic gate before a new execution starts. A short-lived cache key may reduce load, but it does not by itself provide durable duplicate protection. Put the authoritative deduplication decision in storage that can enforce uniqueness.

For Microsoft Graph change notifications, subscription validation requires the endpoint to return the supplied validation token within three seconds. Microsoft Graph also retries notifications that are not acknowledged, using exponential backoff for up to four hours, and supports a caller-provided clientState value for validating notifications. Microsoft Graph webhook documentation These requirements favor a small ingress endpoint that does only the work required to validate and durably accept an event.

Google Workspace Events uses Google Cloud Pub/Sub for event delivery and represents event data using CloudEvents. Its documentation also describes ordering keys for ordered Pub/Sub delivery. Google Workspace Events API overview Ordered delivery can be useful for a particular stream, but it does not remove the need for duplicate protection or durable processing state.

Use a durable receipt boundary

The simplest reliable boundary separates provider-facing HTTP handling from downstream workflow execution. The HTTP endpoint is responsible for accepting an authenticated event and ensuring it has a durable recovery path. Workers are responsible for the slower work: enrichment, calls to internal systems, n8n workflow execution, and externally visible actions.

Stage	Primary responsibility	Why it matters
Verify	Validate the provider-specific signature, token, or validation mechanism.	Stops unauthenticated requests from creating work.
Receive	Create a durable event receipt behind a unique key.	Turns repeated delivery into a safe duplicate outcome.
Commit	Create an outbox record in the same transaction as the receipt.	Ensures accepted work can be dispatched after a crash.
Dispatch	Move committed work to a queue or worker.	Keeps provider response time independent of business processing.
Execute	Perform downstream actions with a stable idempotency key.	Protects external side effects when workers retry.

This is an architectural recommendation based on the cited delivery and idempotent-consumer patterns. The key operational rule is straightforward: acknowledge an event only after it has been stored in a way that survives process failure and can be retried safely.

Verify the original request correctly

Verification must follow the provider’s documented rules, so isolate it in an adapter for each source. Preserve the raw request body when a signature scheme depends on raw payload bytes. Parsing JSON and serializing it again can change the bytes used by signature verification.

The Standard Webhooks specification defines an HMAC-SHA256 signature construction that binds a message ID, timestamp, and payload. It also emphasizes validating timestamp freshness to reduce replay risk. Standard Webhooks specification The exact protocol will vary by provider, but the principle remains useful: authenticate the delivery before allowing it to create queued work.

Stripe’s webhook guidance offers a public example of this approach. It recommends verifying signed events, responding successfully without delay, and expecting duplicate deliveries. It also recommends recording processed event IDs and using database uniqueness to prevent duplicate handling. Stripe webhook documentation Do not apply Stripe’s signature format to another provider; instead, apply the same design principle using that provider’s documented verification model.

Store a canonical event receipt

An event receipt is the durable record of what your system accepted. Useful fields include the provider, tenant or subscription context where applicable, provider event ID when available, event type, receive time, verification outcome, processing state, correlation ID, and failure details. Store a payload reference or an appropriately protected payload where that is required for later processing and diagnosis.

Use a stable event key. When the provider supplies an event identifier with the right semantics, a key such as provider + tenant + provider_event_id is a practical choice. If there is no suitable event ID, derive a deterministic key from immutable fields that identify the logical event. Avoid treating a payload hash as a universal duplicate key: identical payloads can represent separate logical events, while retries can differ in fields that are irrelevant to the event itself.

A business resource ID should not be the only key. A single mailbox message, calendar item, or Drive resource can produce multiple meaningful events. The duplicate key must identify a delivery or logical event, not simply the object involved.

Commit the receipt and outbox together

The critical write path should be small:

1. Verify the incoming request according to its provider contract.
2. Derive the canonical event key.
3. Insert an event receipt protected by a unique database index.
4. When that insert succeeds, create an outbox record in the same database transaction.
5. When the unique key already exists, record or observe the duplicate outcome and return a successful response without starting another action.
6. After the transaction commits, let a dispatcher send outstanding outbox records to a worker queue.

The outbox closes an important failure gap. A direct sequence of “save the receipt, then publish a queue message” can fail between those two actions. The receipt may say the event was accepted even though no worker will receive it. An outbox record committed with the receipt gives a dispatcher something durable to retry after recovery.

This does not make every component exactly once. A dispatcher or worker can still retry. Instead, it provides a durable chain of responsibility: the event was received, its work was recorded, and repeated attempts can be made without creating a second logical event.

Keep n8n behind the provider boundary

n8n can be a useful execution layer, especially when business teams need workflows that connect SaaS systems. Its Queue mode separates webhook handling on main instances from execution on workers, using Redis for queueing and PostgreSQL for persisted data. n8n Queue Mode documentation That separation supports concurrency management and worker isolation during webhook spikes.

For business-critical automation, n8n should receive work after a durable ingress service has accepted the provider event. A practical flow is: provider endpoint, ingress service, receipt database and outbox, dispatcher, queue, then n8n or a custom worker. Pass the canonical event key with the workflow input so every stage can correlate the same logical event.

n8n queueing improves throughput and decouples execution from receipt, but a queue alone does not make processing idempotent. A worker may receive work more than once. Keep the event receipt as the duplicate gate and apply another duplicate guard to externally visible actions where necessary.

For example, before a worker creates a CRM task, it can claim an action_ledger record keyed by event_key + action_name. A uniqueness conflict means the action has already been claimed. Where a downstream API accepts an idempotency key, use the same stable event-derived key. This adds protection at the point where a repeated call would otherwise create a repeated business result.

Model retries as durable states

Retries should be modeled as state transitions, not as an unbounded loop. A receipt or worker job can move through states such as received, queued, processing, succeeded, retryable_failure, dead_lettered, and rejected. Store an attempt count, the last error, and the next eligible retry time.

Classify failure causes deliberately. Authentication failures, malformed payloads, and unsupported event types are usually terminal. Temporary downstream failures may be retryable. A business-rule rejection normally will not change merely because the same request is tried again. Route work that exhausts its retry policy to a dead-letter or review state rather than retrying forever.

Use bounded exponential backoff with jitter for worker retries. The objective is to avoid concentrating recovery load while a dependency is unavailable. Provider-side retry behavior is separate from worker retry behavior, which is why a durable receipt should be acknowledged promptly once it commits.

Do not confuse ordering with correctness

Events may arrive late, be delivered again, or complete in parallel. Google Pub/Sub ordering keys can help preserve order for an ordered stream, but consumers still need to make state changes safely. Google Workspace Events API overview

When the underlying provider model offers a revision, timestamp, or other ordering signal, use that signal when deciding whether an event may update local state. A later-completing worker should not overwrite a more recent local result merely because it finished last. When a notification indicates that a resource changed but does not contain enough state to act safely, retrieve the authoritative resource according to the provider model before performing the business update.

Measure the entire event path

Operational visibility should answer four questions: did the provider deliver an event, did ingress accept it, was it deduplicated or dispatched, and did the resulting business action complete once? Use the receipt’s correlation ID in the outbox record, queue message, worker logs, n8n input, and action ledger.

Track accepted, rejected, duplicate, queued, succeeded, retryable-failure, and dead-lettered events by provider and event type. Also monitor receipt-to-dispatch lag, queue depth, worker age, verification failures, and subscription renewal failures. Together, these signals help distinguish a provider delivery issue from an ingress failure, queue bottleneck, or downstream dependency problem.

Implementation checklist

• Document validation, authentication, retry, and subscription requirements for every provider.
• Choose and document a stable unique event-key strategy.
• Preserve raw request bytes when the provider’s signature process requires them.
• Write the event receipt and outbox record in one transaction.
• Return success only after the durable receipt path commits.
• Use a retrying dispatcher and make workers safe for repeated delivery.
• Pass the canonical event key into n8n and downstream services.
• Add an action ledger or downstream idempotency key for consequential side effects.
• Define terminal, retryable, and dead-letter outcomes before production traffic begins.
• Test duplicate delivery, concurrent delivery, crash recovery after commit, queue unavailability, worker retries, and late events.

FAQ

Can webhook deduplication happen only in n8n?

It can be adequate for lower-impact workflows, but it couples provider acknowledgment and duplicate protection to workflow execution. For consequential actions, use a durable receipt boundary before n8n.

Does adding a queue make webhook processing idempotent?

No. A queue decouples receipt from execution, but workers can still receive repeated work. Idempotency requires a stable key and an atomic duplicate guard, including protection around downstream side effects.

Should an authenticated duplicate receive an error response?

Usually no. Once the duplicate has been recognized as an event already accepted durably, return success. Returning an error can encourage additional provider retries.

What is the smallest reliable database design?

Start with an event-receipts table protected by a unique event key and containing processing state. Add an outbox table written in the same transaction for stronger recovery behavior. Add an action ledger when repeated downstream calls could create irreversible or externally visible effects.

Sources

• Microsoft Graph: Set up notifications for changes in resource data
• Google Workspace Events API overview
• n8n: Scaling with Queue Mode
• Stripe: Receive webhook events and prevent duplicates
• Standard Webhooks specification
• Microservices.io: Idempotent Consumer pattern

Editorial note: AI assisted with research and drafting. Sources were selected for verification.