Why llm tools need distributed-systems design

Production llm tools are not simply model features. They are distributed workflows that can touch calendars, ticketing systems, databases, email, cloud infrastructure, and internal APIs. A model may produce a well-formed tool request, but the surrounding system still has to answer harder questions: Did the action complete after a timeout? Is it safe to retry? Who can approve a consequential change? Where is the evidence for what happened?

The practical shift is simple: treat model output as a proposed command, and treat tool execution as an unreliable external operation. That framing helps engineering teams move from an impressive prototype to an automation service that can survive partial failures, duplicate requests, and operational investigation.

What is tool calling in LLM systems?

Tool calling is a pattern in which a model selects a host-provided function and returns structured arguments for it. The host application, rather than the model itself, validates those arguments, runs the operation, and returns a structured result to the model. Anthropic’s implementation guidance makes this boundary explicit: the model expresses intent while application code retains responsibility for execution and validation. Anthropic’s tool-use documentation

That distinction matters because a valid request is not proof of a completed side effect. A request to create a Microsoft 365 meeting may reach the calendar API, succeed there, and still appear unsuccessful to the calling service if the response is lost. Retrying without a durable identity can then create a second meeting. The same ambiguity applies to sending an email, provisioning an account, approving an invoice, or changing a production configuration.

For this reason, an LLM tool-calling architecture should record two separate facts: the model proposed an action, and the system recorded the action’s outcome. A chat transcript alone cannot reliably provide the second fact.

Why llm tools need distributed-systems discipline

Network failures are ambiguous by nature. A timeout can mean the downstream service is slow, unavailable, or already finished the requested mutation but failed to return a response. Amazon’s guidance on idempotent APIs explains why retrying mutable requests requires a client-provided identifier and server-side tracking of that identifier’s result. Amazon Builders’ Library: Making retries safe with idempotent APIs

LLM tool calling patterns amplify this familiar problem. Model runs may be restarted, a worker may fail after dispatching a request, users may repeat a prompt, and orchestration software may retry a workflow step. Each event can produce a request that looks reasonable in isolation. Without a workflow identity, the downstream service cannot distinguish a recovery attempt from a new instruction.

The reliable unit is not an individual model response. It is a durable execution record that carries an action identity, authorization context, validated arguments, current status, retry history, and a normalized result. Durable workflow systems can persist progress and recover work after failures; Temporal describes agentic flows as distributed systems that benefit from durable activities, retries, replay, and event history. Temporal’s analysis of durable agent workflows

A production architecture for LLM tool calling

A dependable service separates reasoning from execution. The model can choose from constrained tool definitions, but a deterministic executor owns policy decisions and all contact with external systems. This keeps the model useful while putting reliability controls in software components teams can test, operate, and audit.

Layer	Responsibility	Failure it contains
Tool contract	Defines input schema, output schema, permissions, and side-effect class.	Malformed or overly broad requests.
Policy gateway	Validates identity, scope, approval requirements, and business rules.	Unauthorized or unsafe execution.
Execution ledger	Persists action ID, idempotency key, state transitions, and result references.	Lost context after retries or restarts.
Worker	Calls the downstream service with timeout, retry, and backoff policy.	Transient downstream faults.
Observability layer	Connects model request, tool call, worker attempt, and external response.	Unexplained automation incidents.

Start with contracts, not model prompts

Tool schemas should describe more than argument shapes. Include an explicit side-effect classification such as read-only, reversible write, or irreversible write. Define the caller identity expected by the tool, required approval level, maximum scope, and the error categories the executor can return. Teams introducing shared tool interfaces can apply the same compatibility mindset used for APIs; MCP tool contract testing for internal automations is a useful companion for designing changes that do not surprise clients.

Validate model-supplied values at the policy gateway. JSON schema can establish structure, but it cannot decide whether a recipient is permitted, whether a requested date falls within an acceptable range, or whether a user has authority to change a payroll field. Those are deterministic business checks and should remain outside the prompt.

Create an execution ledger before dispatching a write

For each requested operation, create a durable record before invoking the downstream service. A useful record contains a workflow ID, action ID, idempotency key, tool name and version, canonicalized arguments, requesting principal, approval reference when required, attempt count, timestamps, and an outcome state.

Use states such as proposed, validated, awaiting_approval, dispatching, succeeded, failed_retryable, failed_final, and unknown_outcome. The last state needs special care. After a timeout on a non-idempotent downstream API, the executor may not know whether the action happened. It should query a status endpoint or use a reconciliation process where available instead of blindly issuing another write.

Idempotency keys make retries meaningful

Generate an idempotency key from the durable action identity, not from the raw prompt. The key should remain stable through worker restarts and retry attempts, while a genuinely new approved action receives a new key. Store the first accepted request’s result against that key and return the stored outcome for matching retries.

When an external platform supports idempotency keys, pass the key through. When it does not, use a local deduplication strategy based on a business-safe uniqueness rule, such as an immutable request ID recorded with the created resource. This is not perfect equivalence: some operations cannot be safely deduplicated without support from the downstream system. In those cases, reduce automation, add reconciliation, or require confirmation.

Idempotency does not mean every duplicate-looking request should be merged. Amazon notes that callers can intend similar operations more than once. The execution ledger should therefore distinguish a retry of one action from a separate action requested later, rather than deduplicating solely by comparing natural-language text. Amazon’s idempotency guidance

Set timeout and retry policy per tool

A universal retry setting is a source of incidents. A read-only CRM lookup can tolerate a short timeout and several retries. A payment submission, account deletion, or outbound email requires a different policy because an ambiguous outcome may carry a real side effect.

Set a timeout based on the downstream service’s expected latency and the caller’s overall deadline. Retry only errors that are plausibly transient, such as rate limiting, temporary unavailability, or selected connection failures. Use bounded exponential backoff with jitter so simultaneous automated requests do not repeatedly hit a recovering dependency at once. Amazon’s timeout, backoff, and jitter guidance

For Java services, resilience controls can prevent a failing dependency from consuming worker capacity. The operational considerations are explored in designing Java circuit breakers for LLM tool calls. Such controls do not replace idempotency; they limit pressure on a failing dependency while idempotency makes a later recovery attempt safer.

Put human confirmation at consequential boundaries

Confirmation should be based on effect, not on whether the model appears confident. OWASP identifies excessive agency as a risk when an LLM has broad permissions or can act without suitable controls, and recommends least privilege, human oversight for consequential actions, and execution auditing. OWASP LLM06:2025 Excessive Agency

A useful policy is to automatically perform narrow, reversible, low-impact operations under a specific user identity. Require a clear approval step for operations that send external communications, alter access, change financial data, delete records, or affect many people. Present the confirmed action as deterministic data: target, scope, change summary, and intended effect. Do not ask a user to approve a vague restatement of the model’s reasoning.

Least privilege also applies to LLM AI tools themselves. Prefer tools that expose a constrained business action such as schedule_meeting_for_team over a general-purpose administrator endpoint. The narrower interface provides a more reviewable policy surface and reduces the impact of an incorrect or injected request.

Observe the execution, not just the conversation

When an automation incident occurs, operators need to connect a user request to model output, validation, approval, dispatch attempts, downstream request IDs, and final state. OpenTelemetry’s generative AI semantic conventions provide vendor-neutral guidance for recording model operations and tool invocations within distributed traces. OpenTelemetry semantic conventions for generative AI

Record structured events, while protecting sensitive inputs and outputs through redaction and access controls. At minimum, make it possible to answer which tool version ran, who authorized it, which idempotency key it used, how many attempts occurred, which dependency was called, and whether reconciliation confirmed the outcome. This evidence is more useful than retaining a successful-looking assistant message.

Evaluation belongs here as well. LLM evaluation tools can assess whether a model selected an appropriate tool or produced valid arguments, while production telemetry reveals whether the system executed safely and reliably. These are complementary disciplines. A benchmark can identify regressions in tool selection, but it cannot establish that a real write operation was deduplicated after a timeout.

Implementation checklist for llm-based tools

• Classify every tool by side-effect level and required approval boundary.
• Validate model arguments, user identity, authorization, and business constraints outside the model.
• Create a durable execution record before dispatching any mutable operation.
• Assign a stable idempotency key to each approved action and propagate it downstream where supported.
• Use tool-specific deadlines, retryable error classes, exponential backoff, and jitter.
• Represent ambiguous timeouts as an explicit state and reconcile them before another write.
• Trace the full path from model request through worker attempts and downstream response.
• Review tool contracts as part of normal API and security change management.

For teams deciding how to package internal capabilities, this Skills versus MCP decision framework can help clarify interface ownership. The underlying reliability requirements remain the same whichever protocol or orchestration layer is selected.

These controls fit within a broader delivery practice. A partner building dependable internal automation can connect tool policies, workflow persistence, observability, and existing systems through custom software development services, rather than treating AI automation as a disconnected experiment.

Sources

• Anthropic: Tool Use Implementation Guide
• Amazon Builders’ Library: Making Retries Safe with Idempotent APIs
• Amazon Builders’ Library: Timeouts, Retries, and Backoff with Jitter
• OWASP GenAI Security Project: Excessive Agency
• OpenTelemetry: Generative AI Semantic Conventions
• Temporal: Why Agentic Flows Need Distributed-Systems Discipline

FAQ

Editorial note: AI assisted with research and drafting. Sources were selected for verification.

Related articles

• Designing Java circuit breakers for LLM tool calls
• How to Test Prompt Injection Defenses in Tool-Enabled LLM Automations
• How Platform Teams Introduce MCP Tool Contracts Without Breaking Clients