Background jobs move slow, retryable, or independently scheduled work off the path that serves an immediate user request. Examples include sending a receipt, importing a file, generating a report, processing a webhook, or reconciling an external system. The goal is not simply faster HTTP responses. It is a durable contract: the request has been accepted, the work has a traceable identity, a worker will attempt it under defined rules, and the user or operator can understand the final outcome. RFC 9110 explains why retry behavior depends on semantics and idempotency. A background job system must make that same question explicit for every business operation it handles.
Separate acceptance from completion
First decide what a synchronous request promises. It may promise that a payment request was recorded, a report was requested, or a file was accepted for validation; it should not promise a downstream outcome that has not happened. Store a durable business record and job record in the same transactional boundary where possible, then return a status that names the next observable state. Give callers a stable operation identifier and a way to check it. When a request cannot be accepted, return a precise, actionable error rather than quietly creating an impossible job. RFC 9457 provides a standard structure for API problem details, but the application still needs domain-specific fields and language.
| Moment | Durable fact | Caller-facing meaning |
|---|---|---|
| Accepted | Business intent and job identity recorded | Work is pending, not complete. |
| Claimed | A worker holds a bounded lease | Processing is underway; recovery remains possible. |
| Succeeded | Required side effect and result recorded | Outcome is available with evidence. |
| Failed or parked | Attempt history and classification recorded | A retry, correction, or human review is required. |
Make delivery durable
The gap between committing business data and publishing a message is where many systems lose work. If the database commit succeeds but the process crashes before enqueueing, the intended task may never run. If publication succeeds before the business record commits, a worker may see incomplete state. A transactional outbox or equivalent reconciliation process can close that gap: record the intent with the business change, relay it asynchronously, and keep enough state to detect an unpublished record. This does not provide magical exactly-once execution. It provides a recoverable record of intent. Build workers to tolerate redelivery, because brokers, networks, leases, and deploys can all create duplicate attempts.
Design idempotent side effects
Use an idempotency key tied to the business operation, not an incidental worker attempt. Before performing an external side effect, record or query the state that proves whether the effect already happened. For a payment provider, retain the provider operation reference; for an email, decide whether one logical message may be resent and make that decision visible; for data import, use a source row identity and a deterministic upsert rule. NIST's Secure Software Development Framework supports the broader discipline of building security and reliability practices into the lifecycle. Document retryable failures, terminal failures, timeouts, and manual-correction paths so a job does not merely spin until an operator gives up.
| Failure class | Worker response | Evidence to keep |
|---|---|---|
| Transient dependency | Retry with bounded delay and jitter | Attempt count, dependency response, next retry. |
| Invalid input | Stop and expose correction path | Validation detail and originating operation. |
| Uncertain side effect | Reconcile using business key | External reference and reconciliation result. |
| Poison message | Park for review without blocking others | Payload reference, classification, and owner. |
Operate with traces and SLOs
A queue length alone is not an operating picture. Track age of the oldest eligible job, time from acceptance to completion, retry rate, success rate by job type, lease expirations, parked work, and downstream dependency failures. Correlate the user request, business record, job, worker attempt, and external call so support can answer what happened without assembling a forensic puzzle from unrelated logs. OpenTelemetry documentation offers a common vocabulary and tooling path for traces, metrics, and logs; use it to propagate correlation context, not to replace meaningful event names. Set service objectives around the user outcome, such as report availability within a target time, rather than around a broker metric alone.
Plan for change and replay
Jobs can remain queued across deploys, schema changes, and retries, so payloads need versioning and compatibility rules. Prefer a compact reference to durable domain data where a worker can safely reload current state; use self-contained payloads when replay must preserve the original request, then version them deliberately. Make replay a controlled operation with scope, rate limits, observability, and a clearly stated business effect. A mass replay after a bug can amplify the original error if idempotency is only assumed. The related REST API contracts checklist is a useful companion: both APIs and jobs need stable semantics, explicit errors, and careful evolution.
Set capacity and fairness rules
A job system shares finite worker, database, and downstream capacity, so scheduling policy is part of product behavior. Classify work by urgency and cost, then set concurrency, rate, timeout, and queue-age limits for each class. A customer-facing confirmation may deserve a short target and reserved capacity, while a nightly backfill can yield whenever a provider slows down. Guard against one large tenant, malformed import, or retry storm starving unrelated work. Backpressure should be visible to the request path: when the system cannot accept another large job safely, return a clear response or defer the request rather than placing an unbounded promise in a queue. Estimate capacity using actual execution time distributions and dependency limits rather than only average throughput. During an incident, operators need controls to pause a job type, lower concurrency, drain a safe subset, or redirect a dependency without redeploying application code. Those controls require authorization and audit trails because they change customer-visible outcomes. A queue that is merely fast under normal traffic is not yet an operated background-job service.
Give users an honest status
Asynchronous work changes the product interface. A user who starts an export, upload, invitation, or reconciliation needs to know whether the request was recorded, when to expect an outcome, where to find it, and what to do if it fails. Avoid a vague success toast that implies the downstream work has completed. Instead show a status tied to the stable operation identifier, update it from durable state, and preserve a useful history where the result matters later. Notifications should be idempotent too: a retrying job must not confuse someone with repeated completion messages. For operations staff, provide filters for overdue, parked, and repeatedly failing work, with links to the originating record and safe controls to retry or cancel. Define retention for completed job details so support can explain a historical event without retaining sensitive payloads indefinitely. The user experience is a first-class part of job reliability because it translates internal uncertainty into a decision a person can act on.
Key takeaways
- Return acceptance separately from completion and give callers a stable operation identity.
- Persist the intent to do work so publication gaps can be reconciled.
- Make external side effects idempotent at the business-operation level.
- Classify retries, terminal failures, uncertainty, and human review instead of treating every error alike.
- Operate from end-to-end outcome signals and plan payload compatibility and replay before production incidents.
Frequently asked questions
When should work be a background job? Use one when the request need not wait for completion, when the work may be retried independently, or when it benefits from scheduling and rate control. Are background jobs eventually consistent? Often, yes; say what state is authoritative while the operation is pending. How many retries are right? Set them by dependency behavior, business urgency, and harm from delay or duplication, then cap them. Can jobs call each other? They can, but avoid hidden chains with no overall deadline or visibility; model the workflow and its compensation explicitly.
For each job type, run a short operational review: submit one operation, interrupt a worker, retry delivery, inspect the status, and resolve a deliberately invalid input. The team should be able to explain every transition and identify the durable evidence behind it. That rehearsal is far cheaper than discovering hidden ambiguity during a customer-facing delay.
Conclusion
Reliable background jobs are not invisible plumbing. They are a durable, observable promise about work that continues after a request ends. Define the business outcome, preserve intent, make repetition safe, and give both users and operators an honest view of progress and failure.
Choose delivery semantics and ownership
Define a job by its business outcome, not by the queue technology. Write the input, authoritative source, expected effect, deadline, retry policy, idempotency key and human owner. “Send welcome email” is incomplete if the job can run after an account is deleted, if the provider accepts a request but the worker times out, or if a duplicate message is harmful. These cases determine whether the job needs an outbox, a deduplication record, a lease or a manual review state.
| Decision | Choose first when | Evidence to keep |
|---|---|---|
| Boundary | The outcome has one accountable owner. | Named owner, input and success condition. |
| Fallback | A dependency can be slow, unavailable or wrong. | Visible state, retry rule and escalation path. |
| Change | The system will learn or scale after launch. | Migration, review cadence and stop condition. |
Budget retries, workers, and review
Choose delivery semantics deliberately. At-least-once delivery is often practical, but it means the handler must make repeated execution safe. Store a durable operation key, check the current business state before mutating it and record the result atomically where possible. Retries should distinguish transient dependency failure from invalid input, permission loss or a permanently missing record. A retry that ignores that distinction can turn one bad payload into an expensive incident.
Timing is part of the user promise. Set a schedule or delay with a reason, define what “late” means and expose pending status to the caller. Use a dead-letter or review path that an operator can understand; a failed item should include a safe identifier, last error class, attempt history and next action without exposing secrets. OpenTelemetry’s trace and context concepts help connect the request, enqueue, worker attempt and downstream call.
Exercise pause, replay, and recovery
Load-test the queue as a system. Vary burst size, dependency latency, worker restarts, clock skew and a poison message. Watch queue age, throughput, retry rate, concurrency, dead-letter count and downstream saturation. Capacity is not just the number of workers: it is the work each worker can safely perform while preserving database, provider and customer-facing limits. Document how to pause, drain, replay and cancel a job before production makes the question urgent.
| Signal | Healthy question | Action when it drifts |
|---|---|---|
| Outcome | Did the intended business result happen? | Inspect examples and pause unsafe scope. |
| Reliability | Can the path recover from delay or duplication? | Use retry, replay or manual review controls. |
| Ownership | Can a named person explain the current state? | Route the exception and update the runbook. |
Background-job design connects to technical-debt risk when queue behavior is inherited, to internal tool UX when status reaches operators, and to schema design when operation identity and replay records persist. Follow the link that matches the job’s external contract.
A background job is production-ready when acceptance, completion, retry, cancellation, and replay have distinct evidence and named owners. Start with one queue and one side effect; broaden only after operators can explain both delay and duplicate work.