The useful question in batch vs micro-batch vs streaming is not which architecture is most modern. It is when a changed fact must influence a decision before its value falls. A fraud authorization may have hundreds of milliseconds, an inventory promise may have seconds, an operations dashboard may have minutes, and a monthly close may have hours or days. Processing faster than the decision deadline can add continuous compute, state, recovery, and staffing cost without improving the outcome.
Start with the business clock, then work backward through serving, processing, transport, and source availability. The resulting latency budget must coexist with completeness and correction requirements. Apache Spark Structured Streaming uses micro-batch processing by default and offers a continuous mode with different guarantees; Apache Flink is designed for stateful computations over bounded and unbounded streams; Apache Beam separates event time, windows, triggers, and watermarks from a specific runner. These systems show that processing mode is a bundle of semantics and operations, not a refresh-frequency slider.
Define the last useful decision time
Describe the decision, actor, triggering event, action, and deadline. Ask what happens when data arrives after that point. If a warehouse alert reaches a planner in ten minutes and the next operational intervention is at the top of the hour, sub-second computation has no additional value. If a payment decision must occur before authorization, a five-minute incremental job is disqualified. Separate display freshness from action freshness: an animated dashboard can look real time while the underlying process changes only daily.
Create an end-to-end budget rather than assigning the whole deadline to processing. Source commit and capture, broker or file availability, computation, sink publication, cache refresh, network delivery, and user response all consume time. Include tail latency and outage recovery, not only median. Reserve margin for variability. A ten-second decision deadline with an eight-second source delay cannot be fixed by replacing a two-second batch with a one-second stream.
| Decision pattern | Typical useful horizon | Candidate mode | Reason |
|---|---|---|---|
| Regulatory month-end report | Hours to days | Batch | Completeness, control, and repeatability dominate |
| Hourly staffing adjustment | Minutes | Micro-batch | Bounded refresh meets the action cadence |
| Customer inventory promise | Seconds | Streaming or fast micro-batch | Stale state changes a live transaction |
| Fraud intervention | Milliseconds to seconds | Streaming | Decision must precede irreversible action |
| Executive trend dashboard | Tens of minutes to daily | Batch or micro-batch | Human review cadence limits value |
Compare modes by execution and state
Batch processes a bounded input or a bounded slice on a schedule. It can recompute a period from immutable inputs, making reasoning and backfill straightforward, but the schedule and runtime set a freshness floor. Micro-batch treats an unbounded source as a sequence of small bounded increments. It can reuse batch APIs and checkpoint progress, though each trigger adds scheduling overhead and results arrive in steps. Streaming keeps operators active and processes records continuously, often maintaining state for windows, joins, deduplication, or patterns.
The modes can share code and infrastructure. Spark describes an incremental table computation executed by default as micro-batches, while Flink can execute bounded and unbounded dataflows. Therefore, avoid equating a named framework with one mode. Ask how this job is triggered, where its state lives, when output becomes visible, what delivery guarantee the source-to-sink path provides, and how a finite replay is performed.
Choose event-time and correction semantics
Data arrives late and out of order because devices disconnect, transactions span systems, clocks differ, and upstream jobs recover. A batch can wait for a cutoff and later restate a partition. A stream uses event-time windows, watermarks or equivalent completeness estimates, triggers, and a policy for late data. Lower latency usually means publishing before all facts are known. Decide whether output is append-only, updated, retracted, or periodically reconciled.
Define correctness for the consumer. A live counter might accept provisional updates followed by corrections. An invoice or regulatory report may require a closed period and controlled restatement. If the sink cannot apply updates idempotently, an early result can be harder to correct than a later complete result. Measure source delay distribution by event class and partition before setting a watermark or batch cutoff. A single global delay assumption can discard slow mobile regions or hold every partition for one idle source.
| Dimension | Batch | Micro-batch | Continuous streaming |
|---|---|---|---|
| Freshness floor | Schedule plus runtime | Trigger interval plus runtime | Per-record path and buffering |
| State | Often externalized in tables | Checkpoint and incremental state | Long-lived operator state |
| Late data | Next run or restatement | Update later trigger | Watermarks, triggers, updates or side output |
| Replay | Rerun bounded inputs | Reset offset/checkpoint carefully | Restore state or replay log with sink controls |
| Idle cost | Little compute between runs | Can stop or poll | Usually continuous resources and operations |
| Failure scope | Run or partition | Trigger and checkpoint | State, offsets, source, sink, and deployment |
Evaluate end-to-end delivery guarantees
Exactly-once claims are scoped. A processor may checkpoint state exactly once while an external API receives a duplicate after a retry. Kafka's design documentation distinguishes at-most-once, at-least-once, and exactly-once behavior and makes clear that producer, broker, consumer, and destination behavior matter. Identify the unit that must be unique, create a deterministic key, and make sink writes idempotent or transactional. Count effects, not merely consumed messages.
For batch, idempotency usually means rerunning the same input slice produces the same published partition without duplicating side effects. For micro-batch, couple source progress and sink commit or maintain a ledger. For streaming, include checkpoint compatibility, transactional boundaries, and duplicate behavior after failover. Test process death before and after sink acknowledgment, broker redelivery, and partial destination availability. A lower-latency mode should not be approved until its failure outcomes are understood.
Price the operating model, not only compute
Compare steady compute, peak capacity, state storage, broker retention, checkpoints, network, sink writes, observability, test environments, and engineering support. Batch may create expensive peaks and miss deadlines when volume grows. Micro-batch can issue many small queries and writes with poor utilization. Streaming can consume continuously even when traffic is low and requires state and backpressure expertise. Serverless billing changes the rate shape but does not remove per-event, state, network, or operational cost.
Include organizational readiness. A team that can recover a scheduled SQL job but cannot inspect checkpoint state or replay a topic has a real delivery risk. Count on-call training, upgrade procedure, schema evolution, partition changes, backfills, and local testing. The simplest mode that meets the deadline and correctness contract often leaves more capacity for improving data quality and product behavior. Simplicity is not an excuse to miss a valuable deadline; it is a cost included in the decision.
Use hybrid paths when one mode cannot serve every need
Many products need a fast provisional path and a slower authoritative path. A stream can update operational features while a daily batch reconciles full history. A micro-batch can populate dashboards every few minutes while a monthly close freezes a controlled version. Share source events, business keys, metric definitions, and correction rules so paths do not become competing truths. Label provisional and final outputs clearly.
Avoid building two complete stacks without a reconciliation owner. Define which output wins, when it becomes final, and how differences are surfaced. Use the same test fixtures across modes and compare results for closed windows. A streaming speed layer that never reconciles will accumulate silent errors; a batch layer that overwrites recent state without version checks can erase valid updates. Hybrid architecture earns its complexity only when different deadlines create measurable value.
Run a decision-focused pilot
Replay representative traffic, including bursts, skew, late events, schema changes, and destination slowdown. Measure source-to-decision p50, p95, and p99; completeness at the deadline; correction volume; state growth; recovery time; duplicate effects; cost per useful decision; and operator intervention. Compare at least one simpler candidate. A streaming proof that only demonstrates low processing latency does not establish business freshness or production reliability.
Approve with an explicit review trigger. If the decision deadline tightens, volume grows, late-data distribution changes, or operational incidents exceed tolerance, reassess. Conversely, downgrade an expensive stream when consumers use daily snapshots. Processing mode is not a permanent identity for a data platform. It is a current answer to a measured deadline, correctness requirement, and operating capability.
Key takeaways
- Start from the last useful decision time and allocate an end-to-end latency budget.
- Treat event time, completeness, and correction as part of the mode decision.
- Verify delivery guarantees across source, processor, state, and sink.
- Price replay, state, observability, staffing, and recovery alongside compute.
- Use a hybrid path only with shared definitions and an owned reconciliation rule.
- Choose the simplest mode that meets measured deadlines under failure and peak load.
Frequently asked questions
Does real time always mean streaming?
No. For a decision made every five minutes, a reliable one-minute micro-batch may be effectively real time. Reserve continuous streaming for deadlines and interactions that benefit from continuous updates after source and serving delays are included.
Does exactly once prevent duplicate business actions?
Only if the guarantee includes the destination effect. Processor state can be exactly once while an API call repeats. Use deterministic event identity, idempotent or transactional sinks, and failure tests around acknowledgment boundaries.
Can a batch pipeline evolve into streaming later?
Yes, especially when inputs are immutable, transformations are deterministic, business keys are stable, and publication is idempotent. Designing those properties early makes any later mode change safer without paying the streaming operating cost today.
Conclusion
Batch, micro-batch, and streaming are different ways to spend latency, correctness, and operating complexity. The defensible choice begins with the decision deadline, measures the whole path, and proves recovery and correction behavior under realistic load. When teams choose from business timing rather than architectural fashion, they can deliver data while it is still useful without turning every pipeline into a permanently running distributed system.