Streaming and Batch Ingestion Patterns

Goal

The previous section established that trusted analytics begins with data flow. The first practical design decision in that flow is ingestion: how signals arrive, how they are captured, how failures are handled, and how quickly downstream consumers can trust the result.

The durable idea is:

Ingestion design starts with how data arrives and how quickly it must become trustworthy.

The Short Version

The easy question is:

Should we use streaming or batch?

The better architecture question is:

How does the data naturally arrive, what freshness does the business need, and what recovery behavior is required?

Streaming and batch are not maturity levels. They are different reliability shapes.

Pattern	Best Fit	Typical Data	Design Pressure
Streaming	Continuous records that arrive throughout the day.	Clickstream, telemetry, order events, inventory signals, logs.	Throughput, ordering, lag, replay, backpressure, consumer ownership.
Batch	Scheduled extracts, files, snapshots, and historical loads.	Supplier catalogs, settlement files, finance exports, daily inventory snapshots.	Completeness, validation, reconciliation, restart, auditability.

Use the pattern that matches the business promise. A daily settlement file does not become better architecture just because it is forced into a stream. A high-volume clickstream does not become simpler because someone waits for a nightly file.

Streaming Ingestion

Streaming ingestion fits data that arrives continuously or needs timely reaction. It is useful when consumers need fresh records, when volume is steady or high, or when multiple downstream processors need to read the same flow independently.

A streaming path often looks like this:

Producer applications
  -> record stream
  -> validation and routing
  -> durable landing or retention
  -> stream processors and downstream consumers

Streaming design needs more than a broker or event service. It needs decisions about:

Record contracts and schema evolution.
Partitioning, ordering, and consumer position.
Retention and replay windows.
Duplicate handling and idempotent processing.
Late-arriving records.
Backpressure when consumers fall behind.
Monitoring for lag, throughput, rejected records, and cost.

For example, clickstream events may feed behavior analytics, personalization, fraud signals, and experimentation dashboards. Those consumers may not all need the same latency, but they all depend on a reliable record flow.

Batch Ingestion

Batch ingestion fits data that arrives on a schedule, as an export, or as a file. It remains common in enterprise systems because many boundaries are still organized around business periods: daily reconciliation, monthly finance close, supplier catalog refreshes, payroll cycles, partner reporting, and historical backfills.

A batch path often looks like this:

Source system or partner
  -> scheduled file or extract
  -> landing area
  -> validation and reconciliation
  -> accepted, rejected, or quarantine path
  -> curated processing

Batch design needs decisions about:

Expected arrival schedule and lateness thresholds.
File format, schema version, and naming convention.
Completeness checks such as row counts, totals, or control files.
Reprocessing and restart behavior.
Quarantine for corrupt or suspicious records.
Acknowledgement and reconciliation with the source owner.
Monitoring for missing files, rejected rows, and stale outputs.

Batch is not less architectural because it is familiar. A finance export that feeds revenue reporting may need stronger evidence and reconciliation than a near-real-time dashboard.

Change Data Capture, Micro-Batches, and Backfills

Real platforms rarely use only one pure pattern. Several useful variations sit between continuous events and a nightly full extract:

Ingestion Shape	Useful When	Main Risk
Change data capture	Consumers need inserts, updates, and deletes from an operational database without repeated full copies.	Database changes may expose physical schemas rather than stable business events.
Micro-batch	Data should be processed every few minutes, but continuous per-record processing is unnecessary.	Teams may call it real time without defining the actual freshness window.
Snapshot	Consumers need a complete state at a known business time.	Comparing snapshots can be expensive and can hide changes between capture points.
Incremental batch	The source can identify records changed since a checkpoint.	A bad checkpoint or mutable timestamp can skip or duplicate data.
Historical backfill	A new product or corrected rule needs older data reprocessed.	Backfill volume and side effects can overwhelm normal processing or double-count results.

These shapes can coexist. An order product might use change data capture for frequent state changes, a daily reconciled snapshot for finance, and a historical backfill when revenue logic changes.

Event Time, Windows, and Delivery Semantics

Streaming systems introduce two different clocks:

Event time is when the business event happened.
Processing time is when the platform observed or processed it.

They differ whenever devices disconnect, networks delay records, producers retry, or consumers fall behind. A ten-minute sales total based only on processing time can place a late order in the wrong period.

Stream processors therefore group unbounded records into windows, such as five-minute fixed windows, sliding windows, or user-session windows. A watermark is the processor's estimate that most records for an event-time point have arrived. The design must still decide how long to accept late records and whether corrected results replace, retract, or supplement earlier outputs.

Delivery language also needs care:

Delivery Behavior	Architecture Meaning
At-most-once	A record is not retried after uncertain failure, so loss is possible but duplicates are reduced.
At-least-once	Records are retried, so loss is reduced but consumers must expect duplicates.
Effectively-once result	Retries may occur, but idempotency, deduplication, transactions, or deterministic writes keep the business result correct.

An infrastructure feature labelled "exactly once" does not automatically make the whole data product exactly once. The source, transport, processor, state store, output, and replay path all participate in the outcome. For architecture discussions, describe the end-to-end business result rather than relying on a broker label.

Failure and Replay

Ingestion is a reliability boundary. Data can be late, duplicated, malformed, incomplete, out of order, or unexpectedly changed at the source.

Issue	Design Question
Late data	How long can records arrive after the expected window, and which outputs must be corrected?
Duplicate data	Which keys or fingerprints allow safe deduplication?
Schema change	Who owns compatibility, and how are consumers warned?
Bad records	Where do rejected records go, and who investigates them?
Missing feed	Which alert fires, and which team owns recovery?
Replay	Can the pipeline reprocess without double-counting or corrupting downstream products?

These questions should be answered before the platform is under pressure. During an incident, “just rerun it” is not a recovery plan unless idempotency, deduplication, and lineage are already designed.

Example: Clickstream, Inventory, and Finance

A commerce platform may use both patterns in one analytics architecture:

Web and mobile clickstream
  -> streaming ingestion
  -> raw event zone
  -> near-real-time behavior analytics

Inventory updates
  -> event or stream ingestion
  -> inventory availability signals
  -> replenishment and forecast inputs

Supplier catalog and settlement files
  -> batch ingestion
  -> validation and reconciliation
  -> curated product and finance datasets

The platform should not hide these differences behind one generic “data pipeline” label. Clickstream needs lag and throughput monitoring. Inventory needs freshness and duplicate handling. Settlement files need completeness, control totals, and audit evidence. Each path has a different promise.

Common Mistakes

Using streaming when the business only needs a daily reconciled answer.
Treating batch files as less important because they sound old-fashioned.
Monitoring pipeline success but not data freshness, rejected records, or downstream impact.
Forgetting deduplication and replay rules until an incident causes double-counting.
Allowing source teams to change schemas without a contract and consumer notification path.
Using processing time when the business question depends on when an event actually happened.
Treating a transport delivery guarantee as proof that downstream business results cannot duplicate.

If Someone Asks You

How do you choose between streaming and batch ingestion?

A good answer is:

Choose by arrival pattern, freshness need, reliability, and recovery behavior. Streaming fits continuous records, high volume, near-real-time use, and replayable consumer processing. Batch fits scheduled extracts, files, reconciliation, historical loads, and lower-freshness business cycles. Both need contracts, validation, monitoring, replay rules, duplicate handling, ownership, and cost controls.

What You Should Remember

Ingestion is a reliability boundary, not just a copy step.

Questions to Check Understanding

What makes streaming different from batch ingestion?
Why does replay need idempotency or deduplication?
Which ingestion health signals should a team monitor?
When would change data capture be preferable to repeated snapshots?
Why do event time and processing time produce different answers?

Further Study

Apache Beam Programming Guide - Explore windowing, event time, watermarks, late data, triggers, state, and timers in a provider-neutral processing model.