CH12-02: Streaming and Batch Ingestion Patterns

Goal

The previous section introduced trusted data flow. The first practical step in that flow is ingestion: getting data from source systems into the analytics platform reliably enough that later teams can trust it.

Ingestion design is not only about moving bytes. It decides how quickly data arrives, whether missed data can be replayed, how duplicates are handled, who owns failures, and what downstream systems can expect.

The durable idea is:

Ingestion design starts with how data arrives, how quickly it is needed, and how reliably it must be captured.

The Short Version

The easy question is:

Should this be streaming or batch?

The better architecture question is:

How does the data naturally arrive, how fresh must it be, and what happens when ingestion fails?

Streaming and batch ingestion are both valid. They solve different arrival patterns.

Do not make streaming the default just because it sounds modern. If a finance extract arrives once per day and the business reviews it tomorrow morning, batch may be simpler, cheaper, and easier to govern.

Streaming Ingestion

Streaming ingestion fits data that arrives continuously and may have high volume. In Azure, Event Hubs is a common service for high-throughput event streaming and ingestion. It can receive large flows of events, retain them for a configured period, and let consumers process the stream.

Streaming is useful when:

• Events arrive continuously.

• Freshness matters in minutes or seconds.

• Volume is too high or too constant for simple file drops.

• Consumers may need to process records independently.

• The platform needs a replayable stream for downstream processing.

Examples include click activity, device telemetry, application logs, and transaction signals.

A simple streaming path might be:

Web and mobile apps
	-> Clickstream events
	-> Event Hubs
	-> Stream processor or capture path
	-> Raw lake zone

The ingestion design still needs ownership. Someone must define event schemas, partitioning or routing choices, retention, replay expectations, monitoring, and cost limits.

Batch Ingestion

Batch ingestion fits data that arrives on a schedule or as files. It is common for enterprise systems, partner integrations, finance data, inventory exports, and older applications.

Batch is useful when:

• Data arrives hourly, daily, weekly, or on a business schedule.

• Source systems produce files or database exports.

• Latency requirements are measured in hours rather than seconds.

• Validation and reconciliation are more important than immediate reaction.

• The team needs simple restart and audit behavior.

A simple batch path might be:

ERP or partner system
	-> Daily export file
	-> Landing folder in lake storage
	-> Validation job
	-> Validated or rejected zone

Batch does not mean careless. The platform should know which file was expected, when it arrived, whether the row count changed unexpectedly, whether schema changed, and what to do if the file is late or corrupt.

Landing Data Durably

Ingestion should usually land data durably before heavy transformation. This gives the platform a recovery point. If a downstream job fails, the team can replay or reprocess from a known source rather than asking the source system to resend everything.

For streaming, the durable point may be Event Hubs retention, a capture path into lake storage, or both depending on requirements. For batch, it may be an original file in a raw landing zone.

The raw landing area should preserve enough context to answer:

• Where did this data come from?

• When did it arrive?

• What schema or version was expected?

• Was the ingestion complete?

• Can it be replayed or reprocessed?

• Who owns this source feed?

Late, Duplicate, and Bad Data

Real ingestion is messy. Events may arrive late. Files may be resent. Source systems may change a column name. A partner may send duplicate records. A device may upload data after being offline.

The design should define rules for:

These rules matter because analytics errors can look like business facts. A dashboard that silently drops late orders may lead to the wrong decision.

Example: Clickstream and Daily Extracts

An e-commerce platform may need both patterns.

Clickstream data arrives continuously from the website and mobile app. Event Hubs fits that flow because records arrive constantly and may be processed for near-real-time behavior analysis.

Product and finance data may arrive as daily extracts. Batch ingestion fits because the data changes on a schedule, needs validation, and may require reconciliation against source-system totals.

Both paths can land in the same analytics platform, but they should not be treated as the same contract:

Clickstream events -> Event Hubs -> raw clickstream zone
Daily product file -> batch landing -> raw product zone
Daily finance file -> batch landing -> raw finance zone

Downstream processing can later combine them into curated datasets.

Common Mistakes

• Using streaming when batch latency is acceptable.

• Treating ingestion success as guaranteed without monitoring, replay, or reconciliation.

• Mixing raw and curated data without clear landing rules.

• Ignoring duplicate records, late arrivals, and schema changes.

• Letting source teams change feeds without contract ownership.

If Someone Asks You

How do you choose between streaming and batch ingestion?

A good answer is:

I choose by arrival pattern, freshness, reliability, and downstream use. Streaming fits continuous high-volume events such as telemetry or clickstream data, where freshness and replayable processing matter. Batch fits scheduled extracts, files, exports, and slower-moving business data. In both cases, I design durable landing, validation, monitoring, duplicate handling, late-data rules, replay, and ownership.

What You Should Remember

Match ingestion to arrival pattern, latency need, reliability, and downstream use.

Questions to Check Understanding

1. What data patterns fit streaming ingestion?

2. What data patterns fit batch ingestion?

3. What should teams monitor during ingestion?