CH12-06: Data and Analytics Architecture Scenario

Goal

This section applies the Chapter 12 model to a realistic Azure data and analytics platform. The goal is not to choose the fanciest analytics product. The goal is to follow data from source to trusted use and explain the owner, control, and trade-off at each step.

The durable idea is:

A strong analytics architecture explains how data arrives, where it lands, how it is transformed, who can trust it, and how it is operated.

Scenario

An e-commerce company wants a data platform for product performance, customer behavior, inventory planning, and executive reporting.

The company has several source systems:

• Order data from the commerce application.

• Clickstream events from web and mobile applications.

• Inventory exports from warehouse and ERP systems.

• Support data from a customer service platform.

• Product catalog data from merchandising tools.

The business wants dashboards, forecasts, downstream exports, and future data products for personalization and operations. Some data is sensitive, and executives need confidence that reported metrics are consistent.

Architecture Sketch

One possible Azure architecture is:

Source systems
  |-- Commerce database extracts
  |-- Clickstream events
  |-- Inventory and ERP files
  |-- Support platform exports
  |-- Product catalog exports
       |
       v
Ingestion
  |-- Event Hubs for clickstream
  |-- Batch landing for extracts and files
       |
       v
Data lake zones on Azure Storage / Data Lake Storage
  |-- raw
  |-- validated
  |-- curated
  |-- serving
       |
       v
Processing platform
  |-- Databricks, Synapse, Fabric, or suitable combination
       |
       v
Consumption
  |-- dashboards
  |-- forecasts
  |-- exports
  |-- downstream data products

Governance, security, lineage, monitoring, and cost controls cover the full flow.

This design uses names from Azure, but the structure is more important than the product list: source, ingestion, storage, processing, serving, governance, and operations.

Step 1: Define Business Data Products

The team should name the data products before building every possible pipeline.

Naming data products keeps the platform from becoming a copy of everything. It also helps identify owners and quality rules.

Step 2: Choose Ingestion Paths

Clickstream events arrive continuously from applications. Event Hubs can receive this high-volume event stream and provide a controlled ingestion point.

Order data, inventory data, support data, and product catalog data may arrive through batch extracts or scheduled pipelines. Those feeds can land in raw storage with arrival metadata, source identifiers, schema version, and load status.

The ingestion plan might be:

Each feed needs an owner, expected schedule, schema contract, validation rule, and alert path.

Step 3: Organize Lake Zones

The lake should preserve source data and separate it from trusted datasets.

raw/
  clickstream/
  orders/
  inventory/
  support/
  product-catalog/

validated/
  clickstream/
  orders/
  inventory/
  support/
  product-catalog/

curated/
  product-performance/
  customer-behavior/
  inventory-planning/
  revenue/

serving/
  executive-dashboard/
  merchandising-dashboard/
  forecast-exports/

Raw zones preserve what arrived. Validated zones prove basic expectations. Curated zones apply business meaning. Serving zones shape data for consumption.

Sensitive data should have tighter access controls. Retention and lifecycle policies should differ by dataset, not rely on one blanket rule.

Step 4: Process and Serve Data

The processing platform depends on team skill and workload shape.

One team might choose Azure Databricks for large-scale transformations, feature preparation, and notebook-driven data engineering. Another might choose Synapse-oriented warehousing for SQL-heavy analytics. An organization standardizing on integrated analytics and BI may evaluate Microsoft Fabric.

The design should avoid pretending this is only a product decision. Whichever platform is chosen must support:

• Repeatable pipelines.

• Quality checks and reconciliation.

• Lineage from source to curated data products.

• Job scheduling and backfill.

• Access controls for sensitive fields.

• Cost monitoring for compute and queries.

• Clear ownership for failures.

Serving might include Power BI dashboards, governed tables, exports for partner systems, data APIs, or datasets for forecasting and AI use cases. Serving data should be documented enough that consumers know freshness, owner, and intended use.

Step 5: Govern the Platform

Governance should be designed into the platform from the start.

For this scenario:

• The finance team owns revenue definitions.

• Merchandising owns product performance meaning.

• Operations owns inventory planning assumptions.

• Digital product teams own clickstream event contracts.

• The data platform team owns ingestion standards, lake layout, processing platform reliability, and shared controls.

Microsoft Purview or another catalog and governance approach can help track datasets, classifications, lineage, ownership, and discovery. Azure RBAC, Microsoft Entra ID, private networking where appropriate, encryption, audit logging, and access reviews help protect data.

Step 6: Operate and Control Cost

The platform needs monitoring across the flow:

For example, if inventory files arrive late, the inventory planning dataset should show a freshness warning and alert the owning team. If clickstream volume doubles overnight, the platform should show whether Event Hubs, processing jobs, storage, and downstream dashboards are still healthy and affordable.

Alternatives and Phased Rollout

The company does not need to build everything on day one.

A reasonable first phase might deliver order, product, and inventory reporting with batch ingestion, lake zones, curated datasets, and governed dashboards.

A second phase might add clickstream streaming through Event Hubs, more detailed behavior analysis, and near-real-time product signals.

A third phase might add forecasting datasets, downstream data products, and model features.

The important discipline is to keep governance, lineage, quality, monitoring, and cost ownership in every phase. A small trusted platform is better than a large untrusted one.

Common Mistakes

• Copying every production table into analytics without purpose.

• Choosing a platform before deciding data products, governance, and ownership.

• Treating raw lake data as safe for executive reporting.

• Ignoring refresh failures, data quality, lineage, and cost.

• Giving dashboard users inconsistent definitions of revenue, conversion, or active customer.

• Building streaming ingestion before proving that freshness requirements justify it.

If Someone Asks You

How would you design an Azure data and analytics architecture?

A good answer is:

I would start with the business data products and source owners. Then I would design ingestion by arrival pattern: Event Hubs for continuous clickstream or telemetry, and batch ingestion for scheduled extracts and files. Data would land in lake zones such as raw, validated, curated, and serving. Processing could use Databricks, Synapse, Fabric, or another suitable platform based on workload, skills, governance, and operating model. I would include cataloging, lineage, access control, quality checks, monitoring, lifecycle, and cost ownership from the beginning.

What You Should Remember

Follow the data from source to trusted use, and name the owner at each step.

Questions to Check Understanding

1. Which ingestion paths does the scenario need?

2. How should the lake be organized?

3. What makes the final dashboards trustworthy?