CH12-03: Data Lake Organization and Storage

Goal

Ingestion gets data into the platform. Storage decides whether that data remains understandable, secure, usable, and affordable over time.

Azure Storage can provide a durable foundation for object storage and data lake patterns. Azure Data Lake Storage builds on Azure Storage with capabilities useful for analytics workloads, such as a hierarchical namespace. But a data lake is not useful just because it can hold a lot of files. It is useful when the layout, ownership, access, lifecycle, and meaning are designed.

The durable idea is:

A data lake is useful only when storage layout, ownership, access, lifecycle, and meaning are designed.

The Short Version

The easy question is:

Where do we store the data?

The better architecture question is:

How will people know what the data means, whether they can trust it, who owns it, and how long it should live?

A data lake should preserve source data, support transformation, and serve trusted datasets. Without organization, it becomes an unmanaged archive where teams are afraid to delete anything and unsure which data is safe to use.

Lake Zones

A common mental model is to organize data into zones:

Data lake
  +-- raw
  +-- validated
  +-- curated
  +-- serving

These zone names are not a universal law. Some organizations use bronze, silver, and gold. Others use landing, raw, refined, and published. The names matter less than the responsibilities.

The architecture should prevent users from treating raw data as trusted business truth. Raw data is valuable because it preserves evidence and replayability. Curated data is valuable because it has meaning and quality expectations.

Naming, Layout, and Metadata

Data lake organization should make the platform navigable.

Design choices include:

• Containers or top-level folders for zones, domains, or environments.

• Dataset names that reflect business meaning, not only source-system table names.

• Partitioning by date, tenant, region, or other access patterns when useful.

• File formats that fit processing and query needs.

• Schema version and source metadata.

• Ownership tags or catalog entries.

• Arrival time, processing time, and lineage references.

A simple path pattern might be:

/raw/clickstream/events/yyyy/mm/dd/
/validated/orders/order-header/yyyy/mm/dd/
/curated/sales/product-performance/yyyy/mm/dd/
/serving/executive-dashboard/revenue-summary/

This example is intentionally simple. A real platform may need more rules, but the goal is the same: a person or pipeline should be able to understand the zone, domain, dataset, and time range without guessing.

File Formats and Partitioning

File format is an architecture decision because it affects processing, query performance, interoperability, and cost.

Raw zones may preserve original JSON, CSV, Avro, Parquet, or partner-provided formats. Curated zones often use columnar formats such as Parquet or table formats supported by the selected processing platform because they can improve analytical query efficiency.

Partitioning can help when queries commonly filter by time, tenant, region, product category, or another stable dimension. Poor partitioning can create too many small files or make queries inefficient. The design should follow actual access patterns, not a generic rule copied from another platform.

Access, Security, and Private Connectivity

The lake often contains sensitive data. Access should be designed by role, dataset sensitivity, zone, and business purpose.

Important controls include:

• Identity-based access through Microsoft Entra ID and Azure RBAC where appropriate.

• More restrictive access to raw and sensitive zones.

• Private endpoints or private network access where the lake should not be reached publicly.

• Encryption and key management decisions.

• Audit logs for access and administrative changes.

• Data classification and access review for regulated or sensitive datasets.

The raw zone is not automatically safer because fewer people use it. It may contain unmasked source data and should often be more tightly controlled than curated serving datasets.

Retention, Lifecycle, and Cost

Data feels cheap until it grows quietly for years. Storage design should define retention and lifecycle policies early.

Ask:

• How long must raw source data be retained for replay, audit, or compliance?

• When can temporary landing files be deleted?

• Which curated datasets must be kept for historical reporting?

• Which data can move to cooler storage tiers?

• Who approves deletion or retention changes?

• How are duplicated datasets reviewed?

Lifecycle policies can reduce cost by moving older data to lower-cost tiers or deleting data after a retention period. The policy must match legal, compliance, operational, and business requirements.

Example: Telemetry to Curated Dataset

Telemetry may land in raw storage exactly as received. A validation job checks schema, required fields, timestamps, and source identifiers. Valid records move to a validated zone. Processing then transforms records into curated datasets used by dashboards and downstream applications.

Raw telemetry
  -> schema and quality validation
  -> validated telemetry
  -> enrichment and aggregation
  -> curated operational metrics
  -> dashboard and alerting datasets

If a later transformation fails, the team can reprocess from validated or raw data. If a source system changes schema unexpectedly, the validation layer catches the problem before it damages trusted reports.

Common Mistakes

• Letting the lake become an unmanaged dumping ground.

• Mixing raw and curated data without labels, contracts, or ownership.

• Giving broad access because it is easier than designing dataset-level controls.

• Ignoring lifecycle, retention, and cost until storage grows.

• Partitioning or choosing file formats without considering query and processing patterns.

If Someone Asks You

How should a data lake be organized in Azure?

A good answer is:

I organize the lake by responsibility, not just folders. A common model uses raw, validated, curated, and serving zones. Raw preserves source data for replay. Validated confirms basic quality. Curated contains trusted business datasets. Serving shapes data for dashboards, exports, models, or applications. Around that layout, I define naming, formats, partitioning, metadata, ownership, access controls, private connectivity, encryption, audit logs, retention, lifecycle, and cost ownership.

What You Should Remember

Data lake design is about durable storage plus meaning, ownership, access, and lifecycle.

Questions to Check Understanding

1. Why are raw and curated zones different?

2. What metadata helps users trust data?

3. How can lifecycle policies affect cost?