CH19-05: Compliance, Security, Observability, and Disaster Recovery

Goal

Add compliance, data protection, access control, network isolation, audit evidence, observability, disaster recovery, and cost controls around the finance engine.

The Short Version

Finance systems carry sensitive actions and sensitive evidence. A design that can take payments but cannot explain access, protect secrets, detect failed reconciliation, or recover workflow state is not ready for production.

The durable idea is:

Finance systems must be protected, observable, recoverable, and explainable under pressure.

This section is not legal or compliance advice. It gives architecture responsibilities that you should expect to discuss with security, compliance, finance, operations, and platform teams.

Control Overlay

Add controls around every finance responsibility. Do not bolt them on after the workflows are already built.

Finance responsibility	Security control	Evidence	Alert or recovery concern
Finance APIs	Strong authentication, authorization, throttling, input validation.	Caller identity, command ID, request status.	Unusual failure rates, unauthorized attempts, duplicate commands.
Provider integration	Managed identity where possible, Key Vault for secrets, restricted outbound paths.	Provider request and correlation records.	Provider outage, timeout spikes, unexpected response patterns.
Workflow processing	Least privilege, durable state, replay controls.	Workflow instance history and state transitions.	Stuck workflows, retry exhaustion, dead-letter growth.
Ledger and invoice data	Encryption, restricted access, backup and restore testing.	Change records, adjustment history, access logs.	Write failures, unexpected adjustments, restore gaps.
Settlement and reconciliation	Segregated access, approval paths, exception ownership.	Settlement imports, match results, exception resolution.	Unmatched settlement, delayed files, material differences.
Reporting and support views	Masking, role-based access, query logging.	Report lineage and access records.	Sensitive data exposure, manual changes outside controls.

Security And Compliance Architecture

Start with identity and least privilege. Human users, applications, workflows, support tools, and finance jobs should have only the permissions they need. Azure managed identities can reduce secret handling for Azure-to-Azure access. Azure Key Vault helps centralize secrets, keys, and certificates so provider credentials and sensitive connection details are not embedded in code.

Use network boundaries deliberately. Private endpoints can provide private IP access to supported Azure services through Azure Private Link, but private access still needs correct DNS, authorization, and service-level controls. A private endpoint is not a replacement for identity, least privilege, or data protection.

Classify data early. Payment tokens, customer information, seller tax data, invoices, disputes, provider references, and finance reports do not all need the same audience. Minimize sensitive data where possible, mask what support teams do not need, and keep raw provider or payment details out of broad operational logs.

Audit trails should cover both automated and human actions:

Payment authorization, capture, void, refund, and payout commands.
Provider request and response correlation.
Invoice generation, invoice correction, and adjustment actions.
Settlement import, reconciliation match, exception, and resolution events.
Administrative changes to provider configuration, routing, limits, rules, and secrets.
Support actions that view or change financial state.

Observability For Finance Operations

Azure Monitor, Application Insights, and Log Analytics can collect and analyze telemetry across Azure resources and applications. For a finance engine, ordinary infrastructure telemetry is not enough. You need business and workflow telemetry too.

Track signals such as:

Authorization, capture, refund, invoice, payout, and settlement success rates.
Provider latency, timeout, decline, and error patterns.
Workflow states: pending, running, waiting, failed, compensated, completed.
Queue depth, dead-letter counts, retry counts, and message age.
Reconciliation match rates, unmatched amounts, delayed settlement files, and exception age.
Ledger write failures, invoice generation failures, and adjustment volumes.
Support investigation volume and finance exception queues.

Good alerts are tied to action. "Refund failures increased" is useful only if the owning team knows which workflow, provider, region, queue, or deployment changed and what recovery path to use.

Disaster Recovery And Operational Recovery

Disaster recovery is not only compute failover. For a commerce finance engine, recovery must include financial records, workflow progress, provider correlation, messages, secrets, configuration, reports, and reconciliation state.

Ask these questions:

Which records are the source of financial truth?
Which workflow states must survive a region or service outage?
Which messages can be replayed safely, and which need manual review?
How will provider operations be reconciled if Azure processing pauses?
How will customers, sellers, finance, and support see accurate status during recovery?
Which reports can lag, and which operational controls need faster restoration?

Recovery plans should be tested with realistic cases: provider outage, queue backlog, failed deployment, corrupted integration payload, missing settlement file, delayed payout batch, or region-level dependency outage.

Example: Provider Outage During Capture

A payment provider outage delays capture after fulfillment. A weak design keeps retrying aggressively, fills logs with sensitive provider responses, and leaves support unsure whether customers were charged.

A stronger design marks affected capture workflows as pending provider recovery, applies safe retry and circuit-breaker behavior, alerts owners, keeps customer communication separate from provider internals, records every attempted provider call with correlation, and prevents duplicate capture. When the provider recovers, the system resumes workflows and reconciliation compares provider records with internal payment and ledger records.

Cost Controls

Finance engines can generate high-volume logs, workflow executions, messages, provider calls, and analytics data. Cost control should not mean cutting the telemetry needed for audit and recovery. Instead, classify telemetry by purpose:

Keep audit-critical records durable and protected.
Keep operational metrics and alerts actionable.
Sample or aggregate low-value diagnostic noise where safe.
Use retention policies intentionally.
Watch high-volume retries, dead-letter growth, and chatty provider polling.

Common Mistakes

Treating compliance as documentation separate from design.
Logging too little for investigation or too much sensitive data.
Designing disaster recovery for compute but not financial state and workflow progress.
Giving support or operations broad data access when a narrower investigation view would work.
Protecting secrets but forgetting provider configuration, routing rules, and administrative actions.
Alerting on infrastructure health only while missing reconciliation, payout, or ledger risk.

If Someone Asks You

How do you secure and operate a commerce finance engine?

Use least privilege, managed identities, Key Vault, private access where appropriate, encryption, data minimization, audit trails, and role-specific views. Observe APIs, workflows, queues, providers, ledger writes, settlement, and reconciliation. Treat financial records and workflow state as disaster recovery concerns, and test recovery paths with provider outages, replay, duplicate prevention, and reconciliation exceptions.

What You Should Remember

Finance operations need least privilege, audit evidence, clear telemetry, and tested recovery paths.

Questions to Check Understanding

Which finance records need strong protection and retention?
What telemetry helps detect reconciliation risk?
Why is workflow state part of disaster recovery?