Observability, Audit, and Configuration Evidence

Goal

Operational readiness depends on evidence. Different questions need different evidence: behavior, request path, user impact, API activity, and resource configuration.

This section separates observability, audit trail, and configuration evidence so the tools do not blur together.

The Short Version

Different questions need different evidence.

Is it healthy?                 -> metrics, alarms, dashboards
What happened in this request? -> traces and logs
Who changed what?              -> audit trail
What configuration exists?     -> configuration history and rules

CloudWatch, X-Ray, CloudTrail, and AWS Config are service names for different evidence jobs.

Observability: Behavior And Impact

Observability helps teams understand how the workload behaves. In AWS, Amazon CloudWatch is the central service family for many monitoring signals: metrics, logs, alarms, dashboards, log queries, and cross-account visibility.

Useful observability includes:

Metrics for latency, errors, traffic, saturation, queue depth, throttles, and dependency health.
Logs with request IDs, customer-safe context, error details, and enough structure to query.
Traces that show how requests move through services and dependencies.
Dashboards that support real operational questions.
Alarms that have owners and expected actions.

AWS X-Ray and tracing concepts help when a request crosses multiple services. A single slow API call may involve an API entry point, container service, database, queue, Lambda function, and external dependency. Tracing helps the team see where time is spent and where failures occur.

Audit: Who Did What

Audit evidence answers a different question: who or what changed something?

AWS CloudTrail records AWS account activity as events, including actions by users, roles, AWS services, console activity, CLI calls, SDK calls, and API operations. That makes it useful for security review, operational investigation, governance, and compliance evidence.

For example, if an API slows down after a deployment and a security group changed at the same time, observability may show latency and errors. CloudTrail helps answer who changed the security group, when it changed, and which API action was used.

Do not confuse application logs with audit trails. They overlap during incidents, but they answer different questions.

Configuration Evidence: What Exists

AWS Config helps track resource configuration and evaluate resources against rules. It is useful when teams need to understand configuration history, detect drift from expected settings, or show compliance-style evidence.

Examples:

Which security groups allow public ingress?
Which buckets lack required encryption settings?
Which resources changed since last week?
Which accounts are missing required baseline controls?

Configuration evidence is especially useful in multi-account environments. It helps platform, security, and compliance teams reason about what exists without manually inspecting every account.

Example: Latency After A Change

An API latency incident starts after a deployment.

The team may need:

Question	Evidence
Are users affected?	CloudWatch metrics, alarms, dashboards, synthetic checks, real-user signals where available.
Which request path is slow?	Traces and structured application logs.
Did the deployment change behavior?	Pipeline history, release logs, application metrics, error logs.
Did someone change infrastructure?	CloudTrail events and infrastructure change history.
Did configuration drift from the baseline?	AWS Config resource history and rule evaluations.

No single signal answers everything. A strong operating model combines signals into an investigation path.

Common Mistakes

Collecting logs without useful fields, structure, retention, or access controls.
Confusing application observability with compliance evidence.
Creating alarms that no one owns.
Measuring infrastructure health while missing user impact.
Keeping evidence in separate accounts or tools with no incident-time access path.

If Someone Asks You

How do you design observability and auditability for an AWS workload?

A good answer is:

I separate evidence by question. CloudWatch metrics, logs, dashboards, and alarms help show workload health and behavior. Tracing helps follow requests across services. CloudTrail records account and API activity so we can see who changed what. AWS Config tracks resource configuration and compliance-style evidence. I also define retention, centralization, access control, correlation IDs, alarm owners, and incident runbooks.

What You Should Remember

Observability tells you how the workload behaves; audit evidence tells you what changed and who did it.

Questions to Check Your Understanding

Which evidence helps diagnose a slow request?
Which evidence helps investigate an API configuration change?
Why do alarms need ownership?