Identity, Authorization, Safety, and Evaluation

Goal

Retrieval and tools make AI systems useful because they connect models to enterprise context and business action. They also make identity, authorization, safety, and evaluation unavoidable.

The durable idea is:

AI safety is a system property built from access control, policy, evaluation, and accountability.

The Short Version

A weak safety model says:

Add a filter after the model response.

A stronger safety model designs the whole workflow:

User and workload identity
  -> authorization for data and tools
  -> safety policy and evaluation gates
  -> response, action, refusal, or escalation
  -> audit and improvement loop

Safety is not one component. It is the combined behavior of identity, permissions, input handling, retrieval controls, tool restrictions, output rules, evaluation, monitoring, escalation, and ownership.

Identity in AI Workflows

An AI workflow often has more than one identity:

Identity	Architecture Question
User identity	Who is asking, and what are they allowed to see or request?
Workload identity	Which application, agent, or service is calling systems on behalf of the workflow?
Tool identity	Which credentials or delegated permissions are used for each tool call?
Reviewer identity	Who approved an escalated or high-impact action?
Data owner identity	Who owns the source, policy, or dataset used by the answer?

The system should not bypass data permissions by using a broad service account. If a user cannot see another customer's order in the application, the AI workflow should not retrieve it just because the index contains it.

Authorization for Data and Tools

Authorization needs to apply at two levels:

Data access: which records, documents, fields, tenants, regions, or sources may be retrieved.
Tool use: which actions may be performed, under which conditions, and with which approval.

For example, a support assistant may be allowed to retrieve the authenticated customer's order status and policy documents. It may be allowed to create a support case. It may not be allowed to issue a high-value refund, change a delivery address, or expose internal fraud notes without review.

This distinction matters because an AI system can combine information quickly. Even read-only retrieval may reveal sensitive context if authorization is weak.

Safety Controls

Safety controls should be designed around the risk of the workflow.

Useful controls include:

Input handling for malicious prompts, hidden instructions, sensitive data, and unsupported requests.
Retrieval restrictions for private, regulated, or stale sources.
Output constraints for format, citations, uncertainty, and refusal behavior.
Content risk checks for harmful, abusive, illegal, or policy-violating output.
Tool restrictions for allowed actions, thresholds, and approval requirements.
Human escalation for uncertainty, high impact, exceptions, and policy boundaries.
Audit records for prompts, context, tool calls, decisions, and reviewers.

The goal is not to block everything. The goal is to make acceptable behavior reliable and risky behavior visible.

Evaluation as a Release Gate

Evaluation is how teams learn whether the workflow is good enough to change or release.

Evaluation should cover:

Evaluation Area	Example Question
Answer quality	Does the response solve the user's task clearly and correctly?
Groundedness	Is the answer supported by retrieved sources or records?
Retrieval relevance	Did the workflow find the right evidence?
Authorization	Did the workflow avoid data the user should not access?
Safety and policy	Did it refuse, constrain, or escalate when required?
Tool behavior	Were tool calls valid, scoped, idempotent, and auditable?
Latency and reliability	Did the workflow respond within the expected operating window?
Cost	Is the quality worth the model, retrieval, tool, and evaluation cost?

Evaluation sets should include real edge cases, not only happy-path demos. Include ambiguous questions, stale policy examples, conflicting records, unauthorized requests, high-value actions, adversarial prompts, and cases where escalation is the correct outcome.

Example: Refund Assistant

A refund assistant can summarize policy and draft a recommendation. It can check order state, shipment status, return history, and policy documents. But it cannot issue a refund above a threshold without human approval.

The workflow could be:

Refund request
  -> authenticate user and retrieve authorized order
  -> retrieve current refund policy
  -> classify risk and check thresholds
  -> draft recommendation with evidence
  -> approve automatically, refuse, or escalate
  -> log decision, evidence, and reviewer

The human reviewer should see why the assistant recommended escalation: policy section, order state, refund amount, risk signals, missing information, and previous customer history. That turns escalation into an operating workflow, not a vague handoff.

Common Mistakes

Giving agents broad service credentials.
Evaluating only demo prompts before production.
Treating safety as one filter instead of a workflow design.
Logging sensitive prompts or outputs without governance.
Checking user login but not data-level or action-level authorization.

If Someone Asks You

What makes an AI workflow safe enough to operate?

A good answer is:

It needs identity, authorization, policy controls, evaluation, escalation, and audit around the whole workflow. The system should control which data can be retrieved, which tools can be called, which actions require approval, how unsafe or unsupported requests are handled, and how changes are evaluated before release. Safety is not one filter; it is the operating design.

What You Should Remember

Safe AI design controls access, action, evaluation, and escalation.

Questions to Check Understanding

Why must authorization apply to retrieval and tools?
What should evaluation cover besides answer fluency?
When should an AI workflow escalate to a human?