CH14-04: Identity, Authorization, Safety, and Evaluation

Goal

Explain the control model around AI workflows: who is asking, what the system may access, what the system may do, and how behavior is evaluated.

The previous section explained retrieval and tools. This section adds the controls that decide whether retrieval and tool use are safe enough for production.

The Short Version

AI systems need explicit permission, safety, and evaluation boundaries because model-powered workflows can reveal data or take action.

The model does not replace the security model. The application must still know who the user is, which workload identity is calling downstream systems, what data can be retrieved, which tools can run, which actions require approval, and how behavior is tested.

Evaluation belongs in this same conversation. A system can pass authentication and still produce unsafe, inaccurate, ungrounded, or policy-violating behavior. Security controls decide what the system may do. Evaluation checks whether the system actually behaves within those boundaries.

The Control Path

Use this path when thinking about AI security and safety:

flowchart LR
    User["User Identity<br/><small>Who is asking?</small>"]
    Workload["Workload Identity<br/><small>Which app is calling?</small>"]
    Authz["Authorization<br/><small>What is allowed?</small>"]
    Data["Data and Tools<br/><small>What can be retrieved or called?</small>"]
    Safety["Safety Controls<br/><small>What must be blocked or reviewed?</small>"]
    Eval["Evaluation and Audit<br/><small>How do we prove behavior?</small>"]

    User --> Authz
    Workload --> Authz
    Authz --> Data
    Data --> Safety
    Safety --> Eval

This path keeps the architecture honest. Before discussing model prompts or agent planning, the team should be able to explain identity, permissions, safety boundaries, and evidence.

User Identity and Workload Identity

An AI application usually has at least two identity questions:

In Azure, user identity commonly connects to Microsoft Entra ID. Workload identity often uses managed identities so applications can access Azure services without storing credentials in code. Secrets, keys, and certificates that are still needed should be protected with services such as Azure Key Vault.

The design must also decide whether downstream access is delegated or application-level:

• Delegated access uses the user's permissions when calling a downstream system. This can preserve user-specific boundaries but may add complexity.

• Application access uses the application's own permissions. This can simplify integration but increases the need for explicit authorization checks inside the application.

Neither pattern is universally correct. The key is to avoid accidental broad access. The user, application, retrieval source, and tool layer must agree on what is allowed.

Authorization for Retrieval and Tools

Authorization must happen at more than the front door.

It is not enough to check that a user can open the assistant. The system also needs to check:

• Can this user retrieve this document or record?

• Can this user ask the assistant to perform this action?

• Can this workload call this downstream API?

• Are the tool parameters valid for the user's role and context?

• Does the action require human approval?

For retrieval, authorization may use document-level permissions, source-level permissions, metadata filters, security trimming, or separate indexes for different audiences. For tools, authorization may be enforced in the application, API gateway, downstream system, or a combination.

The model should not be trusted to enforce these rules from natural language instructions alone. Prompts can guide behavior, but authorization belongs in system controls and API enforcement.

Safety Controls

Safety controls define what the AI system should avoid, block, flag, or escalate.

Common controls include:

Safety is not one checkbox. It is a set of controls placed around the workflow, especially where data crosses boundaries or actions affect real systems.

Evaluation as Architecture

Evaluation is how the team checks whether the AI application behaves well enough to release and keep operating.

In a traditional application, tests often assert exact output. In an AI application, evaluation may measure quality, groundedness, relevance, tool-call accuracy, safety, refusal behavior, and task completion. Foundry observability and evaluation guidance describes evaluating during model selection, pre-production testing, and post-production monitoring.

A useful evaluation set includes examples such as:

• Normal user questions with expected answer characteristics.

• Questions that require retrieval from approved sources.

• Questions where the correct answer is "I do not know" or "I cannot help with that action."

• Prompt-injection attempts hidden in user input or retrieved content.

• Tool-call requests with missing, invalid, or risky parameters.

• Role-based access cases where one user can see content and another cannot.

• Regression cases for prompts, models, retrieval indexes, and tool schemas.

Evaluation gives the team release evidence. It also creates a feedback loop for improving prompts, retrieval, tool design, and safety controls.

Example: Customer Support Refund Flow

Imagine a support assistant that summarizes customer cases and helps support agents start refund requests.

The control model might be:

Support agent signs in
  -> app confirms the agent can view the customer case
  -> retrieval returns only authorized case notes and policy documents
  -> model summarizes with citations and uncertainty when needed
  -> refund tool validates amount, reason, customer status, and policy limits
  -> high-value refunds require supervisor approval
  -> all retrieval, tool requests, approvals, and results are logged
  -> evaluation tests summary accuracy, authorization, refusal, and tool behavior

The model helps with language and workflow assistance. It does not become the source of permission. The architecture controls what the assistant can see and do.

Common Mistakes

• Assuming the model can inherit broad application access safely. Broad access may create data leakage or unauthorized action risk.

• Checking permissions before retrieval but not before tool calls. Tool actions can be more sensitive than answers.

• Relying on prompts for authorization. Prompts can express rules, but access checks must be enforced by application and platform controls.

• Launching without evaluation data. A demo transcript is not a production evaluation suite.

• Treating safety as only content filtering. Safety also includes authorization, action limits, approvals, audit, and incident response.

If Someone Asks You

The most important AI security controls are identity, authorization, retrieval permissions, tool-call enforcement, safety boundaries, approval for risky actions, audit logging, and evaluation. The model can assist the workflow, but the architecture must control what the workflow can see and do.

What to Remember

Authorize what the AI can see and do, then evaluate whether it behaves within those boundaries.

Questions to Check Understanding

1. Why must retrieval be authorization-aware?

2. Which tool calls should require extra approval or stronger audit evidence?

3. What should an evaluation set test before an AI workflow is released?