Azure Architecture Guide

CH21-12: AI Safety, Retrieval Quality, Evaluation, and Tool Use

Goal

Help readers connect AI safety, retrieval quality, evaluation, and controlled tool use concerns to Azure solution options, useful patterns, trade-offs, and decision prompts.

The Short Version

AI architecture must control model behavior, data grounding, tool permissions, evaluation, and human accountability.

An AI workload is not only a model call. It is a system that decides what data the model can see, what instructions and context it receives, what it is allowed to say, what tools it can call, how outputs are evaluated, how unsafe behavior is handled, and when a human must take over.

Concern

AI and agentic workloads must produce useful responses while protecting data, reducing unsafe behavior, grounding answers, evaluating quality, and controlling actions taken through tools.

Typical Symptoms

  • The team treats the model as the whole system.

  • Retrieval returns irrelevant, stale, or unauthorized content.

  • Prompts and outputs are not evaluated before changes ship.

  • Agents can call tools with broad permissions.

  • There is no human escalation path for uncertain or high-impact actions.

Azure Solution Options

  • Azure OpenAI Service for model access through Azure controls.

  • Azure AI Search for retrieval over indexed enterprise content.

  • Azure AI Content Safety for content risk detection where appropriate.

  • Azure Machine Learning or Azure AI Foundry capabilities where evaluation, orchestration, or lifecycle tooling fits the current design.

  • Microsoft Entra ID, managed identities, Key Vault, and RBAC for access control.

  • Azure Monitor and application telemetry for prompts, retrieval, tool calls, latency, errors, and cost signals.

Useful Patterns

  • Retrieval-augmented generation as an architectural pattern, explained carefully.

  • Gateway Routing or Gateway Offloading for central AI API controls where appropriate.

  • Human-in-the-loop for high-impact decisions.

  • Compensating Transaction for reversible tool actions where business processes allow it.

Trade-Offs

  • More grounding improves answer quality but increases indexing, freshness, access, and evaluation work.

  • More agent autonomy can improve workflow speed but increases safety and permission risk.

  • Stronger filtering can reduce unsafe output but may block legitimate content.

  • Detailed telemetry improves evaluation but must respect privacy and data handling rules.

Decision Prompt

What can the AI system see, what can it say, what can it do, and how do we prove it behaves acceptably?

Common Mistakes

  • Ignoring retrieval quality and blaming only the model.

  • Giving tools broad permissions instead of task-scoped access.

  • Shipping prompt or retrieval changes without evaluation.

  • Logging sensitive prompts or outputs without governance.

If Someone Asks You

How do you design safe AI and agentic workloads in Azure?

I design the AI system around control points. Azure OpenAI Service provides model access through Azure controls, Azure AI Search can support retrieval when content grounding is needed, Azure AI Content Safety can help detect content risk, and Azure AI Foundry or Azure Machine Learning may support lifecycle, evaluation, and orchestration needs. Around those services, I use Entra ID, managed identities, RBAC, Key Vault, telemetry, evaluation sets, tool authorization, human escalation, and privacy-aware logging so the system's behavior can be governed and improved.

What You Should Remember

Safe AI design controls grounding, permissions, evaluation, telemetry, and escalation.

Questions to Check Your Understanding

  1. Why is retrieval quality part of AI safety?

  2. Why should agent tools have narrow permissions?

  3. What should be evaluated before AI changes ship?