CH14-06: AI and Agentic Architecture Scenario

Goal

Apply the Chapter 14 model to an enterprise AI assistant or agentic workflow.

This scenario brings the chapter together. The goal is not to show the most advanced possible agent. The goal is to show how a solution designer can explain a complete Azure AI architecture with controlled data access, model use, tools, identity, safety, evaluation, monitoring, cost, and ownership.

Scenario

A large enterprise wants an internal support assistant. Employees should be able to ask questions about HR, IT, facilities, and internal policy. The assistant should answer from approved private knowledge sources and create support tickets through a controlled API.

The organization has several constraints:

• Some documents are public to all employees, but some are limited to HR, finance, or managers.

• The assistant must show source references or record which sources were used.

• Ticket creation must validate category, urgency, user identity, and required fields.

• Sensitive actions, such as payroll or legal requests, must not be automated without human review.

• The platform team wants shared evaluation, monitoring, governance, and cost visibility.

• Product teams want enough flexibility to improve prompts and add approved tools over time.

The durable architecture idea is:

A strong AI architecture explains what the system can see, what it can call, how it is authorized, how it is evaluated, and how it is operated.

Target Architecture

One reasonable Azure design looks like this:

flowchart LR
    Employee["Employee<br/><small>Microsoft Entra ID</small>"]
    App["Assistant App<br/><small>Session, prompt, orchestration</small>"]
    Foundry["Azure AI Foundry / Foundry Project<br/><small>Models, agents, evaluation, tracing</small>"]
    OpenAI["Azure OpenAI<br/><small>Chat or reasoning model, embeddings</small>"]
    Search["Azure AI Search<br/><small>Authorized knowledge retrieval</small>"]
    Tool["Tool Layer<br/><small>Validate and authorize actions</small>"]
    API["Ticketing API<br/><small>Create approved support tickets</small>"]
    KeyVault["Key Vault / Managed Identity<br/><small>Secrets and workload access</small>"]
    Ops["Monitoring, Evaluation, Safety, Audit<br/><small>Quality, cost, traces, incidents</small>"]

    Employee --> App
    App --> Foundry
    Foundry --> OpenAI
    App --> Search
    App --> Tool
    Tool --> API
    App --> KeyVault
    Tool --> KeyVault
    App --> Ops
    Foundry --> Ops
    Search --> Ops
    Tool --> Ops

This architecture separates responsibilities:

Request Flow

For a policy question, the flow is:

Employee asks a question
  -> assistant authenticates the user
  -> orchestration layer classifies the intent as question answering
  -> retrieval query applies permission filters
  -> Azure AI Search returns relevant policy chunks and source metadata
  -> model generates an answer from grounded context
  -> assistant returns answer with source references or a clear limitation
  -> telemetry records request, retrieval, model call, response, and feedback

For ticket creation, the flow is:

Employee asks to create a ticket
  -> assistant gathers missing fields
  -> model proposes a tool call
  -> tool layer validates parameters and checks authorization
  -> sensitive categories require human approval or redirect
  -> ticketing API is called with controlled workload identity
  -> action result is logged and returned to the employee

Notice the boundary: the model may suggest a tool call, but the tool layer decides whether the call is valid and allowed.

Key Design Decisions

Retrieval-Only or Tool-Calling

Start with the risk level. If the assistant only answers policy questions, retrieval-only may be enough. If the assistant creates tickets, updates records, or starts workflows, tool calling adds value but also adds authorization, validation, audit, and incident-response requirements.

For this scenario, a staged approach is sensible:

1. Launch retrieval-only Q&A for approved policy sources.

2. Add ticket creation for low-risk categories with strict validation.

3. Add approval workflows for sensitive categories.

4. Consider more agentic multi-step workflows only after evaluation and operations are mature.

One Broad Knowledge Base or Segmented Sources

A single broad index is simpler, but it can make authorization harder. Segmented sources, permission metadata, document-level security trimming, or query-time filters help preserve access boundaries.

The design should make it easy to answer:

• Which sources can this user access?

• Which source was used for this answer?

• How is stale content removed or re-indexed?

• How are restricted documents prevented from appearing in context?

Application Access or Delegated Access

Some downstream calls may use application permissions through a managed identity. Others may need delegated user context. The more sensitive the data or action, the more carefully the team should preserve user-specific authorization.

For ticket creation, application access can work if the tool layer enforces user permissions and logs the user behind each action. For reading restricted employee records, delegated access or strict downstream authorization may be more appropriate.

Fully Automated or Human-Approved Actions

Not every action deserves the same autonomy. Creating a basic IT helpdesk ticket can be automated. Requesting payroll changes, legal review, or account access should require approval or handoff.

Approval is not a failure of agentic design. It is part of controlled automation.

Safety, Evaluation, and Monitoring

Before release, the team should create evaluation cases for:

• Common policy questions.

• Questions with no approved answer.

• Role-specific access differences.

• Prompt-injection attempts in user input and retrieved content.

• Ticket requests with missing or invalid fields.

• Sensitive categories that require refusal, approval, or handoff.

• Regression cases for prompt, model, retrieval, and tool changes.

After release, monitoring should track:

• Request volume and adoption by feature.

• Latency, errors, throttling, and dependency failures.

• Token usage and model cost by workflow.

• Retrieval quality, citation coverage, and no-answer rates.

• Tool-call attempts, approvals, denials, and failures.

• Safety events and policy violations.

• User feedback and support escalations.

This is where Foundry observability, Azure Monitor, Application Insights, evaluation results, and application logs become part of the architecture rather than afterthoughts.

Common Mistakes

• Letting the assistant access all data because the application can. Retrieval must preserve user and source boundaries.

• Giving tools broad permissions or no approval boundary. Tools should be narrow, validated, authorized, and auditable.

• Launching without evaluation, monitoring, owner assignment, or cost controls. AI behavior changes under real usage.

• Treating citations as decoration. Source visibility is part of trust, troubleshooting, and auditability.

• Moving to agentic automation before the retrieval-only flow is reliable. Autonomy should grow with evidence and operational maturity.

If Someone Asks You

I would design the assistant as a controlled workflow: authenticate the employee, retrieve only authorized knowledge through Azure AI Search, call Azure OpenAI through an Azure-managed model platform, expose ticket creation through a narrow validated tool layer, protect credentials with managed identity and Key Vault, and operate the system with evaluation, safety checks, tracing, monitoring, budgets, and clear ownership.

What to Remember

Walk from user intent to data access, model call, tool action, safety, evaluation, monitoring, and ownership.

Questions to Check Understanding

1. Which parts of the scenario require authorization checks?

2. Which tool actions need approval or audit evidence?

3. How does the design detect unsafe, wrong, or expensive behavior?