CH20-07: AI and Agentic Services
Goal
AI services help teams build applications that use models, retrieval, search, safety checks, evaluation, and controlled tool use. The mistake to avoid is treating the model as the whole architecture. A production AI system also needs data boundaries, grounding, identity, observability, cost control, and safe rollout.
The durable idea is:
AI service selection should start with the user task, data boundary, safety controls, and operating model.
The Short Version
Start with the AI responsibility:
| AI Need | Common Azure Direction |
|---|---|
| Generative model access | Azure OpenAI Service. |
| Retrieval over enterprise or product content | Azure AI Search. |
| AI application development, evaluation, and lifecycle | Azure AI Foundry. |
| Custom ML model lifecycle | Azure Machine Learning. |
| Content moderation and safety checks | Azure AI Content Safety. |
| Application data for retrieval | Azure SQL, Cosmos DB, Storage, or other governed data sources. |
| AI workload hosting | Functions, Container Apps, App Service, or AKS depending on workload shape. |
Service Reference
| Service or Capability | Purpose | Use When | Avoid When | Common Prompt |
|---|---|---|---|---|
| Azure OpenAI Service | Azure-hosted access to OpenAI models for generative AI scenarios. | You need language generation, summarization, extraction, classification, or assistant-style model capabilities through Azure. | You expect model access alone to solve grounding, authorization, evaluation, safety, and cost. | What model task are we enabling? |
| Azure AI Search | Search and retrieval over indexed content. | You need keyword, vector, semantic, or hybrid retrieval over approved enterprise, product, support, or policy content. | You have not planned document ownership, freshness, permissions, or citation behavior. | What trusted content grounds the answer? |
| Azure AI Foundry | Platform for building, evaluating, deploying, and managing AI applications and models. | Teams need AI application lifecycle support, model catalog access, evaluation, prompt flow concepts, and operational governance. | A simple application only needs direct service calls and local lifecycle practices. | How will AI changes be built and evaluated? |
| Azure Machine Learning | ML lifecycle platform for training, managing, deploying, and operating custom models. | You need custom model training, experimentation, registries, pipelines, endpoints, and ML operations. | A managed AI service or simple model endpoint is enough. | Do we need custom ML lifecycle control? |
| Azure AI Content Safety | Safety classification and moderation capabilities. | You need to detect or filter harmful user-generated or model-generated content. | You treat moderation as a substitute for full safety design, human review, or policy decisions. | What content risks need controls? |
| Azure SQL, Cosmos DB, Storage, and other data sources | Source systems for grounding and retrieval. | AI responses need current application, product, policy, support, or enterprise data. | The model is allowed to see data without consent, authorization, freshness, or minimization rules. | What data may the model use? |
| Functions, Container Apps, App Service, or AKS | Hosting for AI orchestration and serving APIs. | You need to run prompt orchestration, retrieval, tool calls, model gateways, or AI APIs. | You choose hosting before understanding latency, scale, tool permissions, and operations. | Where does the AI application run? |
Commonly Confused Responsibilities
| Comparison | Short Answer |
|---|---|
| Model access vs AI architecture | Model access generates or interprets text; AI architecture includes grounding, permissions, safety, evaluation, observability, and cost. |
| Azure OpenAI vs Azure AI Search | Azure OpenAI provides model capability; Azure AI Search helps retrieve trusted content for grounding. |
| Azure AI Foundry vs Azure Machine Learning | AI Foundry focuses on AI application and model lifecycle experiences; Azure Machine Learning focuses on custom ML lifecycle and operations. |
| Retrieval vs tool use | Retrieval gives the model information; tool use lets the system call controlled actions. |
| Content Safety vs responsible AI program | Content Safety helps detect harmful content; responsible AI also includes policy, review, testing, monitoring, and incident response. |
Architecture Patterns
AI services often appear in patterns such as:
Retrieval-augmented generation with Azure OpenAI, Azure AI Search, governed data sources, identity, and monitoring.
Support or seller assistant with controlled tool calls through Functions, Container Apps, or APIs.
Custom ML lifecycle with Azure Machine Learning and model endpoints.
AI application evaluation and governance with Azure AI Foundry.
Content moderation with Azure AI Content Safety and human review paths.
AI serving API hosted on App Service, Container Apps, Functions, or AKS depending on runtime needs.
Agentic Design Prompt
For agentic systems, ask five questions before naming services:
What can the model see?
What can the model decide?
What tools can it call?
Who approves sensitive actions?
What evidence is logged for review?
These questions keep the architecture grounded in permissions and operations rather than model excitement.
Cloud Comparison
AWS and GCP users will recognize equivalent responsibility families: foundation models, search or vector retrieval, ML lifecycle, content safety, application hosting, and monitoring. Use mappings as orientation only. The Azure design answer should still name data boundaries, identity, retrieval, tool control, evaluation, and cost ownership.
Common Mistakes
Treating the model as the whole architecture.
Ignoring retrieval quality, grounding, evaluation, and observability.
Giving agents broad tool permissions without safety boundaries.
Forgetting cost, rate limits, latency, and data access controls.
Indexing private content without permission filters.
Shipping prompt or model changes without evaluation and rollback.
If Someone Asks You
Which Azure services matter most for AI application architecture?
A good answer is:
Azure OpenAI provides generative model access, Azure AI Search supports retrieval over trusted content, Azure AI Foundry helps with AI application lifecycle and evaluation, Azure Machine Learning supports custom ML lifecycle, and Azure AI Content Safety helps with moderation checks. But the architecture also needs governed data sources, identity, authorization, hosting, monitoring, cost controls, and safe tool-use boundaries.
What You Should Remember
AI architecture is model access plus data grounding, safety, evaluation, permissions, and operations.
Questions to Check Understanding
Why is a language model not the whole AI architecture?
When does Azure AI Search appear in a retrieval design?
What makes agentic tool use risky without controls?