CH14-02: Model and Agent Platform with Azure AI Foundry and Azure OpenAI

Goal

Explain where Azure AI Foundry and Azure OpenAI fit in an AI or agentic architecture at a high level.

The previous section framed AI architecture as a controlled workflow around model capability. This section places Azure platform services into that model. The goal is not to memorize every current product name. The goal is to understand which responsibility each platform surface helps with.

Microsoft's current documentation increasingly uses Microsoft Foundry for the unified platform experience. This chapter keeps the planned book title language, Azure AI Foundry, because it is the working term used in the book plan. When you see Foundry in current Azure documentation, think of the platform layer for building, managing, evaluating, deploying, and operating AI applications, models, agents, and tools.

The Short Version

Model and agent platforms provide AI capability, but the application architecture still controls data, tools, identity, safety, and operations.

Azure OpenAI gives applications access to supported OpenAI model capabilities through Azure-managed endpoints and Azure controls. Foundry provides a broader platform experience for projects, models, agents, tools, evaluation, observability, deployment, governance, and lifecycle management.

Those services are important, but they do not erase the rest of the architecture. A production AI application still needs orchestration, authorization, retrieval design, tool boundaries, monitoring, cost controls, and operational ownership.

The Platform Layers

Use this simple stack to place the services:

flowchart TB
    Users["Users and Channels<br/><small>Web app, Teams, API, workflow</small>"]
    App["Application or Agent Orchestration<br/><small>Prompts, state, routing, tools</small>"]
    Foundry["Azure AI Foundry / Microsoft Foundry<br/><small>Projects, models, agents, evaluation, governance</small>"]
    OpenAI["Azure OpenAI Model Endpoint<br/><small>Language, reasoning, multimodal, embeddings</small>"]
    Data["Data and Tool Integrations<br/><small>Search, vector indexes, APIs, storage, systems</small>"]
    Ops["Operations<br/><small>Monitoring, tracing, cost, safety, incidents</small>"]

    Users --> App
    App --> Foundry
    Foundry --> OpenAI
    App --> Data
    Foundry --> Ops
    App --> Ops

This is a teaching model, not a deployment prescription. Some applications call model endpoints directly from code. Some use Foundry project clients, agents, tools, and managed platform features. Some combine custom orchestration with platform evaluation and monitoring. The right answer depends on the team's required control, lifecycle needs, and operating model.

Azure OpenAI's Role

Azure OpenAI is the model access layer for supported OpenAI models in Azure. It lets teams use model capabilities through Azure-managed resources, endpoints, authentication options, regional availability, quotas, deployment choices, and monitoring integration.

From an architecture perspective, Azure OpenAI helps answer questions such as:

Avoid choosing a model only because it is the most capable one available. A cheaper, faster, smaller model may be better for classification or summarization. A more capable reasoning model may be justified for complex planning or tool use. The architecture decision should match the task, risk, latency expectation, and cost envelope.

Foundry's Role

Foundry is the broader platform layer for AI application and agent work. Current Microsoft documentation describes Foundry as a unified platform for models, agents, tools, projects, evaluation, observability, governance, and enterprise operations.

For a solution designer, Foundry is useful because AI delivery is rarely just one endpoint call. Teams need a place to organize projects, work with models, build agents, manage tools, evaluate behavior, trace execution, monitor production, and apply governance.

That makes Foundry relevant to questions such as:

• How do teams discover and compare model options?

• How are AI projects organized and governed?

• How are agents, tools, and model endpoints managed?

• How are evaluations run before release?

• How are traces, quality signals, safety signals, and usage monitored after release?

• How do platform teams provide reusable AI capabilities without every product team inventing a separate operating model?

The exact portal and API names may evolve. The durable architecture idea is more stable: use a platform layer to make model-powered applications governable, repeatable, observable, and manageable across teams.

Orchestration Is Still Architecture

The orchestration layer decides how the application uses model capability. It may be custom application code, a framework, a managed agent service, or a combination.

This layer usually owns:

• Prompt construction and system instructions.

• Conversation state or task state.

• Retrieval calls and context selection.

• Tool definitions and tool-call validation.

• Error handling and retries.

• Approval steps for risky actions.

• Logging and trace correlation.

• Fallback behavior when the model, retrieval source, or tool call fails.

The model endpoint supplies capability. The orchestration layer turns capability into a controlled workflow.

Example: Enterprise Support Assistant

An enterprise support assistant might use Azure OpenAI for response generation and embeddings, Azure AI Search for retrieval, Foundry for project organization and evaluation, and custom application code for identity-aware orchestration.

The design might look like this:

Employee channel
  -> support assistant application
  -> Foundry project for model and agent lifecycle
  -> Azure OpenAI model deployment
  -> Azure AI Search for approved knowledge retrieval
  -> ticketing API through a controlled tool layer
  -> Application Insights, evaluation, feedback, and cost tracking

In this design, Azure OpenAI does not decide which employee can see which document. Foundry does not automatically make every tool safe. The application and platform teams still design authorization, retrieval filters, tool validation, monitoring, and release controls.

Common Mistakes

• Treating model selection as the whole architecture. Model choice matters, but it is only one decision among data access, orchestration, safety, operations, and cost.

• Ignoring region, quota, latency, and lifecycle. A model that works in a demo may not fit production availability, scale, or compliance requirements.

• Letting every team build disconnected AI apps. Without shared evaluation, monitoring, policy, and cost patterns, AI adoption becomes difficult to govern.

• Confusing the platform with the product experience. Foundry can help manage AI assets, but the team still needs to design the user workflow, authorization, data boundaries, and operational model.

• Using the most capable model for every step. Different tasks may need different model sizes, latency profiles, and cost trade-offs.

If Someone Asks You

Azure OpenAI provides model access through Azure-managed endpoints and controls. Azure AI Foundry, now commonly documented as Microsoft Foundry, is the broader platform for building, managing, evaluating, deploying, and operating AI applications and agents. The architecture still has to design orchestration, data access, tools, identity, safety, monitoring, and cost.

What to Remember

The model platform supplies capability; the architecture controls how capability is used.

Questions to Check Understanding

1. What does model access provide, and what does it not provide by itself?

2. What lifecycle concerns surround AI applications after the first demo works?

3. Why is orchestration separate from the model endpoint?