Retrieval Is the Whole Game: Building AI Apps on Azure’s Data Stack in 2026

# Retrieval Is the Whole Game: Building AI Apps on Azure's Data Stack in 2026 Most AI projects that stall don't stall on the model. They stall about six weeks in, when someone asks why the assistant confidently cited a policy document that was superseded in 2023, and the team realizes the answer lives four layers down in a chunking script nobody owns. The Azure AI story over the last year has been a slow admission of exactly that. The interesting announcements haven't been about bigger models — they've been about the plumbing between your data and the model. If you're building on Azure right now, that plumbing is where the platform has changed most, and where your architecture decisions actually matter. Here's what the stack looks like today, what's genuinely production-ready, and where the sharp edges still are. ## First, the naming Azure AI Foundry became Microsoft Foundry at Ignite 2025. It's the same platform, extended: model catalog, agent hosting, evaluation, observability, and the SDK surface for all of it. You'll still see "Azure AI Foundry" in older docs, blog posts, and half the Stack Overflow answers you'll hit. Assume they're describing the same product unless something contradicts that. More consequentially, Azure AI Search now fronts as **Foundry IQ**. This one isn't just a rename — Foundry IQ is a new abstraction layer built on top of Azure AI Search, and understanding the relationship between them is most of what you need to know about retrieval on Azure in 2026. ## The four layers you're actually assembling Strip the branding and an Azure AI application in 2026 is four things: 1. **Models** — Foundry Models, a catalog rather than a single endpoint. GPT-5.2 and Codex Max went GA in the December–January wave; Claude reached general availability in Foundry in June 2026, hosted on Azure with the Messages API, prompt caching, and extended thinking. Open-weight options (Llama, Qwen, Mistral) are available both serverless and on managed compute. 2. **Retrieval** — Foundry IQ knowledge bases, or your own pipeline on Azure AI Search indexes. 3. **Data** — Cosmos DB for operational state and vectors, Fabric/OneLake for analytics, plus whatever's already in SharePoint and Blob Storage. 4. **Orchestration** — Foundry Agent Service, or Microsoft Agent Framework if you want to hold the reins yourself. The layer that most teams over-build is #2 and most teams under-build is #3. Let's take them in that order. ## Retrieval: stop rebuilding the pipeline The pattern Microsoft is targeting with Foundry IQ is one you've probably lived through. Every new agent gets its own retrieval stack — data connections, chunking logic, embedding model choice, index schema, permission filtering, query rewriting. Five agents later you have five subtly different pipelines, five sets of stale embeddings, and no consistent way to answer "why did it say that?" Foundry IQ inverts this. You define a **knowledge base** around a topic — employee policies, product docs, incident history — and any number of agents connect to it. Behind the knowledge base sit **knowledge sources**: Blob Storage, OneLake, SharePoint, existing Azure AI Search indexes, the public web via Bing, and MCP servers. For indexed sources, Foundry IQ runs the full ingestion pipeline itself — chunking, vectorization, hybrid index preparation — and keeps it synced. The genuinely new part is what happens at query time. Single-shot RAG (one query, one index, one hop) falls apart on ambiguous or multi-part questions. Foundry IQ's agentic retrieval engine treats retrieval as a reasoning task: - The incoming request is decomposed into subqueries - The engine selects which knowledge sources each subquery should hit - Subqueries fan out across local indexes and remote sources in parallel - Results are scored and reranked with the semantic ranker - If relevance is poor, it iterates — provided you've configured sufficient retrieval reasoning effort Microsoft's own numbers put the relevance improvement at around 36% over conventional single-shot RAG. Treat vendor benchmarks as directional, not gospel, but the architectural argument holds independent of the number: multi-hop questions need multi-hop retrieval. Permissions are handled at the retrieval layer rather than bolted on. Document-level access control flows through, and Purview sensitivity labels are respected through indexing and retrieval. If you've ever hand-rolled security trimming into an index filter, you know how much that's worth. ### When not to use it Foundry IQ is a good default. It is not always the right answer. Skip it if you have hard latency budgets — agentic retrieval means multiple model calls before you've generated a single token of the actual answer. Skip it if your retrieval is genuinely simple: one index, well-formed queries, known-good chunking. And skip it if you need chunking strategies it doesn't expose, because you're trading control for convenience by design. The GA story also deserves care. Parts of Foundry IQ are generally available through the Azure AI Search REST API version `2026-04-01`. Several of the more attractive capabilities — answer synthesis, higher reasoning effort levels, some source kinds — still require `2026-05-01-preview`. The portal experiences are preview across the board. Pin your API version explicitly and know which side of that line each feature you depend on sits. ## Operational data: Cosmos DB earned its place The two-database pattern — operational store plus a dedicated vector database — was the default for a long while, and it's now much harder to justify on Azure. Cosmos DB stores embeddings as a property alongside the rest of the document. The product record carries name, price, category, stock status, description, and vector in one item. Vector indexing uses DiskANN for approximate nearest-neighbor search at scale, and semantic reranking arrived in preview at Build 2026. You get hybrid search — vector plus filters plus the ordinary query predicates you were already writing — in one round trip, against data that's already consistent because it never left. Three additions from Build 2026 matter more for daily work than the headline features: - **The Linux emulator went GA.** Local build-test-validate on Linux, macOS, and Windows with no cloud dependency. This is the unglamorous fix that makes CI pipelines and onboarding stop being painful. - **An agent memory toolkit** standardizes persistent agent memory across Cosmos DB, Azure Durable Functions, and Foundry models — worth reading before you design your own conversation store. - **`langchain-azure-cosmosdb`** consolidates vector search, chat memory, and semantic caching behind LangChain and LangGraph interfaces, if that's your orchestration layer. A rough heuristic: if the vectors describe entities your application already stores and mutates, keep them in Cosmos DB. If they describe a document corpus that's ingested and rarely changes, put them behind Azure AI Search and let Foundry IQ manage the pipeline. ## Analytics: mirroring instead of ETL Fabric's contribution to AI apps is mostly about not moving data twice. Mirror Cosmos DB into Fabric and operational data lands in OneLake in near-real time with no ETL job to own, no schedule to babysit, and no drift between what the app sees and what the analytics layer sees. Fabric SQL and Cosmos DB in Fabric write to OneLake by default. The payoff for AI work is that OneLake is a first-class Foundry IQ knowledge source. Your curated lakehouse tables become groundable knowledge without a separate export path. If you're already invested in Fabric, this is the shortest route from governed enterprise data to a grounded agent. ## Agents and memory Foundry Agent Service has been converging on the OpenAI Responses protocol, which is a meaningful simplification if you've been maintaining separate code paths. Build 2026 added hosted runtimes, Toolboxes (public preview) for connecting agents to external tools, and expanded memory — procedural, user, and session — in the Foundry Agent Service memory store, which handles extraction, consolidation, and retrieval across sessions automatically. Agents can also publish directly into Microsoft Teams and Microsoft 365 Copilot. If your users live in those surfaces, that distribution path removes a lot of frontend work. For tool connectivity there's a cloud-hosted Foundry MCP server at `mcp.ai.azure.com` with Entra auth, connectable from VS Code, Visual Studio, or the Foundry portal — no local process to manage. ## The SDK situation, plainly This is the part that will bite you in a code review, so it's worth being blunt about. `azure-ai-projects` v2 is now the canonical SDK for everything Foundry. Agents, inference, evaluations, and memory are unified in one package — the separate `azure-ai-agents` dependency is gone, and `openai` plus `azure-identity` come bundled as direct dependencies. Active development happens on the beta line. Pin your versions. Separately: the Azure Machine Learning SDK v1 reached end of support on June 30, 2026. The CLI v1 extension went a year earlier. If you still have v1-based training pipelines running, they're unsupported now, and the migration to v2's YAML-first job definitions is not a weekend task. Budget for it properly. ## How I'd start a new project today 1. **Inventory before you architect.** Where does the knowledge actually live, and who's allowed to see it? Permission modeling is the thing that kills pilots at the production gate, not retrieval quality. 2. **Start with one Foundry IQ knowledge base** and the simplest retrieval reasoning effort that gets acceptable answers. Turn it up only against a measured baseline. 3. **Keep operational vectors in Cosmos DB**, document vectors behind Azure AI Search. Don't run two vector stores because two blog posts told you to. 4. **Instrument evaluation from day one.** Foundry's evaluation tooling can convert agent traces into evaluation datasets — build that loop before you have opinions to defend, not after. 5. **Write down which API versions you depend on** and which of them are preview. Then set a calendar reminder to recheck, because this surface is moving fast enough that a six-month-old architecture decision deserves a second look. The platform has gotten substantially better at the boring parts — ingestion, permissions, memory, evaluation. That's genuinely good news, because the boring parts were always where the projects died. What hasn't changed is that grounding quality is a data problem wearing an AI costume. No amount of retrieval reasoning effort will fix a knowledge base full of documents nobody has curated since 2021.