AI Agent Platform vs Data Platform: What Is Different, What Is Merging

A data platform is built so that nothing surprising happens. An agent platform is built so that surprising things can happen safely.

Most of this post is me unpacking that. The difference between an AI agent platform and a data platform comes down to what each one treats as normal. A data platform assumes the work is known in advance and has to be repeatable. An AI agent platform assumes the work is found out as it goes and has to be kept under control. In 2026 the two are sharing more and more infrastructure, so the line between them looks blurry. I’ll draw the line first, then show where it’s moving.

When I wrote about running Snowpipe in production and the medallion layout, the whole job was to make surprises boring: the same input, the same output, every night. Agents ask for nearly the opposite. Building infrastructure that supports both is what I’m most interested in right now, so I went looking at what the big vendors have shipped this year and how they see the split.

By the end you will be able to:

  • explain the difference between a data platform and an AI agent platform in one sentence each;
  • read the reference architecture that puts both platforms under one control plane;
  • compare the two platforms side by side across eight dimensions;
  • follow one concrete task, refunding late orders, through both, with a simulation;
  • see who is building what in 2026, and what is GA and what is open source at Microsoft;
  • know the main risks, a four-week pilot plan and where to build.

KEY POINTS IN FIVE LINES

  • A data platform moves, stores and governs data with work defined in advance. Correct means tested, repeatable and reconcilable.
  • An AI agent platform runs software that plans, calls tools and adapts. Correct means evaluated across many runs, plus checks that do not depend on the model.
  • They differ on goal, shape of work, core objects, meaning of “correct”, state, failure handling, governance, and identity and cost.
  • They are merging around a shared control plane: governance, context (semantic layers), memory and traces stored as data, and MCP as the front door.
  • Do not expect one engine for both. Expect agents to run on top of governed data platforms, with the platform enforcing the rules.

HOW TO READ THIS POST


Table of Contents

The one-picture answer: two engines, one control plane

A reference architecture for AI on data separates durable data operations from adaptive agent reasoning, and puts one shared control plane across both. The data plane (storage, SQL, pipelines, catalogue) stays deterministic. The agent plane (runtime, session state, planning) handles open-ended work. A tool and action plane between them exposes narrow, typed operations. The shared control plane applies identity, policy, approvals, audit, cost and evaluation to all of it, and an independent verification layer checks that the desired state was really reached.

Reference architecture with four stacked layers: applications and users, agent plane, tool and action plane, and data plane, with a verification layer on the left and a shared control plane on the right.

I drew this myself from the 2026 launches covered below. It isn’t any vendor’s diagram. Each layer has its own owner and its own job. These are logical responsibilities, not mandatory deployment boundaries: a prototype may implement several layers in one service, and the separation pays off as security, scale, ownership and operations grow.

Layer What it owns Examples of primitives
Applications and users Business interfaces, engineers, analysts UI, CLI, API, event triggers
Agent plane Reasoning, planning, tool choice, bounded delegation Agent runtime, session state, planner, task graph
Tool and action plane Narrow, scoped operations the agent may call MCP tools, typed APIs, job submission, ticketing
Data plane Durable data operations, contracts and lineage Object storage, Parquet and Iceberg, SQL, Spark, orchestrator, catalogue
Shared control plane Who may do what, with evidence Identity, policy, approvals, audit, cost, traces, evaluation
Verification layer Independent proof that the desired state was reached Data-quality checks, schema checks, reconciliation, SLA checks

Practical tip

If you remember one rule from this post, make it this one: the agent may propose, deterministic code decides and executes, and something other than the agent verifies.

AI agent platform vs data platform: the side-by-side comparison

A data platform is the infrastructure that ingests, stores, transforms, catalogues, governs and serves data. Its work is mostly defined in advance as pipelines, which are graphs of steps that run in a fixed order. A job succeeds or fails, and you can run it again to get the same answer.

An AI agent platform is the infrastructure that runs software agents: programs that take a goal, plan, choose tools, observe results and adapt. Its work is not fixed in advance. That flexibility is the point, and it is also the risk.

Dimension Data platform AI agent platform
Primary objective Reliable data processing and access: fresh, correct, available data A task completed safely, correctly and observably
Shape of the work Predefined graph (DAG) or job A loop: plan, act, observe, adapt
Core objects Assets, schemas, lineage, contracts Agents, tools, sessions, state, memory, evaluations
What “correct” means Contracts, tests, reconciliation Evaluations across many runs, plus deterministic checks
Where state lives In tables and files In sessions, memory and context, then in tables
How it fails Retry, backfill, restore Revise the plan, ask a person, be denied by policy
Governance question Who can access which data and operations? Which agent can take which action, under what purpose and conditions?
Who acts, and what it costs Service accounts; compute time An agent acting for a user; tokens and tool calls

COMMON QUESTION

Isn’t an agent platform just a data platform with a chat box?

No, and the difference is not the chat box. A chat box is an interface. The real difference is in the engine underneath: a data platform runs steps someone wrote in advance, while an agent platform lets software decide the next step. That second property needs memory, identity, policy and evaluation that a data platform never needed. The two do share parts, which is why they look alike from far away.

A running example: refund every late order

I’ll use one small task all the way through: refund every order that arrived more than 60 minutes late. The table has five orders. The minutes late are hand-picked, illustrative numbers.

Table of five orders with minutes late 0, 5, 45, 90 and 240, showing that the rule refunds orders 4 and 5, and that order 3 at 45 minutes is the borderline case.

On a data platform this is a nightly job. It reads the five rows, applies the rule “late > 60”, and writes a refund list: orders 4 and 5. Run it a thousand times and you get the same list a thousand times. If it ever produced a different list, you would call that a bug.

On an agent platform the same task looks different. An agent reads the request, looks at the orders, decides what counts as “late”, calls a refund tool, checks the result and replies to the customer. That is more powerful, because it can handle a messy request like “my parcel was ridiculously late, please sort it out”. It is also less predictable. That tension is what the rest of this post is about.

Animated walkthrough of the refund request on a data platform and on an agent platform, ending with a combined path where the agent proposes and a deterministic guard checks.
One request, two paths, then the combined path. Each frame shows its slide number.

The differences in detail

The table has eight rows. Six of them need a closer look, because they change how you design infrastructure.

What each platform is trying to achieve

A data platform’s success metric is about the data: is it fresh, correct and available? An agent platform’s success metric is about the task: was the goal reached, safely and in a way you can inspect? In the refund example, the data platform asks “is the refund list right?”. The agent platform asks “did the customer get a correct outcome, and can we see why?”.

The shape of the work: a graph versus a loop

A pipeline is a directed acyclic graph (DAG): steps that run in a fixed order with no cycles. You can draw it before it runs. An agent runs a loop: it plans, calls a tool, reads what came back, and decides what to do next. You can only draw what it did after it ran. That is why agent platforms invest in tracing, and why data platforms invest in lineage.

Left: a data pipeline drawn as a directed acyclic graph from source to table. Right: an agent drawn as a loop of plan, act, observe and adapt around a goal.

What “correct” means: tests versus evaluations

A data job is tested. You write expected output, run the job, compare. A deterministic transformation should give the same result for the same inputs, code, configuration and execution semantics. Production pipelines can still fail because their inputs, dependencies or environment change, which is why data platforms add contracts, quality checks, lineage and monitoring around the transformation. An agent adds another source of uncertainty, because it may choose a different action or reading of the task even when the situation looks the same. An agent is evaluated. Because its behaviour varies from run to run, one passing run proves little, so you measure how often it succeeds across many runs and many cases. The next section shows this with numbers.

This is not a hypothetical worry. A 2026 benchmark for data agents, built from 104 queries across 17 datasets and four database systems, reported that the best agent it tested reached 57% pass@1 (that is, the share of questions answered correctly on the first try). See the Data Agent Benchmark paper. It is one benchmark, not a verdict on any product, but it shows why evaluation has become its own discipline.

Where state lives

In a data platform the state is the data: tables, files, checkpoints. In an agent platform there is extra state that has no data-platform equivalent: the conversation, the plan so far, what the agent has learned, and what it has been told it may do. Agent platforms call this session state and memory. Google’s Gemini Enterprise Agent Platform, for example, ships a Memory Bank that curates long-term memories from conversations. The more agents do, the more this state looks like data that needs storing, governing and querying.

How each one fails, and recovers

Data platforms fail in known ways: a job crashes, a file arrives late, a schema changes. The answers are known too: retry, backfill, restore. Agent platforms add new failure modes: a wrong diagnosis, a tool called with the wrong arguments, an instruction hidden in a log line. Retries are still dangerous. A retried load that appends rows, for example, turns five orders into ten. Idempotency, meaning running twice has the same effect as running once, matters more when a model decides what to retry.

Who is acting, and what it costs

A data platform usually runs jobs as a service account and bills for compute time. An agent acts on behalf of someone, so the platform must know who. Databricks describes agents inheriting the invoking user’s permissions through on-behalf-of token passing, with each action logged against both the user and the agent (Databricks, 20 May 2026). Google gives each agent its own cryptographic identity. Cost also changes unit: tokens and tool calls instead of cluster hours. Databricks says Genie has no seat-based pricing, and organizations pay only for the AI actually used.

Flow from user to agent to policy gateway to data platform, above an illustrative audit trail table that logs each action against both the user and the agent with the policy decision.

Proof: one run versus a thousand runs

To make this concrete I wrote a small simulation of the refund task. It isn’t a benchmark of any product. It just isolates one idea: a deterministic job and a probabilistic decision-maker need different kinds of evidence.

How the simulation works

This is a deliberately simplified simulation, not a benchmark of an LLM, an agent framework or any AI agent platform. I add random noise to the stand-in agent’s view of the data to model an uncertain decision-maker, and the deterministic job gets no noise. So the accuracy percentages are a consequence of how I built it. The point isn’t the percentage. It’s the architecture: a deterministic guard can enforce an explicit business rule even when the proposal in front of it is unreliable.

  • The five orders are late by 0, 5, 45, 90 and 240 minutes. The rule refunds anything over 60, so the right answer is orders 4 and 5.
  • The data job applies the rule exactly.
  • The stand-in agent sees each lateness with noise (standard deviation 25 minutes, hand-picked) and refunds if the noisy value is over 60. It is a stand-in for any probabilistic decision-maker, not a real language model.
  • The guarded agent lets the stand-in propose refunds, then a deterministic check confirms each one against the real lateness. Any disagreement with the rule is flagged for a person.

I ran each version 1,000 times, with seeds 1, 2 and 3.

About the code: it’s a small illustration, not production code. I ran it and the output shown is real. It uses only the Python standard library and doesn’t call any model or product.

The data job, the stand-in agent and the guard
import random, hashlib, json

LATE = {1: 0, 2: 5, 3: 45, 4: 90, 5: 240}      # order_id -> minutes late (illustrative)
THRESHOLD = 60
TRUTH = sorted(o for o, m in LATE.items() if m > THRESHOLD)   # [4, 5]

# --- the data-platform way: a rule, run as a job ------------------------------
def data_job():
    return sorted(o for o, m in LATE.items() if m > THRESHOLD)

# --- the agent-platform way: a noisy decision-maker ---------------------------
def agent_run(rng, sd=25):
    return sorted(o for o, m in LATE.items() if m + rng.gauss(0, sd) > THRESHOLD)

# --- the blended way: agent proposes, deterministic guard checks the evidence --
def guarded_run(rng, sd=25):
    proposed = set(agent_run(rng, sd))
    allowed = {o for o in proposed if LATE[o] > THRESHOLD}      # evidence check
    flagged = set(TRUTH) ^ allowed                               # disagreement -> human
    return sorted(allowed), sorted(flagged)

def fingerprint(x): return hashlib.md5(json.dumps(x).encode()).hexdigest()[:8]
The experiment and what it prints
def experiment(seed, runs=1000):
    rng = random.Random(seed)
    job_prints = {fingerprint(data_job()) for _ in range(runs)}
    agent_results = [agent_run(rng) for _ in range(runs)]
    per_order = {o: sum(o in r for r in agent_results) / runs for o in LATE}
    exact = sum(r == TRUTH for r in agent_results) / runs
    wrongful = sum(len(set(r) - set(TRUTH)) for r in agent_results)
    missed = sum(len(set(TRUTH) - set(r)) for r in agent_results)
    g_wrong = g_missed_flagged = g_exact = 0
    for _ in range(runs):
        allowed, flagged = guarded_run(rng)
        g_wrong += len(set(allowed) - set(TRUTH))
        g_missed_flagged += bool(flagged)
        g_exact += allowed == TRUTH
    return dict(job_distinct_outputs=len(job_prints), per_order=per_order, exact=exact,
                wrongful=wrongful, missed=missed, g_wrong=g_wrong,
                g_flag_rate=g_missed_flagged / runs, g_exact=g_exact / runs)

if __name__ == "__main__":
    print("Truth (rule says refund):", TRUTH)
    print("Data job, 1000 runs: distinct outputs =", len({fingerprint(data_job()) for _ in range(1000)}))
    for seed in (1, 2, 3):
        r = experiment(seed)
        po = "  ".join(f"o{o}:{p:.0%}" for o, p in r["per_order"].items())
        print(f"\nseed {seed}: agent refund rate per order -> {po}")
        print(f"  agent: whole batch exactly right in {r['exact']:.0%} of runs; "
              f"wrongful refunds {r['wrongful']}, missed refunds {r['missed']} (out of 1000 runs)")
        print(f"  agent + deterministic guard: wrongful refunds {r['g_wrong']}, "
              f"runs flagged for a human {r['g_flag_rate']:.0%}, exactly right unaided {r['g_exact']:.0%}")

What the numbers say

Bar chart of refund rate per order for the stand-in agent: orders 1 and 2 near 1 percent, order 3 near 28 percent, order 4 near 89 percent, order 5 100 percent, against the data job which is always correct.
Version Whole batch exactly right Wrongful refunds (per 1,000 runs) Needs a person
Data job 100% (one distinct output in 1,000 runs) 0 No
Stand-in agent 63% to 64% 295 to 309 No, and nobody would know
Agent with deterministic guard 87% to 90% unaided 0 10% to 13% of runs flagged

A few things stand out. A single run of the stand-in agent could easily look fine, since it is right about 64% of the time, and you would need many runs to see the problem. The trouble sits on the borderline: order 3, 45 minutes late, was refunded in 28% to 29% of runs even though the rule says no. And the guard removed every wrongful refund without needing a better model, turning what was left of the uncertainty into a visible flag.

It’s a toy, so it hides a lot. A real agent’s errors aren’t Gaussian noise on one number, a real guard needs rules you can actually write down, and some tasks have none.

In production the uncertain part might be an LLM reading a customer’s complaint, classifying an exception or proposing an action, and the guard would check that proposal against authoritative order data, business rules and permissions before anything consequential happens. A better experiment would use a real model on ambiguous requests and measure its mistakes, the guard’s rejection rate and how often a person has to step in. What I take from this one is the structure: let the model propose, let deterministic code decide what is allowed, and measure the model over many runs.

COMMON QUESTION

So why not just give the agent SQL access and let it work?

You can, and for reading data it often works well. The trouble starts with actions that change things. Permissions on a database say what the agent can touch, not whether this action makes sense now, in this environment, for this reason. That gap is why platform vendors now ship policy checks that run outside the model’s reasoning loop, as the next sections show.


What an AI agent platform is made of in 2026

An AI agent platform is a stack of seven layers: runtime, tools and gateway, memory, identity, policy, observability and evaluation, and orchestration. Major cloud and data-platform vendors are assembling offerings across most of them, although maturity, integration and availability vary by product. Each layer has a cousin in the data-platform world, which is the first hint that the two are related.

Agent platform layer What it does Data-platform cousin Examples (2026)
Runtime Runs the agent’s code, sometimes for days Compute engine Google Agent Runtime; Foundry Hosted Agents (GA 9 Jul 2026)
Tools and gateway One governed door to tools and other agents Connector and catalogue layer Foundry Toolboxes (GA); Google Agent Gateway and Registry; AWS AgentCore Gateway
Memory Remembers across turns and sessions Storage, feature store Google Memory Bank; Databricks Lakebase as agent memory
Identity Says which agent, acting for whom IAM, service accounts Google Agent Identity; Databricks on-behalf-of tokens
Policy Allows, denies or asks before a tool call Catalogue permissions, row and column security AWS AgentCore Policy (Cedar, GA 3 Mar 2026); Databricks Service Policies; Microsoft Agent Governance Toolkit
Observability and evaluation Traces what happened and scores it Logs, lineage, data quality AgentCore Evaluations (GA 31 Mar 2026); Google Agent Observability
Orchestration Coordinates steps, with checkpoints Workflow orchestrator Microsoft Agent Framework graph workflows with checkpointing
Diagram mapping seven agent platform layers to their data platform cousins: runtime to compute, tools to connectors, memory to storage, identity to IAM, policy to catalogue permissions, observability to lineage and quality, orchestration to workflow schedulers.

Sources for the table: Google, Microsoft Foundry, AWS, Databricks and the Agent Framework 1.0 announcement.

One thing missing from that list is the data itself. An agent platform with nothing to act on is an empty stage, which is why the data platform keeps showing up inside the agent stack.

Who is building what: the 2026 AI agent platform and data platform map

Both sides are moving toward each other: data and cloud vendors are adding agent features, and agent vendors are adding governance and data context. The table summarises public announcements. I haven’t run any of these products, and where a source didn’t say GA or preview, I’ve said so.

Vendor What was announced Status as stated What it signals
Microsoft Agent Framework 1.0, Foundry Hosted Agents, Agent Governance Toolkit GA framework (3 Apr 2026) and Hosted Agents (9 Jul 2026); toolkit public preview SDK, runtime and governance are separate layers
Google Gemini Enterprise Agent Platform (Agent Runtime, Memory Bank, Agent Identity, Registry, Observability); Agentic Data Cloud with Knowledge Catalog Announced 22 Apr 2026; several related pieces preview Agent platform and data platform launched together around one catalogue
Databricks Unity Catalog governing agents and MCP servers, on-behalf-of tokens, Service Policies; Genie Ontology 20 May and 16 Jun 2026; many items carry no GA or preview label The data catalogue is extended to govern agents
AWS AgentCore Policy (Cedar), AgentCore Evaluations, AWS MCP Server GA 3 Mar, 31 Mar and 6 May 2026 Policy and evaluation as managed services outside the model
Snowflake Cortex Agents with MCP connectors; co-led Open Semantic Interchange Cortex Agents documented; OSI announced 23 Sep 2025 Agents work inside warehouse privileges
Palantir Ontology MCP exposing object types, action types and query functions as MCP tools beta documentation A business ontology as the agent’s tool surface
Orchestrators Prefect and Dagster Labs combining; Airflow common-ai provider Announced 13 Jul 2026; no closing date published Schedulers are learning to run agent steps
Observability ClickHouse and Langfuse; OpenTelemetry GenAI conventions Jan 2026; OpenTelemetry status is development Traces are becoming data

Sources for each row are in the sections that follow.

Open-source pipeline tools: what dbt and SQLMesh solve, and what they leave open

dbt and SQLMesh are widely used open-source tools for building complex pipelines, and they sit on the data-engine side of this picture. They turn SQL files into a graph of tested, versioned tables, which is exactly the known, repeatable work from earlier. It’s also why agents are easier to add on top of a well-modelled warehouse than on top of a pile of scripts.

Tool What it does well Open-source status What it leaves to you
dbt SQL models, tests, documentation and lineage as code open source dbt Core v2.0, the Rust-based Fusion runtime, announced under Apache 2.0 and in alpha from 1 Jun 2026 Ingestion, scheduling, compute choice and access control are separate tools
SQLMesh Defining, testing, versioning and deploying SQL transformations Contributed by Fivetran to the Linux Foundation, announced 25 Mar 2026 The same list: it focuses on the transformation step
Airflow, Dagster, Prefect Scheduling and running the steps, with retries and alerts open source core projects What the data means, who may read it and what a change costs

In practice you assemble a stack: a transformation tool models the data, an orchestrator runs it, a catalogue governs it, and a cloud account pays for it. Each piece is good on its own. The trouble is in the joins between them, and four questions fall through the cracks:

WHERE THE JOINS HURT

  • Who may read what?
  • Where does each step run?
  • What will the next change cost?
  • Who approved it?

Small teams lose weeks on these, and large teams lose control. They’re the same questions a shared control plane has to answer for an AI agent platform, so the gap matters more once agents start proposing changes.

Sources: Fivetran on dbt Core v2.0 and the SQLMesh contribution to the Linux Foundation.

What this means for open-source data infrastructure: the gap I am working on

For the last six months I’ve been working on closing this gap. The questions I keep coming back to are the ones the stack above leaves open:

  • How should compute scale up or down with the size of each dataset, without someone tuning every job by hand?
  • How do you get a full data platform without handing your data, and your control, to a SaaS vendor?
  • Can the platform give you a compiled view of itself, built from lineage and schedules, that you can read before anything runs?
  • How do schedulers, dependencies and backfills stay in step with lineage instead of being wired together by hand?
  • What will this change cost, and can you see that in the pull request before it merges?
  • Who can read what, and can that be checked in review instead of discovered after an incident?
  • How do you let an agent write pipelines without letting it deploy them, or widen its own access?
  • How do you make sure nothing unreviewed reaches the people and dashboards that consume the data?
  • Can the whole loop run on a laptop first, then move to your cloud account unchanged?

My attempt at the answers: a pipeline is described in a few lines of YAML, checked, and compiled into a plan you can read before anything runs: what will run, where, on what schedule, who can read what, and what it costs. The transformation itself can be written in SQL, Spark, pandas, Polars or other engines. Nothing runs until a person approves, and then it runs in your own cloud account.

My goal is that a small team can stand up a working data platform quickly, a medium team can add governance without a dedicated platform team, and a large team keeps full authority over its own account and data. I’m still working through these questions, and I’ll share more in coming posts, so I won’t go into detail here.

Where the lines blur: five areas of convergence

Read the vendor map by theme and five areas stand out. In each, an idea from the data platform and an idea from the agent platform are turning into the same thing.

Map of five convergence zones between data platforms and agent platforms: governance, semantic context, memory and traces as data, MCP as front door, and orchestration, each with example vendors.

Governance now covers agents, tools and MCP servers

Governance for agents means treating an agent, its tools and its tool calls as governed objects, the way a data platform treats tables. Databricks describes registering external MCP servers in Unity Catalog as governed securables, and Service Policies that evaluate each tool call and return allow, deny or ask. AWS says AgentCore Policy, generally available since 3 March 2026, uses the open-source Cedar language and is applied outside the agent’s reasoning loop.

Analysts see the same pressure. Gartner predicted in March 2026 that by 2030, 50% of AI agent deployment failures will be due to insufficient AI governance platform runtime enforcement and for capabilities and multisystem interoperability (Gartner). That is a prediction, not a measurement, but it points the same direction as the product launches.

Gartner has also predicted, in June 2025, that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls (Gartner). Governance is one of three named causes, not the only one.

The semantic layer becomes agent context

A semantic layer is the shared definition of business terms, metrics and relationships that sits on top of raw tables. Agents need it, because “revenue” means something specific in your company. Databricks launched Genie Ontology on 16 June 2026 as a context layer that keeps business knowledge up to date so answers come from governed data rather than guesses (Databricks). On 22 April 2026 Google said it had evolved Dataplex Universal Catalog into the Knowledge Catalog. Its Agentic Data Cloud post labels several related pieces, such as the LookML Agent and the Data Agent Kit, as preview.

On the open side, Snowflake, Salesforce and dbt Labs led the Open Semantic Interchange initiative, announced in September 2025, to standardise how semantic definitions are shared. Gartner also predicted that by 2030 universal semantic layers will be treated as critical infrastructure, alongside data platforms and cybersecurity.

Agent memory and traces are becoming data

Every agent run produces a trace: the prompt, the tool calls, the results, the cost. At scale that is a data set. In January 2026 ClickHouse announced that it had acquired Langfuse, an open-source LLM engineering platform, and analysts framed it as a move to turn every LLM call into a structured, queryable record. ClickHouse said it plans to keep Langfuse open source (InfoWorld, 19 Jan 2026). Memory is moving the same way: Databricks documents Lakebase, its managed Postgres database, as a durable memory store for agents. OpenTelemetry moved its GenAI semantic conventions, which cover agent spans and MCP, into a dedicated repository with semantic-conventions v1.42.0. Their status is Development, so they are not yet stable.

MCP is becoming the front door to data

The Model Context Protocol (MCP) is an open protocol that standardises how AI applications connect to tools and data sources. Data and cloud vendors now put MCP in front of their systems. AWS made its managed MCP server generally available on 6 May 2026, using IAM authentication, and says administrators can use IAM or service control policies to keep the MCP server read-only. Snowflake’s Cortex Agents connect to remote MCP servers over the public internet, while data access follows Snowflake privileges. Palantir’s Ontology MCP exposes object types, action types and query functions as MCP tools; its documentation is marked beta. A connection isn’t an authorisation, though. Policy still has to sit behind the door.

Orchestrators are consolidating and adding agents

The scheduler layer is changing too. On 13 July 2026 Prefect and Dagster announced that Prefect is acquiring Dagster Labs. Their own pages use both “acquires” and “is acquiring”, and I found no closing date. The open-source project is to continue, and Dagster and Dagster+ keep their names. Prefect describes the combined stack as Dagster defining outcomes, Prefect executing the work and FastMCP governing access, aimed at agent orchestration (Dagster, Prefect). That is a roadmap, not proof that a unified platform exists yet.

Apache Airflow’s common-ai provider includes an AgentOperator, MCP support and human approval steps for LLM output, per its changelog. When classic pipeline tools learn to run agent steps, the DAG and the loop start to share a canvas.

Practical tip

If you only have time to try one of these, pick the context layer. A clear definition of your five most important metrics, with owners and freshness, makes every agent you add later more accurate and easier to check.

Microsoft’s agent stack: what is GA, and what is open source

Microsoft’s pieces are easy to blur, and they have different status and licences. Microsoft Agent Framework is an open-source, MIT-licensed SDK for building agents and multi-agent workflows in Python and .NET, and it reached version 1.0 in April 2026. That is the Microsoft project most closely matching “agent platform, GA and open source”. It is the SDK, not a hosted service.

Piece Status Open source? What it is
Microsoft Agent Framework 1.0 GA, 3 Apr 2026 announcement; features added since October preview Yes, MIT (repository) SDK for agents and workflows; Semantic Kernel plus AutoGen; MCP supported
Foundry Hosted Agents GA, 9 Jul 2026 No, managed service Runs agent code from your chosen framework in Foundry’s managed runtime
Foundry Toolboxes GA (skills and tool search still preview) No, managed service One MCP-compatible endpoint with managed tool authentication
Agent Governance Toolkit Public preview per its README Yes, MIT Policy, identity, runtime controls for agents
Fabric data agent GA (Purview controls preview) No Read-only Q&A over OneLake data using the asking user’s permissions
Diagram of Microsoft's agent stack split by status and openness: open-source MIT pieces on one side, managed GA services on the other, and preview items marked.

To me the design matters more than the labels. Microsoft separates an SDK (build and orchestrate), a managed runtime (host and operate) and a governance layer (check and audit). Its May 2026 post describes this as “build and orchestrate” versus “govern and audit”. The Fabric data agent shows the data side joining in: it enforces read-only access and runs with the asking user’s credentials and permissions, and row-level and column-level security still apply for Power BI semantic models. That is the data-platform habit applied to an agent. Purview security controls for data agents are still in preview.

Microsoft’s 365 roadmap says Microsoft 365 E7 and Agent 365, a control plane for agents inside Microsoft 365, became generally available on 1 May 2026. See its product roadmap. It is a commercial product, not open source.

What is standardised, and what is not

An open standard in this space is a shared specification that several vendors implement, usually governed by a neutral foundation. The Agentic AI Foundation (AAIF), formed under the Linux Foundation on 9 December 2025, now hosts several of them. Its founding projects were MCP, goose and AGENTS.md. By its 17 August 2026 post, the project list included agentgateway, and the Agent2Agent protocol (A2A) was joining as a hosted project. A2A v1.0, its first stable specification, shipped in March 2026.

These cover the wires: how an agent reaches a tool, another agent, or a set of instructions. I did not find an agreed open standard for the stateful and governance layers: memory, agent identity and policy. Cedar is an open policy language, and OAuth underpins identity, but each vendor assembles those parts differently. That is where portability is hardest and where I would expect the next standards effort.

Diagram of the seven agent platform layers coloured by whether an open standard exists: tools, agent-to-agent and instructions have standards; memory, identity, policy and durable execution do not yet.

One more signal of how fast this moves: OpenAI says Agent Builder and Evals will no longer be available from 30 November 2026, and recommends the Agents SDK for workflows that continue as code. That is a product lifecycle decision, not a sign that evaluation is going away. Evaluation stays a necessary discipline for agent systems; what is unsettled is which hosted product or SDK carries it inside an agentic AI platform.


Will the two platforms merge? My assessment

My view is that an AI agent platform and a data platform won’t collapse into one product. I expect two engines under one control plane, the picture at the top of this post. The data engine keeps doing what it is good at: deterministic, repeatable, reconciled work. The agent engine keeps doing what it is good at: open-ended planning and tool use. The evidence for a shared layer is in the vendor map: identity, policy, context, memory and observability are being built on the catalogue and governance systems data teams already run. The evidence against one engine is the simulation: the two engines need different kinds of proof.

Data platform and AI agent platform side by side, both resting on one shared control plane made of identity, policy, context, memory and observability.
Two engines, one shared plane: the same identity, policy and evidence for both.

My assessment

The question I would ask of any 2026 platform is not “is it a data platform or an agent platform?”. It is “when the model is wrong, what stops the damage, and where is the evidence?” The vendors converging fastest are the ones answering that with governance, not with a bigger model.

Risks of an AI agent platform on your data, and design principles

The main risk of putting agents on data isn’t that they’re slow or expensive. It’s that they act with confidence on incomplete context. The table turns the main failure modes into design responses. It’s my own judgement, not a vendor checklist.

Risk What goes wrong Design response
Confident misdiagnosis The agent applies a wrong fix with conviction Require evidence-backed plans, deterministic diagnostics, and human review for uncertain or high-impact actions
Prompt injection through data Text in a log line, comment or document acts as an instruction Treat logs, comments and documents as untrusted data, keep them separate from instructions, and enforce policy outside the model
Over-privileged tools The agent can do far more than the task needs Least privilege, scoped identities, narrow typed operations, default-deny
Duplicate side effects A retry runs the action twice Idempotency keys, durable workflows, reconciliation checks
Partial completion A multi-step action stops half-way Durable state, compensation or rollback where supported, recovery tests
Stale or missing context The agent acts on out-of-date definitions or lineage Show freshness and provenance, and fail closed when context is missing
Vendor lock-in Operational logic is tied to one cloud’s agent stack Keep operational contracts portable and treat cloud features as adapters

What to do next if you build or run data infrastructure

A practical checklist

  1. Make your top assets readable by agents. For the five tables people ask about most, write the definition, owner, freshness and known caveats in the catalogue.
  2. Start read-only, with the asking user’s identity. Expose data through a governed MCP endpoint, so the agent can never see more than the person who asked.
  3. Log agent actions as data. Store requests, tool calls, policy results and costs in a table you can query.
  4. Keep writes narrow and typed. Replace “run any SQL” with specific operations that check preconditions and verify results.
  5. Turn incidents into evaluations. Every wrong answer becomes a test case you run many times, not once.
  6. Set budgets. Cap tokens, tool calls and cost per task before the first surprise bill.
  7. Pilot one workflow. Pick something with a clear owner, such as triaging a failed pipeline, and measure it for four weeks.

Where an infrastructure builder can add value

  • Context quality: tooling that keeps definitions, owners and freshness accurate, since agents amplify whatever the catalogue says.
  • A trace schema: an open, queryable layout for agent runs that joins traces to lineage and cost.
  • Policy as data: rules that live next to the catalogue and can be tested and versioned like code.
  • Evaluation sets from your own incidents: a growing, domain-specific test suite beats a generic benchmark. My post on LoRA fine-tuning makes a similar point about small, specific data.

A four-week pilot for an AI agent platform

The cheapest way to learn is to prove one workflow end to end before building a platform.

Four-week pilot roadmap: week 1 audit and baseline, week 2 pick one workflow, week 3 end-to-end prototype, week 4 adversarial test, ending in a go or no-go decision.
Week Activity Exit criterion
Week 1: audit Inspect the platforms and projects you already run. List your five most-asked tables with owners and freshness. A capability matrix and a one-page baseline
Week 2: one workflow Pick one failure that costs time, such as a late source file. Measure how it is handled today. A named user, trigger and success metric
Week 3: prototype A read-only agent diagnoses a seeded failure. A deterministic guard checks the plan. A person approves. A repeatable demo with an audit trail
Week 4: adversarial test Try bad plans, duplicate retries, timeouts and stale metadata. A threat list and a go or no-go decision

Summary

  • A data platform runs known work repeatably and proves it with tests. An AI agent platform runs open-ended work and judges it with evaluations.
  • They differ on goal, shape of work, core objects, meaning of correct, state, failure handling, governance, and identity and cost.
  • An agent platform has seven layers, and each one echoes something data teams already know.
  • The lines blur at governance, context, memory and traces, MCP and orchestration, all with dated launches in 2026.
  • Microsoft’s open-source, GA piece is Agent Framework 1.0 (MIT). Hosting and Foundry services are managed, and the Governance Toolkit is in preview.
  • dbt and SQLMesh model and test the transformation step well. Ingestion, scheduling, access and cost are still separate tools, and that joining work is the gap I am building for.
  • Expect two engines under one control plane. Let the model propose, let deterministic code decide, measure over many runs, and pilot one workflow for four weeks.

Frequently asked questions

What is the difference between an AI agent platform and a data platform?

A data platform moves, stores and governs data with work defined in advance, and correct means tested and repeatable. An AI agent platform runs software that plans, calls tools and adapts, and correct means evaluated across many runs plus checks that do not depend on the model. They differ on goal, shape of work, meaning of correct, state, failure handling, and identity and cost.

Are AI agent platforms and data platforms merging?

They are converging on a shared control plane, not on one engine. The overlap is governance for agents and tools, semantic layers as agent context, memory and traces stored as data, MCP as the access layer to data, and orchestrators that can run agent steps. The data engine and the agent engine still do different jobs.

What is an agentic AI platform?

An agentic AI platform is infrastructure for building, running and governing AI agents: software that plans, calls tools and adapts to results. Typical layers are runtime, tools and gateway, memory, identity, policy, observability and evaluation, and orchestration.

What is a data platform for AI agents?

It is a data platform that gives agents governed access to data: shared definitions of metrics, permissions that follow the asking user, audit of every action, and checks that run outside the model. The data platform supplies the facts and the rules, and the agent supplies the reasoning.

How do you test an AI agent compared with a data pipeline?

A pipeline is tested by comparing its output with an expected result, and a deterministic transformation gives the same result for the same inputs and code. An agent varies from run to run, so you run many cases many times and measure how often it succeeds. In the simulation in this post, a stand-in agent got the whole five-order batch exactly right in about 63 to 64 percent of 1,000 runs.

What is GA and open source in Microsoft's agent stack?

Microsoft Agent Framework 1.0 is an MIT-licensed open-source SDK for agents and workflows, announced as generally available in April 2026. Foundry Hosted Agents is generally available but is a managed service, not open source. The Agent Governance Toolkit is MIT licensed but described as public preview in its README.

Should a data engineer learn agent platforms?

Yes, in a practical way. Start with the context layer by defining your key metrics with owners and freshness, add policy and verification checks outside the model, and keep the agent's access read-only until the checks prove themselves.

Sources: Microsoft Agent Framework 1.0; Foundry July–August 2026; Gemini Enterprise Agent Platform; AWS MCP Server; OpenAI AgentKit; Agentic AI Foundation; MCP specification; Gartner on agentic AI project cancellations.

Nikhil Aggarwal

Written by Nikhil Aggarwal

Nikhil Aggarwal is an ML engineer with 15+ years in data engineering, distributed systems and cloud. He builds ML and GenAI capabilities for enterprise data platforms, and writes DataForGeeks to explain complex topics through real production problems.

About me · LinkedIn · Newsletter