Summarize this content to 100 words:
crossroads in the data world.
On one hand, there is a universal recognition of the value of internal data for AI. Everyone understands that data is the critical foundational layer that unlocks value for agents and LLMs. And for many (all?) enterprises, this isn’t just one more innovation project — it is viewed as a matter of life or death.
On the other hand, “legacy” data use cases (business intelligence dashboards, ad-hoc exploration, and everything in-between) are increasingly viewed as nice-to-have collections of high-cost, low-value artifacts. The C-suite and other data stakeholders are slowly but steadily starting to ask the uncomfortable question out loud: “Why are we spending $1M on Snowflake just to generate a bar chart we look at once and then forget about?” (Well, fair enough.)
This puts data teams in a precarious spot. For the last five years, we invested heavily in the Modern Data Stack. We scaled our warehouses and treated every problem as a nail that needed a dbt hammer. (Because one more dbt model will create all the difference, right? Rigth?) We collectively convinced ourselves that surely more tooling and more code will result in more business value and happier data consumers.
The result? Unnecessary complexity and “model sprawl.” We built an ecosystem that was easier than Hadoop, sure, but we optimized for volume rather than value.
Today, data teams are paralyzed by mountains of tech debt — thousands of dbt models, hundreds of fragile Airflow DAGs, and a sprawling vendor list — while the business asks why we can’t just “plug the LLM into the data” tomorrow.
We were caught off guard. The killer use case finally arrived, and it’s more exciting than we ever anticipated, but our tooling was built for a different era (and critically, a different type of data consumer). For a group of people who work with predictions daily, we turned out to be terrible at predicting our own future.
But it’s not too late to pivot. If data teams want to survive this shift, we need to stop building like it’s the peak of the dbt gold rush. In this article, I’ll cover six strategic imperatives to focus on right now, as you, fellow data person, transition to a completely new raison d’être.
1. Features as Products, No More: Putting the Stack on a Diet
This sounds counterintuitive, but hear me out: The first step to survival isn’t adding; it’s subtracting.
We need to have an honest (and slightly uncomfortable) conversation about “Modern Data Stack” bloat. For a few years, we operated under a model where every single feature a data team needed turned into a separate vendor contract. We basically traded configuration friction for credit card swipes. While the architecture diagrams we (myself included) designed during this era, featuring dozens of logos and a dedicated tool for every minor step in the pipeline, might have looked impressive on a slide, they created an ecosystem that is hostile to quick iteration.
The landscape has shifted. Cloud data platforms (the Snowflakes and Databricks of the world) have aggressively moved to consolidate these capabilities. Features that used to require a specialized SaaS tool, from notebooks and lightweight analytics to lineage and metadata management, are now native platform capabilities.
The necessity for a fragmented “best-of-breed” stack is becoming an anomaly, applicable only to niche use cases. For the masses, built-in capabilities are finally good enough (really!). In 2026, the most successful data teams won’t be the ones with the most complex architectures; they’ll be the ones who realized their cloud data platform has quietly eaten 70% of their specialized tooling.
There is also a hidden cost to this fragmentation that kills AI projects: Context Silos.
Specialized vendors are notoriously protective (to say the least) of the metadata they capture. They build walled gardens where your lineage and usage data are trapped behind limited (and barely documented) APIs. This, unsurprisingly, is fatal for AI. Agents rely entirely on context to function — they need to “see” the whole picture to reason correctly. If your transformation logic is in Tool A, your quality checks in Tool B, and your catalog in Tool C, with no metadata standards in between, you have fragmented the map. To an AI agent, a complex stack just looks like a series of black boxes it cannot learn from.
The Diet Plan:
Declarative Pipelines over Heavy Orchestration: Do you really need a complex Airflow setup to manage dependencies when capabilities like Snowflake’s Dynamic Tables or Databricks’ Delta Live Tables can handle the DAG, retries, and latency automatically? The “default” orchestrator layer is shrinking: It’s still relevant (and necessary) in some cross-system steps, but 90% of the orchestration can be managed natively.
Platform over Plugins: Do you need a separate vendor just to run basic anomaly detection when your platform now offers native Data Metric Functions or pipeline expectations? The closer the check is to the data, the better.
The Artifact Audit: We’ve spent years rewarding “shipping code.” This incentive structure led to a codebase of thousands of models where 40% aren’t used, 30% are duplicates, and 10% are just plain wrong. It is time to delete code. (You won’t miss it, I promise! Code is a liability, not an asset.)
Built-in over Bolt-on: The “best-of-breed” overhead — the integration cost, the procurement friction, and the metadata silos — is now higher than the marginal benefit of those specialized features. If your platform offers it natively, use it.
Survival depends on agility. You cannot pivot to support AI agents if you are spending 80% of your week just keeping the “Modern Data Stack” Frankenstein monster alive.
2. True Decoupling: Storage (and Data!) is Yours, Compute is Rented
For the last decade, we’ve been sold a convenient half-truth about the “separation of storage and compute.”
Vendors told us: “Look! You can scale your storage independently of your compute! You only pay for what you use!” And while that was true for the resources (and the bill), it wasn’t true for the technology. Your data, while technically sitting on cloud object storage, was locked inside proprietary formats that only that specific vendor’s engine could read. If you wanted to use a different engine, you had to move the data: We separated the bill, but we kept the lock-in.
A New Ice(berg) Age:
For the new wave of data use cases, we need true separation. This means leveraging Open Table Formats (long live Apache Iceberg!) to ensure your data lives in a neutral, open state that any compute engine can access.
This isn’t just about avoiding vendor lock-in (though that’s a nice bonus). It is about AI readiness and agility.
The Old Way: You want to try a new AI framework? Great, build a pipeline to extract data from your warehouse, convert it, and move it to a generic lake.
The New Way: Your data sits in Iceberg tables. You point Snowflake at it for BI. You point Spark at it for heavy processing. You point a new, cutting-edge AI agent framework at it directly for inference.
No migration. No movement. No toil.
To be clear, this doesn’t mean abandoning native storage entirely. Keeping your high-concurrency serving layer (your “Gold” marts) in a warehouse format for performance is fine. The critical shift is that your central gravity (the source of truth, the history, etc. ) now resides in an open format, not proprietary ones.
This architecture ensures you are future-proof. When the “Next Big Thing” in AI compute arrives six months from now (or less?), you don’t need to rebuild your stack. You just plug the new engine into your existing storage, with no “translator” or friction in between.
3. Stop Being a Service, Start Being a Product
The dream of “universal self-serve” was a noble one. We wanted to build a platform where anyone could answer any data question and create elegant artifacts/visualizations, with 0 Slack messages involved. In reality, we often built a “self-serve” buffet where the food was unlabeled and half the dishes were empty.
Data teams are almost always understaffed. Trying to win every battle means you lose the war. To survive, you must pick your verticals.
The Shift to Data Products:
Instead of shipping “tables” or “dashboards,” you need to ship Data Products. A product isn’t just data; it’s a package that includes (but isn’t limited to):
Clear Ownership: Who is the “Product Manager” for the Revenue Data?
SLAs/SLOs: If this data is late, who gets paged? How fresh does it actually need to be?
Success Metrics: Is this data/product actually moving the needle, or is it just “nice to have”?
I’ve written extensively about the mechanics of data products before — from writing design docs for them to structuring the underlying data models — so I won’t rehash the details here. The critical takeaway for the next era is the mindset shift: This isn’t just about the data team changing how we build; it’s about the entire organization changing how they consume.
So, where to start? First, stop trying to democratize everything at once. Identify the three business verticals where data can actually create a “quick win” — maybe it’s churn prediction for the CS team or real-time inventory for Ops — and build a cohesive, high-quality product there. You build trust by solving specific business problems, rather than spreading yourself thin across the entire company.
4. Foundations for Agents: The Context Library
We’ve spent a decade optimizing for human eyes (dashboards). Now, we need to optimize for machine “brains” (AI Agents).
As data teams, we were collectively taken off guard by the emergence of enterprise AI: While we were busy buying yet more SaaS tools to create more dbt models for more dashboards (sigh), the ground shifted. Now, there’s a supercharged AI that’s hungry for “context.” The initial reaction in the space was a rush to portray this context as simply connecting an LLM to your warehouse and catalog and calling it a day.
On the surface, that approach may sound “good enough”, sure. It will result in some nice demos and impressive 10-minute showcases at data conferences. But the bad (good?) news is that production-grade context is much, much more than that.
An AI agent doesn’t care about your neat star schema if it doesn’t have the semantic meaning behind it. Giving an LLM access to only breadcrumbs (whether it’s table/field names or a Parquet file with columns like attr_v1_final) is like giving a toddler a dictionary in a language they don’t speak. It drastically limits the field of possibilities and forces the LLM to hallucinate generic, low-value context to fill the massive void left by our collective lack of standardized documentation.
Building the Context Library:
The “Semantic Layer” has been an on-and-off hot topic for years, but in the AI era, it is a literal requirement. Agents deserve (and require) much more than the thin layer of metadata we’ve built in the Modern Data Stack world. To get things back on track, you need to start doing the “unglamorous” groundwork:
The Documentation Debt: It’s not enough to know how to calculate a metric. AI needs to know what the metric represents, why it is calculated that way, and who owns it. What are the edge cases? When should a condition be ignored? And most importantly, what needs to happen once a metric moves? (More on this later.)
Capturing the “Oral Tradition”: Most business context currently lives in “tribal knowledge” or forgotten Slack threads. We need to move this into machine-readable formats (Markdown, metadata tags, etc.) that detail how the business actually operates — from the macro strategy to the micro nuances.
Standards & Changelogs: Agents are highly sensitive to change. If you change a schema without updating the “Context Library,” the agent (understandably) hallucinates. Documenting means ensuring that your context is a living organism that accurately reflects the current state of the world and the events that led to it (with their own context).
The format matters less than the content. AI is great at translating JSON to YAML to Markdown (so definitely use it to bootstrap your context library from raw code and Google docs, giving you a solid baseline to refine rather than a blank page). It is not great, however, at guessing the business logic you forgot to write down.
In short: Document, document, document. The AI gods will figure out how to read your documentation later.
(Note: If you want a deeper dive on the AI-ready semantic layer, I recently published a blog post on this topic specifically.)
5. From “What Happened?” to “What Now?”
The pre-AI world was a passive, descriptive one. We called it BI.
The workflow went like this: You build a dashboard, it sits in a corner, and a human has to remember to look at it, interpret the squiggle on the chart, and then decide to take an action (or, much more frequently, just do what they were planning to do anyway). This is the “Data-to-Decision” gap, and it’s where value goes to die.
In tomorrow’s brave new world, the micro-decision will no longer be taken by humans. Humans set the strategy, sure, but the execution is getting automated at an impressive pace.
We need to stop being the team that “provides the numbers” and start being the team that builds the systems that turn those numbers into immediate action.
Architecting the Feedback Loop:
We need to shift from passive dashboards to automated feedback loops.
Metric Trees over Flat Metrics: Don’t just track “Revenue.” Track the granular metrics that feed into it and map how they are interconnected. The formula isn’t always exact or scientific, but capturing the relationships is critical. An AI agent needs to know that Metric A influences Metric B (+ how and why) to traverse the tree and find the root cause.
The “If This, Then That” Strategy: If a granular metric moves outside of a defined threshold, what is the automated response? We need to encode this logic and the different paths that align with the overall business strategy. (Scenario: Churn risk for Tier 1 users spikes. Old Way: A dashboard turns red. Someone maybe sees it next week. New Way: Trigger an automated outreach sequence (with fine-tuned AI-powered messaging) and alert the account manager in Salesforce instantly.)
Active Navigation over Passive Validation: The industry is still unfortunately plagued by “Validation Theater”: using charts to retroactively justify decisions already made. Changing this dynamic is mandatory as AI becomes more capable. The goal is to build systems where data acts as a strategic navigator: actively analyzing real-time context to propose the optimal path forward and, where appropriate, automatically triggering the next step (within defined guardrails). The dashboard shouldn’t be a report card; it should be a recommendation engine.
The question isn’t “What does the data say?” It is: “Now that the data says X, what action are we taking automatically?”
6. The Evolving Data Persona: “Who Writes the SQL” Doesn’t Matter
A few years ago, the “Analytics Engineer” was essentially a dbt model factory. Today, that role is slowly evaporating as humans move one abstraction layer up in practically all professions. If your primary value prop is “I write SQL,” you are competing with an LLM that can do it faster, cheaper, and increasingly better.
The data roles of the next wave will be defined by rigor, architecture, system thinking, and business sense, not syntax or coding skills.
The Full-Stack Data Mindset:
Moving Upstream (Governance): We can no longer just clean up the mess once the data reaches our clean and tidy data platform (is it?). We need to move left by establishing Data Contracts (regardless of format) at the source and enforcing quality at the point of creation. It is no longer enough to “ask” software engineers for better data; data teams need the engineering fluency to actively collaborate with product teams and build data-literate systems from day one.
Moving Downstream (Activation): We need to get closer to the activation layer. It’s not enough to “enable” the business; we need to act as Data PMs, ensuring the data product actually solves a user problem and drives a workflow. (Thus, as a data person, understanding the business you’re building products for is quickly becoming a requirement.)
Working Above the Code: Your job is to define the standards, the guidelines, and the governance. Let the machines handle the boilerplate while you ensure the business logic is sound and the AI has the right context.
It doesn’t matter who (or what) writes the code. What matters is the rigor: Data mistakes in the AI era are exponentially more costly. A wrong number in a dashboard is an annoyance that, let’s be honest, gets ignored half the time. A wrong number in an AI agent’s loop triggers the wrong action, sends the wrong email, or turns off the wrong server — automatically and at scale.
A final reality check: It’s all about the business
When I transitioned from data engineering to product management a couple of years ago, my perspective on the data team’s role shifted instantly.
As a PM, I realized I don’t care about neat data models. I don’t care if the pipeline is “elegant” or if the data team is using the coolest new tool. I have a meeting in 15 minutes where I need to decide whether to kill a feature. I just need the data to answer my question so I can move forward.
Data teams are, by design, a bottleneck. Everyone wants a piece of your time. If you cling to “the way we’ve always done it” — insisting on perfect cycles and rigid structures while the business is moving at AI speed — you will be bypassed.
The Survival Kit is ultimately about flexibility. It’s about being willing to let go of the tools you spent years learning. It’s about realizing that “Data Engineer” is just a title, but “Value Generator” is the career.
Embrace the mess, cut the fat, and start building for the agents. Over the next decade, the data landscape is going to be wild — make sure you’re not distracted by the impressive architecture diagrams or cool tech you see along the way; the only outcome that matters will always be how much value you generate for the business.
Mahdi Karabiben is a data and product leader with a decade of experience building petabyte-scale data platforms. A former Staff Data Engineer at Zendesk and Head of Product at Sifflet, he is currently a Senior Product Manager at Neo4j. Mahdi is a frequent conference speaker who actively writes about data architecture and AI readiness on Medium and his newsletter, Data Espresso.
