Table of Contents
The Era of Chaos – and Snowflake’s Rise
Back in 2017, most of us were drowning in messy data. Files were everywhere in S3 buckets, Hadoop jobs kept failing at the worst times, and analysts? They were always chasing clean data that never seemed to arrive when needed. It was frustrating, and honestly, it felt like we were constantly putting out fires just to keep the lights on.
Then came Snowflake — a warehouse that just worked.
It brought:
-
Separation of compute and storage (finally!)
-
Auto-scaling that felt like magic
-
A SQL-first approach that made analysts fall in love again
For a while, Snowflake was the promised land. If you could ETL your data into a table, it would handle the rest. And for 80% of use cases — dashboards, KPIs, finance reports — it was unbeatable.
But in that remaining 20% lived the next big frontier: machine learning, real-time analytics, and semi-structured chaos.
And this is where Databricks played its hand.
The Lakehouse Gambit — and the Birth of Medallion Architecture
Databricks didn’t just sell a product. It sold a vision — one that engineers could hold onto.
Enter: the Lakehouse.
It promised the reliability of a warehouse with the flexibility of a data lake. But a vision needs a map. And that’s when someone at Databricks did something genius:
They gave it a name. A shape. A story.
Bronze → Silver → Gold
The Medallion Architecture was born.
This wasn’t just another pipeline. It was a framework that gave every data engineer a playbook:
-
Bronze: Raw ingestion, straight from source.
-
Silver: Cleaned and deduplicated.
-
Gold: Business-level aggregates ready for consumption.
Simple? Yes. Powerful? Absolutely. Because it wasn’t just about layering data — it was about layering trust.
And suddenly, pipeline chaos had order. Data engineers, scarred by years of schema drift and late-night failures, had a new religion.
Viral by Design — Why Everyone Started Following the Medallion Playbook
I remember the first time I heard a junior engineer explain our stack to a product lead.
“We’re using the Medallion architecture. So this dataset is still in Bronze — you probably don’t want to trust it yet.”
That moment stuck with me.
Because Databricks didn’t just build a platform — it built shared language.
-
It was easy to put in a slide deck.
-
Easy to teach new hires.
-
Easy to defend in an architecture review.
Even consultants and tech influencers latched onto it. Courses, whitepapers, webinars — suddenly, Medallion was everywhere.
And behind the scenes? Databricks was tightening its grip:
-
Delta Lake matured.
-
Structured Streaming became first-class.
-
MLflow integrated model ops.
It wasn’t just a platform. It was an ecosystem. And Medallion was the banner under which the movement marched.
The Warehouse Fights Back — Why Snowflake Isn’t Dead Yet
Let’s not write the eulogy for Snowflake just yet.
Because while Databricks was winning engineers, Snowflake was still winning executives.
-
You can’t beat Snowflake’s simplicity.
-
You don’t need to be a DevOps savant to manage it.
-
For many companies, just getting data from CRM to Tableau is enough — and Snowflake makes that effortless.
But here’s the catch: data needs are evolving, and Snowflake knows it. Which is why it’s no longer just a warehouse — it’s now:
-
A platform for data apps
-
A runner of Python and Snowpark workloads
-
A host for Iceberg tables
It’s trying to reclaim relevance where Databricks took the lead.
“Snowflake is no longer your SQL sanctuary — it’s trying to be your everything.”
And therein lies the friction. Snowflake’s platform is growing more complex, more layered, and — let’s be honest — more DevOps-heavy than it once was.
The irony? The platform that once won because it didn’t need infrastructure engineers… now needs them again.
Many teams are starting to feel the strain:
-
“We used Snowflake because we didn’t want to hire infra engineers.”
-
“Now we need a full team just to manage connectors, pipelines, and costs.”
More services. More knobs. More decisions.
And it raises the fundamental question:
“Do we really need more scale — or do we need better governed data?”
Because let’s face it — most companies don’t have Netflix-scale data. But they do have Netflix-scale chaos.
And that’s where the Medallion story continues to resonate: It promises governance, clarity, and intentional design.
The Rise of Iceberg — and the Lakehouse Arms Race
Now enter another plot twist: Apache Iceberg.
If Delta Lake was Databricks’ secret sauce, Iceberg is the community’s open response.
-
Supported by Snowflake, AWS, Dremio, and more
-
Brings table formats to open lake storage
-
Decouples compute from storage even further
Snowflake’s support of Iceberg is a direct acknowledgment: the future isn’t warehouse vs lakehouse — it’s interoperable layers over open formats.
The battle is no longer who owns the stack. It’s who plays nicest with the stack you already have.
And that means:
-
Open formats like Iceberg are rising
-
Proprietary lock-in is fading
-
Tools that respect your architecture will win
Databricks saw this early. And its Medallion framework works just as well on Iceberg as it does on Delta — because it’s about structure, not software.
“In a world where everything is open, clarity becomes the true currency.”
What Medallion Really Taught Us
It’s easy to think that Databricks won market share because it had a better product. But I think the truth runs deeper:
It won mindshare by telling a better story.
Medallion architecture gave us more than just Bronze, Silver, and Gold layers. It gave us narrative clarity in a field drowning in complexity.
And that’s the real lesson.
Snowflake may have optimized the engine. But Databricks painted the road.
And in this industry, that can be the difference between a product and a movement.
What Should You Do as a Data Leader?
If you lead a data team, ask yourself:
-
Have we defined a clear architecture that engineers and analysts both understand?
-
Do our pipelines reflect trust boundaries — or just transformation steps?
-
Can someone new navigate our data like a map?
If not — Medallion isn’t a Databricks-only idea.
It’s a mindset.
And it might be the clarity your data stack is missing.
“We don’t need another data tool. We need a compass.”