for Data Enthusiasts
Your coding agents are out of hand. Good. Claude Code is refactoring something. Codex is halfway through a test run. A third agent has been sitting on an approval prompt for eleven minutes, waiting for a keystroke you didn’t know it wanted. And in a tab you forgot existed, a benchmark has been running since … Read more
Spark job suddenly slow? Learn to find data skew in the Spark UI and fix it with AQE, broadcast joins and salting. Real screenshots and interview answers.
It’s 9:14 PM on a rainy Friday in Bengaluru. Nikita has just wrapped up a long week at work and wants to get home. She taps Request Ride. Within about 400 milliseconds her request has to be authenticated, rate-limited, priced, safely recorded and handed to dispatch. Within a few seconds, a dozen systems in her … Read more
Some system design questions sound small and turn out to be huge. “A third party sends us data; put it on a dashboard” is one of them. There’s no ride to follow and no fancy product. There’s a vendor you don’t control, a network you don’t share, a sale that turns a trickle into a … Read more
We adopted a practical Medallion-style approach to structure our data flows – segmenting data flows into Bronze, Silver, and Gold layers. As part of this redesign, we needed to optimize how curated data was exported to Snowflake. That’s when we hit performance issues with external tables. I know the common suggestion is to use the … Read more
By the end of this read, you’ll understand why Apache Iceberg is not just another open table format — it’s the seismic shift that’s redefining the role of Delta Lake in the modern data ecosystem and transform how the world thinks about data lakes. From Data Lakes to Data Icebergs: A New Era Begins Over … Read more
The Era of Chaos – and Snowflake’s Rise Back in 2017, most of us were drowning in messy data. Files were everywhere in S3 buckets, Hadoop jobs kept failing at the worst times, and analysts? They were always chasing clean data that never seemed to arrive when needed. It was frustrating, and honestly, it felt … Read more
Tired of messy Python setups? Ever screamed at your terminal? Been there, done that, deleted Anaconda. Let me show you how I set up a clean, beautiful, and powerful Python development environment on my Mac. It’s light, customizable, and perfect for devs who love a good-looking terminal and tight control over Python versions. 🤓 Why … Read more
Python offers a rich set of built-in and extended data structures to efficiently manage and process data. In this blog, we’ll deep dive into essential ones: List, Tuple, Dictionary (Dict), Set, Frozenset, and also explore some powerful structures from the collections and dataclasses modules. We’ll cover their properties, use-cases, constructors, and how to convert between them using intuitive examples. Note: Since Python 3.7, dictionaries … Read more
When writing SQL queries, it is essential to understand the order in which SQL clauses are executed. This helps in writing optimized queries, especially when transitioning from SQL to PySpark. In this blog, we’ll walk you through the SQL execution order, the SQL clauses, and provide their corresponding PySpark syntax. SQL Execution Order and Corresponding … Read more