Python Data Structures Simplified: List, Tuple, Dict, Set, Frozenset & More

by
Colorful icons representing Python data structures — List, Tuple, Dict, Set — visual from the blog Python Data Structures Simplified.

Python offers a rich set of built-in and extended data structures to efficiently manage and process data. In this blog, we’ll deep dive into essential ones: List, Tuple, Dictionary (Dict), Set, Frozenset, and also explore some powerful structures from the collections and dataclasses modules. We’ll cover their properties, use-cases, constructors, and how to convert between them using intuitive examples. Note: Since Python 3.7, dictionaries … Read more

Understanding SQL Execution Order and Corresponding PySpark Syntax

by

When writing SQL queries, it is essential to understand the order in which SQL clauses are executed. This helps in writing optimized queries, especially when transitioning from SQL to PySpark. In this blog, we’ll walk you through the SQL execution order, the SQL clauses, and provide their corresponding PySpark syntax. SQL Execution Order and Corresponding … Read more

Snowflake – Performance Tuning and Best Practices

by
Snowflake Performance Tuning with charts and best practices visual illustration

Snowflake’s cloud-native architecture makes it incredibly easy to get started — but running it efficiently at scale is a whole different game. If you’ve ever faced slow queries, ballooning credit consumption, or unpredictable performance, you’re not alone. Tuning Snowflake workloads requires more than just adjusting warehouse sizes — it involves understanding how Snowflake stores data, … Read more

Apache Spark – Performance Tuning and Best Practices

by
Visual representation of Apache Spark performance tuning with Spark logo and performance gauge, from the blog Apache Spark – Performance Tuning and Best Practices.

Apache Spark has revolutionized the way we process large-scale data — delivering unparalleled speed, scalability, and flexibility. But as many engineers discover, achieving optimal performance in Spark is far from automatic. Your job runs — but takes longer than expected. The cluster scales — but the costs rise disproportionately. Memory errors appear out of nowhere. … Read more

Data Serialisation – Avro vs Protocol Buffers

by
Visual comparison of Avro vs Protocol Buffers for data serialisation, with arrows representing data flow for each format

Background File Formats Evolution Why not use CSV/XML/JSON?  Repeated or no meta information. Files are not splittable, so cannot be used in a map-reduce environment. Missing/ Limited schema definition and evolution support. Can leverage “JsonSchema” to maintain schema separately for JSON. It may still require transformation based on a schema, so why not consider Avro/Proto? … Read more

Count(*) – Explaining different behaviour in Joins

by

Observations :  Count(1) or Count(*) – This is never expanded on each column individually so will work perfectly fine on complete data.  Count(1) is more optimized then Count(*) Count(source.*) – source represents “Left table” of “Left Outer Join”: This will be evaluated as Count(source.col1, source.col2, …. source.colN ) So, if any column has NULL, then the complete row … Read more

Cost and Performance Analysis : CSV and Parquet Format

by

I was doing some cost comparison of using CSV files vs Parquet File. Interestingly, when using Parquet format, data scanning for similar queries, cost 99% less as compared to CSV format. Queries ( Mentioned only for Parquet) CSV ( 11.32 GB )Run Time (in sec) CSV ( 11.32 GB )DataScanned (in GB) PARQUET ( 4.1 GB )Run Time (in sec) PARQUET ( 4.1 GB )DataScanned (in GB) … Read more

Connect to RedShift database from DBeaver

by

DBeaver is a SQL Client which is helpful in querying data and its analysis. It is loaded with many generic JDBC drivers for connecting to popular databases. For others, we can add driver separately and establish a connection.  Following are the steps to connect to Redshift data from DBeaver :  Step 1 : Create new … Read more

Sales Analysis – 3

by

Tables – product and sales are provided. Write SQL query to identify products which were sold only in first quarter of year 2019. If a product is sold in any other quarter then that is not supposed to be present in final output. Output should be product_id and product_name. Input Table: sales seller_id product_id buyer_id … Read more