Impala – Use Incremental stats instead of Full Table stats

If you have a table which is partitioned on a column then doing
Compute stats TABLE_NAME
will execute on all partitions.

Internally compute stats run NDV function on each column to get numbers. However NDV function works faster then other count(COLUMN), but it will run for each partition which may be irrelevant when you are working/updating/modifying values on only one partition.

So, instead use :
Compute INCREMENTAL Stats TABLE_NAME;

This will run on a partition for which stats are not collected. So if you have dropped/added a partition it will run on that. You can specify partition separately as well to compute stats on it only.

Note : Please check the version as this may not be available for the version you are running.

Nikhil Aggarwal

Written by Nikhil Aggarwal

Nikhil Aggarwal is an ML engineer with 15+ years in data engineering, distributed systems and cloud. He builds ML and GenAI capabilities for enterprise data platforms, and writes DataForGeeks to explain complex topics through real production problems.

About me · LinkedIn · Newsletter