hdfs

Impala – Create Table AS Select * FROM Table – is SLOW

by

Below query seems like the simplest way to create a replica of table. But simplicity comes with some cost as well. Above query will : NOT create partitions if there are any on TABLE_NAME_2 run very slow Instead of above we should follow following 2 way approach :  CREATE TABLE TABLE_NAME    Like TABLE_NAME_2;  — … Read more

HDFS – Data Movement across clusters

by

You can move data in HDFS cluster using distcp command. distcp uses 10 mappers by default to bring data from source system. While doing data movement I encountered a problem in which data movement was failing because of checksum mismatch. If any block mismatch in the checksum then the complete data block was getting discarded.  Checksum is … Read more

Sqoop – How to import data in HDFS

by

Sqoop is a tool to transfer data between Hadoop and relational databases. It uses MapReduce to import and export the data, which provides parallel operation as well as fault tolerance. Basic syntax to run Sqoop is :sqoop tool-name [Generic-Args] [Tool-arguments] All Generic arguments should precede any tool arguments. All Generic Hadoop arguments are preceded by a … Read more