December 29, 2021
A quick answer that might come to your mind is to call the count() function on the dataframe and check if the count is greater than […]
October 29, 2021
Hive by default store all the files behind the Hive table under the warehouse directory. But this location can be overridden during table or later when […]
October 25, 2021
If you have worked with tools like Toad, DB Visualizer or SQL Server Management Studio you know it is quite easy to select an existing table […]
October 22, 2021
In this post we will explain the architecture of Hive along with the various components involved and their functions. HiveServer2 HiveServer2 is an improved implementation of […]
October 18, 2021
As simple as this question sounds, we get this question a lot. So we decided to write a post on finding the version of Hadoop and […]
October 15, 2021
LATERAL VIEW and EXPLODE are 2 different things in Hive. Lateral view is used in conjunction with user-defined table generating functions such as explode(). Problem Let’s […]
October 11, 2021
Comparing 2 dates is quite common when you deal with data. Hive has datediff function to help you compare 2 dates. Solution datediff function in Hive […]
October 8, 2021
Simple problem with a simple solution. Solution Order the records first and then apply the LIMIT clause to limit the number of records. SELECT * FROM […]
October 4, 2021
foreach() and foreachPartition() are action function and not transform function. Both functions, since they are actions, they don’t return a RDD back. Do you like us […]
October 1, 2021
Simple problem with a simple solution. Solution Use the powerful regexp_replace function to replace characters. regexp_replace(string INITIAL_STRING, string PATTERN, string REPLACEMENT): Replace tab in the string […]
September 29, 2021
We get a lot of questions on the differences in Spark applications, jobs, stages and tasks. Also we see there is a lot of misunderstanding about […]
September 27, 2021
Pretty simple problem with a simple solution. Solution CURRENT_DATE will give the current date and CURRENT_TIMESTAMP will give you the date and time. 0: jdbc:hive2://ms2.hirw.com:2181,wk1.hirw.co> SELECT […]
September 24, 2021
It is quite common to see 200 tasks in one of your stages and more specifically at a stage which requires wide transformation. The reason for […]
September 22, 2021
Sometimes you would have a need to list all the columns in a table. You can do this pretty easily if you are using a RDBMS […]
September 20, 2021
Both partitioning and bucketing are techniques in Hive to organize the data efficiently so subsequent executions on the data works with optimal performance. Partitioning Let’s take […]
September 17, 2021
Let’s see this with an example. Here is our series of instructions in our Spark code. Let’s see how Spark decide on stages and tasks with […]
September 15, 2021
Accumulators are like global variables in Spark application. In the real world, accumulators are used as counters and keep to keep track of something at an […]
September 13, 2021
Use cat followed by a head or tail to see few lines from the top or end of the file. Few lines from top of the […]
September 10, 2021
Hive provides 3 options to order or sort the result of records – order by, sort by, cluster by and distribute by. Which option you choose […]
September 8, 2021
It is a quite common need to find the available free space in HDFS. There are a couple of ways to find this information. Using cluster […]
