May 19, 2021
It is a pretty common use case to find the list of duplicate elements or rows in a Spark DataFrame and it is very easy to […]
May 17, 2021
In a Big Data platform, when you manage a lot of data, it is quite inevitable at some point that data could be corrupted. This post […]
May 14, 2021
Both explode and posexplode are User Defined Table generating Functions. UDTFs operate on single rows and produce multiple rows as output. explode() There are 2 flavors […]
May 12, 2021
This is a very common use case. Consumer offset is used to track the messages that are consumed by consumers in a consumer group. A topic […]
May 10, 2021
We get this question a lot so we thought we would write a small post to answer this question. Spark leverages Hadoop’s InputFileFormat to read files […]
May 7, 2021
HiveServer (or HiveSerer1) was introduced when Hive first came out and it had several limitations. HiveServer2 was later introduced with Hive 0.11 and aimed to solve […]
May 5, 2021
Filtering based on a range like greater than, less than, greater than equal etc. are pretty common requirements when you work with data. In this post […]
May 3, 2021
Both map and mapPartitions are narrow transformation functions. Both functions don’t trigger a shuffle. Let’s say our RDD has 5 partitions and 10 elements in each […]
April 30, 2021
Hive has 3 different types of functions – User Defined Function (UDF), User Defined Aggregate Function (UDAF) and User Defined Table generating Function (UDTF). User Defined […]
April 28, 2021
A common question with a simple solution. Solution Use the below YARN command to list all applications that are running in YARN. yarn application -appStates RUNNING […]
April 26, 2021
There is a lot of confusion when it comes to consumer offset, what is the purpose for it and how to change the offset. The goal […]
April 23, 2021
It is pretty common in certain Hadoop distributions to get the below error when you attempt to start resource manager service or other services. ERROR [main] […]
April 21, 2021
Let’s say you created a table with one of the columns as STRING and after a couple of days you realized the column should actually be […]
April 19, 2021
reduceByKey() reduceByKey() has the below properties The result of the combination (e.g. a sum) is of the same type that the values The operation when combined […]
April 16, 2021
This post will list out the steps to properly remove a node from a Hadoop cluster. It is not advisable to just shut down the node […]
April 14, 2021
We can specify conditional expressions like OR, AND using the Query expression during search in Elasticsearch. We have an index named account and in the index […]
April 12, 2021
This is a very common requirement and it has an easy solution. Solution aws s3api get-object to get the object and use to –range option to […]
April 9, 2021
If you have read about MapReduce you know what a word count problem is. Word count is simply counting the number of words in a dataset. […]
April 7, 2021
Both reduceByKey and groupByKey result in wide transformations which means both triggers a shuffle operation. The key difference between reduceByKey and groupByKey is that reduceByKey does […]
April 5, 2021
By default the messages you can send and manage in Kafka should be less than 1 MB. To increase this limit there are few properties you […]
