February 15, 2021
Kafka Distributed, durable and reliable message broker which can handle high volume of real time messages coming from realtime producers. Storage for real time streaming data […]
February 12, 2021
There are several factors Spark takes into account before deciding on the type of join algorithm to use to join datasets at runtime. Spark has the […]
February 10, 2021
Let’s create RDD first. Below we are creating a RDD with some sample data. scala> val data = Seq( | (1, "Shirt", 20, […]
February 8, 2021
In this post we are going to create a S3 policy to make the S3 bucket read-only. We will be denying all users access to create […]
February 5, 2021
With Spark 3.0, we can specify the type of join algorithm we would like Spark to use at runtime. Do you like us to send you […]
February 3, 2021
In Hadoop you will find Hadoop specific types for basic types. For eg. you will find Text for String. IntWritable instead of Integer. For all primitive […]
February 1, 2021
It is quite common that you have a need to find the count of the number of documents in an Elasticsearch index. There are several ways […]
January 29, 2021
Cartesian Product Join (a.k.a Shuffle-and-Replication Nested Loop) join works very similar to a Broadcast Nested Loop join except the dataset is not broadcasted. Shuffle-and-Replication does not […]
January 27, 2021
You probably heard this – Amazon S3 is not a file system, it is an object storage. So in S3, there is no technical concept of […]
January 25, 2021
Spark spits out a lot of INFO messages in the logs. These messages stand in the way when you are troubleshooting an error and looking for […]
January 22, 2021
Shuffle Sort Merge Join, as the name indicates, involves a sort operation. Shuffle Sort Merge Join has 3 phases. Shuffle Phase – both datasets are shuffled […]
January 20, 2021
Hive stores metadata information about the tables created in Hive in a relational database like Derby, MySQL etc. The metadata information includes table name, structure of […]
January 18, 2021
This is a very common need in the day to day Big Data world with a simple solution. Solution Use the hdfs du command to get […]
January 15, 2021
Broadcast Hash Join in Spark works by broadcasting the small dataset to all the executors and once the data is broadcasted a standard hash join is […]
January 13, 2021
When you run a MapReduce or a Spark job the number of files will equal to the number of reducers involved in the MapReduce job or […]
January 11, 2021
There are 2 ways to get the list of available brokers in a Kafka cluster. Both with the help of scripts from zookeeper. Zookeeper manages the […]
January 8, 2021
Broadcast Nested Loop join works by broadcasting one of the entire datasets and performing a nested loop to join the data. So essentially every record from […]
January 6, 2021
There are so many properties in Spark that affect the way you can add jars to a Spark application. We understand it could be confusing and […]
January 4, 2021
Converting a String column to an Integer column or converting a column from one type to another is quite simple in Hive. Simply use the cast […]
January 1, 2021
Shuffle Hash Join, as the name indicates works by shuffling both datasets. So the same keys from both sides end up in the same partition or […]
