{"id":1373,"date":"2017-10-25T18:19:56","date_gmt":"2017-10-25T23:19:56","guid":{"rendered":"http:\/\/hadoopinrealworld.com\/?p=1373"},"modified":"2023-02-19T07:32:42","modified_gmt":"2023-02-19T13:32:42","slug":"what-is-zookeeper-and-its-use-case","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/what-is-zookeeper-and-its-use-case\/","title":{"rendered":"What is ZooKeeper and it’s Use Case"},"content":{"rendered":"

ZooKeeper is a coordination service for distributed applications. Well, what the heck is that right? Glad you asked. If you are looking to understand what is ZooKeeper and to understand it’s use case or the problem ZooKeeper is trying to solve in a little bit of depth, you have come to the right place. ZooKeeper’s documentation even doesn’t do a good job of explaining what is ZooKeeper in simple terms and it’s use case. So we thought we will give it a shot.<\/p>\n

Imagine you are a kindergarten teacher and you have 15 kids in your class. You would like to elect a class leader. You as the teacher of the class can pick a kid of your choice to be the class leader but that won\u2019t be fair to the other kids. So you tell your kids, the kid who jumps the highest will be the class leader. I know that is lame and a terrible method to elect a class leader but nevertheless, in the class leader election process, you the teacher, became the coordinator.<\/p>\n

Here in the class leader election process, the teacher is the ZooKeeper and the kids are distributed applications.<\/p>\n

\"Zookeeper<\/a><\/p>\n

ZooKeeper is an open source Apache project usually installed on 3 nodes. And the 3 nodes form the ZooKeeper cluster or quorum, the ZooKeeper cluster is then used to coordinate certain activities between distributed applications. I know that is still not clear. Let’s see how ZooKeeper is used in real world.<\/p>\n

ZooKeeper & YARN HA<\/h3>\n

In this post we will see, how ZooKeeper is used to achieve high availability in YARN? Resource Manager in YARN is a single point to failure – meaning if we have Resource Manager running on only one node and when that node fails, we have a problem and we won’t be able to run any jobs in our Hadoop cluster. To avoid Resource Manager to be a single point of failure we need to enable High Availability with Resource Manager. This means we will run another Resource Manager instance on another node in stand by mode. When the active Resource Manager fails with high availability configuration in place the stand by resource manager will take the role of the active resource manager. Now, when the active resource manager fails, we can manually failover the resource manager so the standby resource manager will become active. This means there is manual intervention involved from the Hadoop administrator to initiate the failover. In a busy and highly active production cluster, manual failover is not a great option because you will lose valuable time in contacting the administrator and initiating the failover.<\/p>\n

The other option we have is automatic failover. We can configure ZooKeeper to do an automatic failover of the resource manager when the active resource manager fails. How does ZooKeeper help or facilitate the automatic failover? There are 2 things we want to understand here.<\/p>\n

1. How ZooKeeper is used to elect the active resource manager when there are more than one resource managers? This is referred to as leader election.
\n2. How ZooKeeper handles automatic failover? That is, how does a stand by resource manager becomes active in an event the active resource manager fail.<\/p>\n

\"2<\/a><\/p>\n

If you have read so far, we promise you will love to hear what other things we have to say. \u00a0