What do the numbers on the progress bar mean in Spark shell or Spark UI?
September 6, 2021What is the difference between order by, sort by, cluster by and distribute by in Hive?
September 10, 2021It is a quite common need to find the available free space in HDFS. There are a couple of ways to find this information.
Using cluster manager
If you are using a cluster manager like Ambari or Cloudera Manager then you can go to the HDFS service and see this information in the summary section.
Below is a screenshot from Ambari.
Using dfsadmin command
Use hdfs dfsadmin command If you prefer to get this information using commands.
[hdfs@wk1 osboxes]$ hdfs dfsadmin -report Configured Capacity: 443659396096 (413.19 GB) Present Capacity: 417325324646 (388.66 GB) DFS Remaining: 413394139494 (385.00 GB) DFS Used: 3931185152 (3.66 GB) DFS Used%: 0.94% Under replicated blocks: 819 Blocks with corrupt replicas: 1 Missing blocks: 1 Missing blocks (with replication factor 1): 0
Use the below command if you are using an older version of Hadoop.
hadoop dfsadmin -report
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.
