LeaseExpiredException: No lease error on HDFS

So you got this below execution and wondering how to fix it. In this post we are going to explain the possible causes for this exception which will help you fix the issue.

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: 
No lease on /data/work/_temporary/_attempt_xxx/xx/data1/file.xxx File does not exist. 
Holder DFSClient_attempt_xxx does not have any open files. at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1557) at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1548) at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1603) at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1591) at 
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:675) at 
sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at 
java.lang.reflect.Method.invoke(Method.java:597) at 
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) at 
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) at 
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at 
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)

Possible causes and solution

Race condition – 2 sets of jobs or tasks or processes writing to the same file or directory. The folder or file is deleted or made unavailable by one process and the exception results in another process.

This exception is quite common with speculative execution in MapReduce, Spark jobs etc. Where the framework executes Map or a task speculatively because it sees a map or task running slow and kicks off a “duplicate” execution of the task hoping the duplicate will complete faster. In some instances of speculative execution, the original task will complete before the “duplicate” task, deleting the files or folders that are shared by the duplicate process as well resulting in the exception. 

You can try setting the below speculative properties to false if you frequently get the LeaseExpiredException exception.

• spark.speculation
• spark.hadoop.mapreduce.map.speculative
• spark.hadoop.mapreduce.reduce.speculative