How to make Hive recursively read files from all the sub directories?

Let’s say you have a Hive table and the Hive table is pointing at a location or directory which has several sub directories and each subdirectories has files underneath it.

When you query the table however, Hive is only reading the files at the top level folder and ignoring all the files under the subdirectories.

Solution

There are two properties you need to make sure that is set during Hive execution. If recursive directories/files are common in your environment. Make sure to add the below properties to the hive-site.xml rather than setting this property at the application level.

SET hive.mapred.supports.subdirectories=TRUE; 
SET mapred.input.dir.recursive=TRUE;

Also note that these properties are not table specific properties they are Hive execution environment specific properties.