How to show full column content in a Spark DataFrame?

Most often when we are trying to work with data in Spark we might want to preview the data or the solution in Spark shell right on screen. When you do so, by default, Spark will only show part of the output when the data in column is long. 

Here is an example of truncated output. It is quite easy to fix this.

scala> results.show(); 
+--------------------+ 
| col	             | 
+--------------------+ 
|2019-11-20 08:30:...| 
|2019-11-20 08:15:...| 
|2019-11-21 07:15:...| 
|2019-11-22 09:35:...|

Do you like us to send you a 47 page Definitive guide on Spark join algorithms? ===> Send me the guide

Solution

show() function takes in 2 parameters – number of rows and true/false whether to truncate output or not. By default the truncate is set to true to truncate the results. Set it false to not truncate the result. 

Quick fix

scala> results.show(200, false);

Above will not truncate the results.

Better Solution

If you are trying to print out a lot of columns, let’s say 100, it would be hard to see the output on screen. There is even a better solution in our opinion. Simply write the data to a file in JSON format. JSON gives structure to your data and you can quickly format data using an online JSON formatter like jsonlint. We find this simple technique quite handy. Below will write the file to your local file system which you can then review.

results.write.json("file:///yourPath")