What are applications, jobs, stages and tasks in Spark?
September 29, 2021What is the difference between foreach and foreachPartition in Spark?
October 4, 2021Simple problem with a simple solution.
Solution
Use the powerful regexp_replace function to replace characters.
regexp_replace(string INITIAL_STRING, string PATTERN, string REPLACEMENT):
Replace tab in the string with an empty string
select regexp_replace(description, '\\t', '') as description
Replace special characters in the string with empty character
select regexp_replace(description, ‘[^0-9a-zA-Z]+’, ‘’) as description