Hi ,
I am trying to read a csv file with one column has double quotes like below.
James,Butt,“Benton, John B Jr”,6649 N Blue Gum St
Josephine,Darakjy,“Chanay, Jeffrey A Esq”,4 B Blue Ridge Blvd
Art,Venere,“Chemel, James L Cpa”,8 W Cerritos Ave #54
Lenna,Paprocki,Feltz Printing Service,639 Main St,Anchorage
Donette,Foller,Printing Dimensions,34 Center St,Hamilton
Simona,Morasca,“Chapman, Ross E Esq”,3 Mcauley Dr
I am using the below code to keep the double quotes as its from the csv file.(few rows having double quotes and few dont)
val df_usdata = spark.read.format(“com.databricks.spark.csv”)//
.option(“header”,“true”)//
.option(“quote”,""")//
.load(“file:///E://data//csvdata.csv”)
df_usdata.show(false)
But it didn’t preserve the double quotes inside the dataframe but it should be.
The .option(“quote”,""") is not working. Am using Spark 2.3.1 version.
The output should be like below.
±---------±--------±------------------------±--------------------+
|first_name|last_name|company_name |address |
±---------±--------±------------------------±--------------------+
|James |Butt |“Benton, John B Jr” |6649 N Blue Gum St |
|Josephine |Darakjy |“Chanay, Jeffrey A Esq” |4 B Blue Ridge Blvd |
|Art |Venere |“Chemel, James L Cpa” |8 W Cerritos Ave #54 |
|Lenna |Paprocki |Feltz Printing Service |639 Main St |
|Donette |Foller |Printing Dimensions |34 Center St |
|Simona |Morasca |“Chapman, Ross E Esq” |3 Mcauley Dr |
±---------±--------±------------------------±--------------------+
Regards,
Dinesh Kumar