Failed to save data to CSV

Technical Discussions Spark SQL, DataFrames and Datasets
#1

Saving the data frame to creating the folder with filename and extension

df.coalesce(1).write().format("csv").option("header", "true").csv("/tmp/5.csv");

#3

Hi

I don’t think csv is the member of org.apache.spark.sql.DataFrameWriter first you need to map your data to comma separated fields then you can save it.

See an Example below.

val avroFile = sqlContext.read.avro("/FileFormates/TextToAvro")

avroFile.map(x => x(0)+","+x(1)+","+x(2)+","+x(3)).saveAsTextFile()

Hope it will help you.

#4

I am using spark 2.1.0, the csv is inlined,

df.coalesce(1).write().format("csv").option("header", "true").csv("/tmp/5.csv");

This creating dir with file name as 5.csv