how to override column names in csv file with custom schema? - apache-spark

Question

I have a CSV file with headers and data like this:
Date,Transaction,Name,Memo,Amount
12/31/2018,DEBIT,Amazon stuff,24000978364666403396802,-62.48
I want to override the column names to be like this:
transaction,credit_debit,description,memo,amount
Here is how I manually specify the schema I want to use and then read the file:
StructType schema = DataTypes.createStructType(new StructField[] {
DataTypes.createStructField("transaction_date", DataTypes.TimestampType, true),
DataTypes.createStructField("credit_debit", DataTypes.StringType, true),
DataTypes.createStructField("description", DataTypes.StringType, true),
DataTypes.createStructField("memo", DataTypes.StringType, true),
DataTypes.createStructField("amount", DataTypes.DoubleType, true)
});
String csvPath = "input/mytransactions.csv";
DataFrameReader dataFrameReader = spark.read();
Dataset<Row> dataFrame =
dataFrameReader
.format("org.apache.spark.csv")
.option("header","true")
.option("inferSchema", false)
.schema(schema)
.csv(csvPath);
dataFrame.show(20);
But when I do, the actual column values are null when I read the file.
+----------------+------------+-----------+----+------+
|transaction_date|credit_debit|description|memo|amount|
+----------------+------------+-----------+----+------+
| null| null| null|null| null|
| null| null| null|null| null|
| null| null| null|null| null|
Any idea what I'm doing incorrectly?

Answer 1

Problem is with Date Column and you are missing an option on csv called dateFormat.
Code below.
StructType schema = DataTypes.createStructType(new StructField[] {
DataTypes.createStructField("transaction_date", DataTypes.DateType, true),
DataTypes.createStructField("credit_debit", DataTypes.StringType, true),
DataTypes.createStructField("description", DataTypes.StringType, true),
DataTypes.createStructField("memo", DataTypes.StringType, true),
DataTypes.createStructField("amount", DataTypes.DoubleType, true)
});
Dataset<Row> dataFrame =
dataFrameReader
.format("org.apache.spark.csv")
.option("header","true")
.option("dateFormat", "MM/dd/YYYY")
.option("inferSchema", false)
.schema(schema)
.csv(csvPath);

Answer 2

I wanted to rename the columns. This does it:
Dataset<Row> dataFrame =
dataFrameReader
.format("org.apache.spark.csv")
.option("header","true")
.option("inferSchema", true)
.csv(csvPath);
// Rename Columns
dataFrame = dataFrame.toDF("transaction_date","debit_credit", "description", "memo", "amount");