AWS Redshift Spectrum not working with apache parquet files - python-3.x

Question

following is my sample csv file.
id,name,gender
1,isuru,male
2,perera,male
3,kasun,male
4,ann,female
i converted above csv file into apache parquet using pandas library. following is my code.
import pandas as pd
df = pd.read_csv('./data/students.csv')
df.to_parquet('students.parquet')
after that i uploaded the parquet file into the s3 and created a external table like below.
create external table imp.s1 (
id integer,
name varchar(255),
gender varchar(255)
)
stored as PARQUET
location 's3://sample/students/';
after that i just run select query, but i got following error.
select * from imp.s1
Spectrum Scan Error. File 'https://s3.ap-southeast-2.amazonaws.com/sample/students/students.parquet'
has an incompatible Parquet schema for column 's3://sample/students.id'.
Column type: INT, Parquet schema:\noptional int64 id [i:0 d:1 r:0]
(s3://sample/students.parquet)
Could you please help me to figure out what's the problem in here ?

Answer 1

For NULLable integer values, Pandas use the dtype Int64 that correspond to Bigint in Parquet Amazon S3.
Parquet Amazon S3 File Data Type
Transformation
Description
Int32
Integer
-2,147,483,648 to 2,147,483,647 (Precision of 10, scale of 0)
Int64
Bigint
-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (Precision of 19, scale of 0)
You need to explicitly set the column type of id when calling pandas.read_csv.
df = pd.read_csv('./data/students.csv', dtype={'id': 'int32'})