copy command in aws kinesis firehose not working - amazon

Question

I am using Kinesis firehose which loads data in S3 and creates Menifest file.
I am copying data from Menifest, but it is throwing an error.
My menifest file: mydeliverystream-2018-07-22-08-01-06-bd895f6a-4fad-485
{
"entries" : [ {
"url" : "s3://testings3/2018/07/22/07/mydeliverystream-3-2018-07-22-07-56-04-89605e0e-26bf-4017-a102-338ceb15481d",
"mandatory" : true
} ]
}
Copy command which I am using firehose
COPY redshiftproduct FROM 's3://testings3/mydeliverystream' CREDENTIALS 'aws_iam_role=arn:aws:iam::329723704744:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift' delimiter ',' MANIFEST region 'us-east-1'
Please note; I have also try by adding delimiter after manifest keyword.
Error which I am receiving on the copy command contains a syntax error.
Data file:
delhi,ac,4000,2011
Haryana,TV,5000,2001
Channai,TV,3000,2011
Mumbai,Laptop,4000,2012
new delhi,ac,5000,2012
Kolkatta,fridge,1000,2012
Kanpur,TV,2000,2013
Haryana,ac,2000,2019
Kanpur,ac,2000,2019
What am I doing wrong and how can I fix it?

Answer 1

1- You need to add FORMAT AS JSON 's3://yourbucketname/aJsonPathFile.txt'. AWS has not mentioned this already. Please note that this only works when your data is in JSON form like
{'attr1': 'val1', 'attr2': 'val2'} {'attr1': 'val1', 'attr2': 'val2'} {'attr1': 'val1', 'attr2': 'val2'} {'attr1': 'val1', 'attr2': 'val2'}
2- You also needs to verify the column order in kinesis firehouse and in csv file.and try adding
TRUNCATECOLUMNS blanksasnull emptyasnull
3- An example
COPY testrbl3 ( eventId,serverTime,pageName,action,ip,userAgent,location,plateform,language,campaign,content,source,medium,productID,colorCode,scrolltoppercentage) FROM 's3://bucketname/' CREDENTIALS 'aws_iam_role=arn:aws:iam:::role/' MANIFEST json 'auto' TRUNCATECOLUMNS blanksasnull emptyasnull;