I'm using node.js to try to upload a csv file via slackAPI's upload file method. The method is post. I'm unsure how to make this possible because if I use the content argument instead of the file, I get the error:
{ ok: false, error: 'invalid_array_arg' }
If I use the file aargument, I still get the error:
{ ok: false, error: 'invalid_array_arg' }
There are multiple fault points in this code and I've tried to test each one but I'm sure I'm missing some information here. Here's the uploadFile Method that I created:
function uploadFile(file){
console.log(botToken);
axios.post('https://slack.com/api/files.upload', qs.stringify({token: botToken, file: file, channels: 'testing'}))
.then(function (response) {
var serverMessage = response.data;
console.log(serverMessage);
console.log("inside file upload function");
})
}
here's how I call the method:
var file = fs.createReadStream(__dirname + '/' + csvFilePath); // <--make sure this path is correct
console.log(__dirname + '/' + csvFilePath);
uploadFile(file);
And finally the output:
Bot has started!
C:\Users\i502153\WebstormProjects\slackAPIProject/accessLogs/CSV/1548430592860output.csv*
{ ok: false, error: 'invalid_array_arg' }
inside file upload function
What am I doing wrong and how to rectify this?
Links:
https://api.slack.com/methods/files.upload
https://www.npmjs.com/package/axios
Your solution won't work because you are attempting to take a stream object (file) and stringify it into a query string, which is just going to insert the nonsense string "[object]" into the query. It won't actually stream data to Slack.
Axios, unfortunately, doesn't work in node exactly like it does in the browser, and their docs can be a little confusing.
I would suggest an approach like this (untested):
const axios = require('axios');
const FormData = require('form-data');
function uploadFile(file) {
const form = new FormData();
form.append('token', botToken);
form.append('channels, 'testing');
form.append('file', file, 'optionalfilenamehere');
return axios.post('https://slack.com/api/files.upload', form, {
headers: form.getHeaders()
}).then(function (response) {
var serverMessage = response.data;
console.log(serverMessage);
console.log('inside file upload function');
});
}
I adapted this code from the suggestion in ticket https://github.com/axios/axios/issues/1006#issuecomment-320165427, there may be other helpful comments there as well if you run into issues. Good luck!
EDIT: For people reading this later, for a similar approach using request instead of axios, see related question Slack API (files.upload) using NodeJS.
Related
I´ve got a weird problem.
Using Node, React, Express, MongoDB -> MERN Stack.
So my page generates a PDF file which then gets send to the backend (as blob data) and is being stored on there.
The problem I have, now I need to send a payment ID along with that blob data to save the order in the data base. I need both in one post request, to make it as smooth as possible:
await axios
.post(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
like so.
Before, when I just sent the blob data, I could simply access the data in the backend by writing:
exports.createCashOrder = async (req, res) => {
const { filename } = req.file; // THIS RIGHT HERE
const fileHash = await genFileHash(filename);
try {
await saveOrder(filename, fileHash, "cash", paymentId);
//await sendOrderCreatedEmail(req.body, fileHash);
//await sendOrderReceivedConfirmEmail(req.body, fileHash);
res.send({ filename: filename });
}
But that doesn't work anymore. I dont have access to that file object anymore when sending that request object.
Neither by trying
req.body.blobData
req.body.blobData.file
req.file
Any idea how to achieve that, except from making two seperate post requests?
Glad for any help, cheers!
Send the data as a form
await axios
.postForm(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
And then use multer middleware to handle the form in express.
I need to upload a file to an API-Gateway. After adding some meta information, the file should be send to another (micro) service (as Content-Type: multipart/form-data). I am having some problems to build a FormData object within the API-Gateway. I do not want to persist the file on the gateway, so I am basically just trying to pass it through.
For creating the formData-object, I am using Form-Data
This is what a tried:
// Controller
#Post()
#UseInterceptors(FileInterceptor('file'))
async create(#Res() res, #UploadedFile('file') file, #Body() body: any) {
return await this.someService.create(file);
}
// Service
async create(file: any) {
const formData = new FormData();
formData.append('file', file);
formData.append('key', 'value');
const formHeaders = formData.getHeaders();
try {
const result = await this.httpService
.post('http://some-other-service/import', formData , {
headers: {
...formHeaders,
},
})
.toPromise();
return result.data;
} catch (e) {
throw new BadGatewayException();
}
}
This results in the following error:
TypeError: source.on is not a function
at Function.DelayedStream.create (/usr/app/node_modules/delayed-stream/lib/delayed_stream.js:33:10)
at FormData.CombinedStream.append (/usr/app/node_modules/combined-stream/lib/combined_stream.js:44:37)
at FormData.append (/usr/app/node_modules/form-data/lib/form_data.js:74:3)
at ImportService.<anonymous> (/usr/app/src/import/import.service.ts:47:18)
This question is a bit old, but someone might benefit from this solution.
The problem is that you are passing the whole #UploadedFile object to the formData.append method. The #UploadedFile object contains the data from from the file, but also mimetype, size, fieldname ('file' in this case), originalname (the original file name), etc.
You need to pass the actual contents of the file you are trying to upload to the formData.append method.
So to make it work, use
formData.append('file', file.buffer);
//OR
formData.append('file', file.buffer, file.originalname);
I am trying to use busboy inside a lambda function to process a post request which is supposed to upload an image file. I notice that the whole file content is not making it to be parsed by busboy.
I tried changing the call to busboy.write to just use 'base64' since it looks like the file arrives in binary, but that didn't work either.
my client code
const formData = new FormData();
formData.append("file", params.file, params.file.name);
const request = new XMLHttpRequest();
request.open("POST", "https://myapi/uploadphoto");
request.setRequestHeader('Authorization', this.props.idToken);
request.send(formData);
my lambda code
function getFile(event) {
const busboy = new Busboy({headers: event.headers});
const result = {};
return new Promise((resolve, reject) => {
busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
file.on('data', data => {
result.content = data;
console.log("got data... " + data.length + ' bytes');
});
file.on('end', () => {
result.filename = filename;
result.contentType = mimetype;
resolve(result);
});
});
busboy.on('error', error => reject(error));
busboy.write(event.body, event.isBase64Encoded ? 'base64' : 'binary');
busboy.end();
});
}
When trying with an example photo, I notice that the "got data" console log is showing me that I am not receiving the whole file. The file I am using is 229707 bytes but the console log says that it received 217351 bytes.
I am wondering if I am using busboy wrong or if this is some quirk of lambda + api gateway. Any ideas or help troubleshooting is much appreciated.
I was struggling with this issue too, but in the end it was a problem with API Gateway.
I was able to solve the problem by adding multipart/form-data as a binary media type inside the settings of API Gateway.
To do it go to API Gateway > "Your API" > Settings > Add Binary Media Type and add multipart/form-data.
After that, deploy again your API and it should work.
I hope this helps anyone!
I am currently working on parsing a remote csv product feed from a Node app and would like to use Papa Parse to do that (as I have had success with it in the browser in the past).
Papa Parse Github: https://github.com/mholt/PapaParse
My initial attempts and web searching haven't turned up exactly how this would be done. The Papa readme says that Papa Parse is now compatible with Node and as such Baby Parse (which used to serve some of the Node parsing functionality) has been depreciated.
Here's a link to the Node section of the docs for anyone stumbling on this issue in the future: https://github.com/mholt/PapaParse#papa-parse-for-node
From that doc paragraph it looks like Papa Parse in Node can parse a readable stream instead of a File. My question is;
Is there any way to utilize Readable Streams functionality to use Papa to download / parse a remote CSV in Node some what similar to how Papa in the browser uses XMLHttpRequest to accomplish that same goal?
For Future Visibility
For those searching on the topic (and to avoid repeating a similar question) attempting to utilize the remote file parsing functionality described here: http://papaparse.com/docs#remote-files will result in the following error in your console:
"Unhandled rejection ReferenceError: XMLHttpRequest is not defined"
I have opened an issue on the official repository and will update this Question as I learn more about the problems that need to be solved.
After lots of tinkering I finally got a working example of this using asynchronous streams and with no additional libraries (except fs/request). It works for remote and local files.
I needed to create a data stream, as well as a PapaParse stream (using papa.NODE_STREAM_INPUT as the first argument to papa.parse()), then pipe the data into the PapaParse stream. Event listeners need to be implemented for the data and finish events on the PapaParse stream. You can then use the parsed data inside your handler for the finish event.
See the example below:
const papa = require("papaparse");
const request = require("request");
const options = {/* options */};
const dataStream = request.get("https://example.com/myfile.csv");
const parseStream = papa.parse(papa.NODE_STREAM_INPUT, options);
dataStream.pipe(parseStream);
let data = [];
parseStream.on("data", chunk => {
data.push(chunk);
});
parseStream.on("finish", () => {
console.log(data);
console.log(data.length);
});
The data event for the parseStream happens to run once for each row in the CSV (though I'm not sure this behaviour is guaranteed). Hope this helps someone!
To use a local file instead of a remote file, you can do the same thing except the dataStream would be created using fs:
const dataStream = fs.createReadStream("./myfile.csv");
(You may want to use path.join and __dirname to specify a path relative to where the file is located rather than relative to where it was run)
OK, so I think I have an answer to this. But I guess only time will tell. Note that my file is .txt with tab delimiters.
var fs = require('fs');
var Papa = require('papaparse');
var file = './rawData/myfile.txt';
// When the file is a local file when need to convert to a file Obj.
// This step may not be necissary when uploading via UI
var content = fs.readFileSync(file, "utf8");
var rows;
Papa.parse(content, {
header: false,
delimiter: "\t",
complete: function(results) {
//console.log("Finished:", results.data);
rows = results.data;
}
});
Actually you could use a lightweight stream transformation library called scramjet - parsing CSV straight from http stream is one of my main examples. It also uses PapaParse to parse CSVs.
All you wrote above, with any transforms in between, can be done in just couple lines:
const {StringStream} = require("scramjet");
const request = require("request");
request.get("https://srv.example.com/main.csv") // fetch csv
.pipe(new StringStream()) // pass to stream
.CSVParse() // parse into objects
.consume(object => console.log("Row:", object)) // do whatever you like with the objects
.then(() => console.log("all done"))
In your own example you're saving the file to disk, which is not necessary even with PapaParse.
I am adding this answer (and will update it as I progress) in case anyone else is still looking into this.
It seems like previous users have ended up downloading the file first and then processing it. This SHOULD NOT be necessary since Papa Parse should be able to process a read stream and it should be possible to pipe 'http' GET to that stream.
Here is one instance of someone discussing what I am trying to do and falling back to downloading the file and then parsing it: https://forums.meteor.com/t/processing-large-csvs-in-meteor-js-with-papaparse/32705/4
Note: in the above Baby Parse is discussed, now that Papa Parse works with Node Baby Parse has been depreciated.
Download File Workaround
While downloading and then Parsing with Papa Parse is not an answer to my question, it is the only workaround I have as of now and someone else may want to use this methodology.
My code to download and then parse currently looks something like this:
// Papa Parse for parsing CSV Files
var Papa = require('papaparse');
// HTTP and FS to enable Papa parse to download remote CSVs via node streams.
var http = require('http');
var fs = require('fs');
var destinationFile = "yourdestination.csv";
var download = function(url, dest, cb) {
var file = fs.createWriteStream(dest);
var request = http.get(url, function(response) {
response.pipe(file);
file.on('finish', function() {
file.close(cb); // close() is async, call cb after close completes.
});
}).on('error', function(err) { // Handle errors
fs.unlink(dest); // Delete the file async. (But we don't check the result)
if (cb) cb(err.message);
});
};
download(feedURL, destinationFile, parseMe);
var parseMe = Papa.parse(destinationFile, {
header: true,
dynamicTyping: true,
step: function(row) {
console.log("Row:", row.data);
},
complete: function() {
console.log("All done!");
}
});
Http(s) actually has a readable stream as parameter in the callback, so here is a simple solution
try {
var streamHttp = await new Promise((resolve, reject) =>
https.get("https://example.com/yourcsv.csv", (res) => {
resolve(res);
})
);
} catch (e) {
console.log(e);
}
Papa.parse(streamHttp, config);
const Papa = require("papaparse");
const { StringStream } = require("scramjet");
const request = require("request");
const req = request
.get("https://example.com/yourcsv.csv")
.pipe(new StringStream());
Papa.parse(req, {
header: true,
complete: (result) => {
console.log(result);
},
});
David Liao's solution worked for me, I did tweak it a little bit since I am using local file. He did not include the example how to solve the file access in node if you did get Error: ENOENT: no such file or directory message in your console.
To test your actual working directory and to understand where you must point your path to console log the following, this gave me better understanding of the file location: console.log(process.cwd()).
const fs = require('fs');
const papa = require('papaparse');
const request = require('request');
const path = require('path');
const options = {
/* options */
};
const fileName = path.resolve(__dirname, 'ADD YOUR ABSOLUTE FILE LOCATION HERE');
const dataStream = fs.createReadStream(fileName);
const parseStream = papa.parse(papa.NODE_STREAM_INPUT, options);
dataStream.pipe(parseStream);
let data = [];
parseStream.on('data', chunk => {
data.push(chunk);
});
parseStream.on('finish', () => {
console.log(data);
console.log(data.length);
});
EDITED
I'm trying to structure the files.upload() API provided via Slack but am having a hard time understanding the correct format. At the moment, I am able to use the API to upload a text file but cannot for the life of me figure out how to upload an image.
Here's my issue: I have an image on my development server, let's call it image.png. I want to use the files.upload() API to post that image into a #general Slack channel. Below is the code I have that is successfully generating the image, but currently is just sending the text:
var myBarChart = new Chart(ctx).Bar(barChartData, barChartOptions);
var myBarChartDataURL = leaderboardBarChart.toBase64Image();
canvas.toBuffer(function(err, buf) {
if (err) throw err;
fs.writeFile(__dirname + "/leaderboard.png", buf);
});
bot.api.files.upload({
token: process.env.token,
title: "Image",
filename: "image.png",
filetype: "auto",
//content: "Posted with files.upload API",
file: fs.createReadStream("path/to/image_file.png"),
channels: filtered[0].id
}, function(err, response) {
if (err) {
console.log("Error (files.upload) " + err);
} else {
console.log("Success (files.upload) " + response);
};
});
When I run the code I get one of the following error:
"invalid_array_arg" which Slack details as: "The method was passed a PHP-style array argument (e.g. with a name like foo[7]). These are never valid with the Slack API."
I'm not entirely sure what to make of this error as I'm not using PHP nor anything that I can identify that would be PHP-like.
I've experimented with several different approaches for including the file path, whether using the 'fs' module, storing it in a variable, or just referencing it's absolute path (and even a relative path). I'm a bit lost and am just looking for some guidance.
I understand that this particular API uses multipart/form-data but I don't have a form. This app is strictly a NodeJS app. There is no framework (like Express) working in tandem with the main node script.
Any and all help is really appreciated. Again, just looking for some insight/guidance on what I'm missing or doing wrong.
Thanks in advance!
It looks like you'll have to go outside of Botkit's API here, since Botkit doesn't seem to support sending multipart/form-data.
Give this a try, using request directly (already in use by Botkit itself):
var request = require('request');
...
request.post({
url: 'https://slack.com/api/files.upload',
formData: {
token: bot.config.token,
title: "Image",
filename: "image.png",
filetype: "auto",
channels: filtered[0].id,
file: fs.createReadStream('test.png'),
},
}, function (err, response) {
console.log(JSON.parse(response.body));
});
I recommend you using the nodejslack.
It uses the Promises pattern, powered by Bluebird .
There is a sample code for uploading file in its documentations, here it is:
var Slack = require('nodejslack');
var fs = require('fs');
var SLACK_TOKEN = process.env.SLACK_TOKEN || 'YOUR_GENERATED_SLACK_TOKEN';
var slack = new Slack(SLACK_TOKEN);
var form = {
file: fs.createReadStream('test.csv'), // Optional, via multipart/form-data. If omitting this parameter, you MUST submit content
// content: 'Your text here', // Optional, File contents. If omitting this parameter, you must provide a `file`
filename: 'test.csv', // Required
fileType: 'post', // Optional, See more file types in https://api.slack.com/types/file#file_types
title: 'Title of your file!', // Optional
initial_comment: 'First comment about this file.', // Optional
channels: 'general' //Optional, If you want to put more than one channel, separate using comma, example: 'general,random'
};
slack.fileUpload(form)
.then(function(response){
// Slack sends a json with a boolean var ok.
// Error example : data = { ok: false, error: 'user_not_found' }
// Error example : data = { ok: true, file: 'user_not_found' }
if(!response || !response.ok){
return Promise.reject(new Error('Something wrong happened during the upload.'));
}
console.log('Uploaded Successfully:',response);
return Promise.resolve(response);
})
.catch(function(err){
return err;
});