I am calling an API that returns a gzip file in it's body. The problem I am having is I don't know how to extract specific data from that csv file as I need to use it further in my code. This is my code:
const response = await axios.request({
method: 'GET',
url: api,
decompress: true,
responseType: 'arraybuffer',
headers: {
Authorization: `Bearer ${token}`
}
}).then(resp => {
zlib.gunzip(resp.data,(err,output) => {
console.log(output);
})
});
The console log in the zlib callback displays the file as I want it but i don't know how to assign that value to response once the gunzip is finished.
Do you want the variable response to be set to output? If so you can just return output using an intermediary variable. I just called it out, but of course you can rename it to whatever seems fitting. Of course, you also need to handle the error cases, which I did not do below, but that should be a relatively easy task.
const response = await axios.request({
method: 'GET',
url: api,
decompress: true,
responseType: 'arraybuffer',
headers: {
Authorization: `Bearer ${token}`
}
}).then(resp => {
let out;
zlib.gunzip(resp.data,(err,output) => {
out = output;
})
return out;
});
Above the const response is set to the variable output in the gunzip func.
Related
In Pipedream, I have an OpenWeatherAPI request that successfully retrieves forecast data for temperature on a given day.
My goal is to send this forecast to send a PUT request to a service called Apilio that stores a temperature value for further evaluation.
I’m able to conduct the GET and PUT HTTP requests respectively, but unable to pass the data within Pipedream. In other words, I can manually type in a temperature value in the PUT request, but unable to pass it as a variable.
Here is my script thus far:
import axios from "axios"
export default defineComponent({
async run({ steps, $ }) {
const feelslikeresponse =
steps.OpenWeatherAPI.$return_value.daily[7].feels_like.morn;
return feelslikeresponse;
const { data } = await axios({
method: "PUT",
url: "https://api.apilio.com/api/v1/numeric_variables/(my UUID)",
})
return data.species
}, })
var data = { "numeric_variable": { "value": feelslikeresponse } };
let config = {
"method": 'put',
"url": 'https://api.apilio.com/api/v1/numeric_variables/(my UUID)',
"headers": {
'Content-Type': 'application/json',
'Authorization': 'Basic (my header code)'
},
data : data
};
axios(config)
.then((response) => {
console.log(JSON.stringify(response.data));
})
.catch((error) => {
console.log(error);
});
Within the code preview, I am able to hover over steps.OpenWeatherAPI.$return_value.daily[7].feels_like.morn and see the temperature value, but subsequent references to it are lost.
I am still new to coding and would appreciate any pointers or guidance.
Thanks to Wesley for the lesson and scrutinizing syntax. Between this and some hints from the Pipedream forums, how Axios works, and other documentation, I worked out the solution as follows:
import {axios} from "#pipedream/platform";
export default defineComponent({
async run({ steps, $ }) {
const feelslikeresponse =
steps.OpenWeatherAPI.$return_value.daily[7].feels_like.morn;
let data = JSON.stringify({
"numeric_variable":{
"value":feelslikeresponse
}
})
const { config } = await axios($,{
method: "put",
url: "(myUUID)",
headers: {
"Content-Type": "application/json",
Authorization:
"Basic (headercode)",
},
data : data
});
return feelslikeresponse;
},
});
Been spending the past few hours figuring out why Axios is doing this.
I tried to do this in the request library in nodeJS and it works fine, but Axios isn't.
Essentially what i'm doing is sending a request of XML data:
var options = {
'method': 'POST',
'url': 'https://rvices.svc',
'headers': {
'Content-Type': 'text/xml; charset=utf-8',
'SOAPAction': 'http://etProject'
},
data: xmlData};
with XMLData looking simliar to :
let xmlData = `<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:com="http://p" xmlns:pws="http://"
etc etc (it has mostly private data in it so I stripped most of it out).
when I try to use axios to get this
const testData = await axios(options)
i get returned a whole lot of code that looks like this:
'���_\x1B�\f�10�\x17�Ч�{O�\x00��5h�S�������\x0F���s�(+Ғ�\x0F�����m�\x15\x01\x13��6b��\x06%\x00��\x15p8<;��W�4����\x0B���\x01���e�\x7FvZ�{���Ï������\x06��-�z��\x01}�!�r�A�\x13��\x11O�w6ũ���{�\x03����;{����\x01\x7Fy��KoՎ���\x1Bߚe��W��mЇ�qD�a��[�7Ӄ���#��F<\x1C/mF�{\x03�h��#�\x16�\x11\x1F\x1F�L9\x0FM\x8A\x0E�\x
17�h���\x03�4�7�f=bj*8�p�\x13_�\x17�5���_�Ӑ�|M>����\r��F�8q�iE�#��\x0E?�v�������O�xq3���x�Q�튱\x1F?G&HG9��6���V\x1B⫯Ev\x01rc\x13\x10�\'�7��`�Ii��x�~LM6�#˒74##�����f�*\x7F\x16(5|\x1CWl��\x07\t\x1F��z�\x15\x00\x1B��4�\x13���LCTG�\x1FI�����\fec�h\x02�~��i`�:Ғ�\x0F���y\b#�]V��g��Ӈ�\x14|���6~\x19~c`�/�O���M\x01��k\x
10�\'+���\x07S\r?|��T�A�\x0FӒ�\x0F��ܷ\'.s�!>�tbX\x05�\fs\x18�\r�"`���\x10lV٠\x05#ܲ�\x02\x0E\x07h���\n' +
'���[�7}�>54 r�����ʦ\x15�\x17��\x0E:
that is the right amount of characters (100k +) but jumbled
compared to doing this with request which returns the xml back I expect ala:
</b:ProjectTaskTypeDetail></b:PwsProjectTaskTypeElement><b:PwsProjectTaskTypeElement><b:ProjectTaskTypeDetail><b:ExternalSystemIdentifier i:nil="true"/><b:ProjectTaskTypeId i:nil="true"/><b:ProjectTaskTypeUid>5776</b:ProjectTaskTypeUid><b:ProjectTaskTypeName>Faon</b:Proj
ectTaskTypeName>
one thing I noticed is axios is breaking my request up into multiple lines like this:
'<com:PwsProjectRef><com:ProjectCode>201268</com:ProjectCode></com:PwsProjectRef>\n' +
'\n' +
'<com:PwsProjectRef><com:ProjectCode>210115-01</com:ProjectCode></com:PwsProjectRef>\n' +
'\n' +
even though there's no \n's in my request or breaks like that.
So i'm wondering if anyone has ran into this before and knows how to solve it?
Request is working but request (from what I can tell?) doesn't work with asynch code (i'm probably wrong about this)
Sorry for the vagueness!
You should be using the responseType config option to set the expected response which reflects the Accept HTTP header and not the Content-Type one:
const options = {
method: 'POST',
url: 'https://rvices.svc',
headers: {
'Content-Type': 'text/xml; charset=utf-8',
'SOAPAction': 'http://etProject'
},
data: xmlData,
responseType: 'document',
responseEncoding: 'utf8'
};
const testData = await axios(options);
The response is being returned in a compressed binary format, but
Axios does not understand the compression format that is being returned from the server. Try forcing the response to a specific compression algorithm like 'deflate' which axios understands. 'gzip' may also work.
The axios 'decompress' option tells axios to automatically decompress the binary data.
var options = {
'method': 'POST',
'url': 'https://rvices.svc',
'headers': {
'Accept-Encoding': 'deflate'
'Content-Type': 'text/xml; charset=utf-8',
'SOAPAction': 'http://etProject'
},
data: xmlData,
decompress: true
};
<!-- end snippet -->
In v1.2.1, it is fixed this error.
try with this code
Save as get-data.js file
const axios = require("axios");
const getData = async () => {
try {
const resp = await axios.get('your xml URL',
{
headers: {
'Accept-Encoding': 'application/xml',
}
}
);
console.log(resp.data);
} catch (err) {
// Handle Error Here
console.error(err);
}
};
getData()
npm install axios
node get-data.js
I have a external API request as given below.
Now I need to write this postman API into an axios API call. But I tried to do many alternative things, but nothing seems to work.
The below code explain the current code I tried to do.
const url = `${this._url}/rest/v1.0/files?project_id=${projectId}`;
const response = await Axios.default.post(
url,
{
file: {
parent_id: +parentId,
data: file,
},
},
{
headers: {
Authorization: `Bearer ${updatedToken}`,
'Content-Type': 'multipart/form-data',
'Procore-Company-Id': company.id,
},
maxBodyLength: Infinity,
maxContentLength: Infinity,
}
);
using form-data
const form = new FormData();
form.append( 'my_file', fs.readFileSync('/foo/bar.jpg') );
// In Node.js environment you need to set boundary in the header field 'Content-Type' by calling method `getHeaders`
const formHeaders = form.getHeaders();
axios.post('http://example.com', form, {
headers: {
...formHeaders,
},
})
.then(response => response)
.catch(error => error)
I want to download the first megabyte of a link, and then upload that to another server without putting any data on the disk.
What I have tried is this:
axios({
method: 'get',
url: "#DIRECT_DOWNLOAD_LINK#",
responseType: 'stream',
}).then((response) => {
let sizeCounter = 0
response.data.on('data', (newChunk) => {
sizeCounter += newChunk.length
if(sizeCounter > 1024000){
response.data.emit('end')
}
})
let formData = new FormData()
form.append('file', response.data)
axios({method: 'POST', data: formData, url: '#UPLOAD_URL#' headers: {...formData.getHeaders()}}).then((response) => {
console.log(response.data)
})
})
What happens here is that the first request looks like it has stopped, since the data event doesn't get called anymore. However, the next request only gets sent after the first request has finished downloading the whole file!
I have tried calling response.data.destroy() as well, but that doesn't work either.
I have an API endpoint that lets the client post their csv to our server then post it to someone else server. I have done our server part which save uploaded file to our server, but I can't get the other part done. I keep getting error { message: 'File not found', code: 400 } which may mean the file never reach the server. I'm using axios as an agent, does anyone know how to get this done? Thanks.
// file = uploaded file
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
Update
As axios doc states as below and the API I'm trying to call requires a file
// data is the data to be sent as the request body
// Only applicable for request methods 'PUT', 'POST', and 'PATCH'
// When no transformRequest is set, must be of one of the following types:
// - string, plain object, ArrayBuffer, ArrayBufferView, URLSearchParams
// - Browser only: FormData, File, Blob
// - Node only: Stream, Buffer
Is there any way to make axios send a file as a whole? Thanks.
The 2 oldest answers did not work for me. This, however, did the trick:
const FormData = require('form-data'); // npm install --save form-data
const form = new FormData();
form.append('file', fs.createReadStream(file.path));
const request_config = {
headers: {
'Authorization': `Bearer ${access_token}`,
...form.getHeaders()
}
};
return axios.post(url, form, request_config);
form.getHeaders() returns an Object with the content-type as well as the boundary.
For example:
{ "content-type": "multipart/form-data; boundary=-------------------0123456789" }
I'm thinking the createReadStream is your issue because its async. try this.
Since createReadStream extends the event emitter, we can "listen" for when it finishes/ends.
var newFile = fs.createReadStream(file.path);
// personally I'd function out the inner body here and just call
// to the function and pass in the newFile
newFile.on('end', function() {
const form_data = new FormData();
form_data.append("file", newFile, "filename.ext");
const request_config = {
method: "post",
url: url,
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios(request_config);
});
This is what you really need:
const form_data = new FormData();
form_data.append("file", fs.createReadStream(file.path));
const request_config = {
headers: {
"Authorization": "Bearer " + access_token,
"Content-Type": "multipart/form-data"
},
data: form_data
};
return axios
.post(url, form_data, request_config);
In my case, fs.createReadStream(file.path) did not work.
I had to use buffer instead.
const form = new FormData();
form.append('file', fs.readFileSync(filePath), fileName);
const config = {
headers: {
Authorization: `Bearer ${auth.access_token}`,
...form.getHeaders(),
},
};
axios.post(api, form.getBuffer(), config);
I have made an interceptor you can connect to axios to handle this case in node: axios-form-data. Any feedback would be welcome.
npm i axios-form-data
example:
import axiosFormData from 'axios-form-data';
import axios from 'axios';
// connect axiosFormData interceptor to axios
axios.interceptors.request.use(axiosFormData);
// send request with a file in it, it automatically becomes form-data
const response = await axios.request({
method: 'POST',
url: 'http://httpbin.org/post',
data: {
nonfile: 'Non-file value',
// if there is at least one streamable value, the interceptor wraps the data into FormData
file: createReadStream('somefile'),
},
});
// response should show "files" with file content, "form" with other values
// and multipart/form-data with random boundary as request header
console.log(response.data);
I had a same issue, I had a "pdf-creator-service" for generate PDF document from html.
I use mustache template engine for create HTML document - https://www.npmjs.com/package/mustache
Mustache.render function returns html as a string what do I need to do to pass it to the pdf-generator-service ? So lets see my suggestion bellow
//...
async function getPdfDoc(props: {foo: string, bar: string}): Promise<Buffer> {
const temlateFile = readFileSync(joinPath(process.cwd(), 'file.html'))
mustache.render(temlateFile, props)
const readableStream = this.getReadableStreamFromString(htmlString)
const formData = new FormData() // from 'form-data'
formData.append('file', options.file, { filename: options.fileName })
const formHeaders = formData.getHeaders()
return await axios.send<Buffer>(
{
method: 'POST',
url: 'https://pdf-generator-service-url/pdf',
data: formData,
headers: {
...formHeaders,
},
responseType: 'arraybuffer', // ! important
},
)
}
getReadableStreamFromString(str: string): Readable {
const bufferHtmlString = Buffer.from(str)
const readableStream = new Readable() // from 'stream'
readableStream._read = () => null // workaround error
readableStream.push(bufferHtmlString)
readableStream.push(null) // mark end of stream
return readableStream
}
For anyone who wants to upload files from their local filesystem (actually from anywhere with the right streams architecture) with axios and doesn't want to use any external packages (like form-data).
Just create a readable stream and plug it right into axios request function like so:
await axios.put(
url,
fs.createReadStream(path_to_file)
)
Axios accepts data argument of type Stream in node context.
Works fine for me at least in Node v.16.13.1 and with axios v.0.27.2