I'm trying to pipe a file from service A trough service B into my Postman cliente. Service A builds an delivers a CSV file, and service B (nodejs) has to pipe into my client.
After researching a lot I have managed to successfully pipe the files into service B and then into Postman. Now I want to handle the ugly cases: what if the request token is invalid? What if I can't find the file?
As of this moment, I have found zero documentation or examples on how successfully handle errors while piping a request using superagent.
This is what I have so far
router.post("/csv", (req, res) => {
download_csv(req.get("Authorization"), req.body.ids)
.then((response) => {
res.sendFile(path.resolve(response));
})
.catch((err) => {
res.status(error.status).json(error.response.body);
})
});
function download_csv(token, ids) {
const stream = fs.createWriteStream("filters.csv")
let request = agent
.post(`${profiles}/api/documents/csv`)
.set("authorization", token)
.send({
ids: ids,
action: DOWNLOAD_CSV_PROFILES
})
request.on("response", res => {
// Maybe I can use abort to handle this thing, but can't figure out how!
// if (res.status !== 200) request.abort()
console.log(res.status)
})
request.on("abort", () => {
console.log("aborted")
return new Promise((resolve, reject) => {
resolve("request aborted")
})
})
request.pipe(stream)
return streamToPromise(stream);
}
function streamToPromise(stream) {
return new Promise((resolve, reject) => {
stream.on("error", (err) => {
console.log("error in error")
})
stream.on("finish", () => {
console.log("File saved")
resolve(stream.path);
});
});
}
This code handles the creation of the files correctly. When I fake the token or misspell the Authorization header, I get a correct 401 response, but a file gets written anyway with its contents being the authentication error.
Can anyway give me a hint on how to:
actually catch and manage the request when fails
in such case, how to escape the piping by going back to the express context and just returning a failed express request?
Many thanks!
If I understand you correctly, simply create the fs write stream in on('response') and make a small fix on the resultion.
function download_csv(token, ids) {
return new Promise((resolve, reject) => {
let request = agent
.post(`${profiles}/api/documents/csv`)
.set("authorization", token)
.send({
ids: ids,
action: DOWNLOAD_CSV_PROFILES
})
request.on("response", res => {
// Maybe I can use abort to handle this thing, but can't figure out how!
if (res.status === 200) {
res
.on("end", resolve)
.pipe(fs.createWriteStream("filters.csv"));
} else {
reject();
}
})
request.on("abort", reject);
});
}
I'm not sure what is the "request" you're using - but assuming it's actually the request npm module that will help.
Ideally, upload the file to a temporary directory and move it when the promise is resolved, delete on rejected. This way you'll solve the issue of partial downloads.
If you want to make any on-the-fly transforms, check out my "scramjet". It'll make everything easier with promises.
Related
I am using this <res.write()> ==> https://nodejs.org/api/http.html#responsewritechunk-encoding-callback (in nodejs)
and using this fetch ==> https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch
My situation is that I didn't see any response when using the res.write() function inside of the fetch function. for example, in the below backend code, I tried to put res.write("456") inside of the first then function below fetch, but I only see 123 returned in the frontend, no 456.
res.write("123");
fetch('http://example.com/movies.json')
.then((response) => {response.json(), res.write("456")})
.then((data) => console.log(data));
I have searched Google for a while, but didn't see anything related. My guess is that this could be because of async usage.
appreciate if someone can give suggestions.
===== update =====
res is express's res obj
async sendText(res: Response){
res.write("123")
fetch('http://example.com/movies.json')
.then((response) => {return response.json()})
.then((data) => {
console.log(data);
res.write('456');
res.end();
});
}
seeing behavior: only see 123 in the frontend.
VS
async sendText(res: Response){
res.write("123")
await fetch('http://example.com/movies.json')
.then((response) => {return response.json()})
.then((data) => {
console.log(data);
res.write('456');
res.end();
});
}
seeing behavior: can see both 123 and 456 in the frontend.
I never use await and .then together before, not fully understand the difference. searching the web rn.
You aren't checking for any errors or an unsuccessful response so response.json() may be failing, preventing anything after it from executing.
Something like this should work better for you
async sendText (res: Response) {
res.write("123");
const response = await fetch("http://example.com/movies.json");
// check for an unsuccessful response
if (!response.ok) {
const error = new Error(`${response.status} ${response.statusText}`);
error.text = await response.text();
throw error;
}
const data = await response.json();
res.write("456");
res.end(); // finalise the HTTP response
console.log(data); // log data for some reason ¯\_(ツ)_/¯
};
This will return a promise that resolves with undefined or rejects with either a networking error, HTTP error or JSON parsing error.
When calling it, you should handle any rejections accordingly
app.get("/some/route", async (res, res, next) => {
try {
await sendText(res);
} catch (err) {
console.error(err, err.text);
next(err); // or perhaps `res.status(500).send(err)`
}
});
I never use await and .then together before
Nor should you. It only leads to confusing code.
I am making an xhr request in nodejs using Axios. And I am trying to save a value from this response to a variable, but it isn't working and I am not sure why.
Code:
let redirectUrl = 'placeholder'
axios.get(url)
.then(res => {
redirectURL = res.url
console.log(res.url, "HERE")
})
.catch(err => console.log(err))
return res.render('index',{url: redirectURL})
I have tried declaring redirectUrl in both global as var and local scope, but the value isn't changing, on index when I console log this value, it logs placeholder. index is index.jsx as I am using jsx as the template engine.
Please let me know if I should provide any more information.
Thank You in advance.
axios.get is an async call, so your return will use the original (placeholder) value instead of the new one. Normally to solve this kind of asynchronicities i use async/await
async function getRedirectURL(url) {
try {
const res = await axios.get(url);
// Note that if you want some return value from axios call you access it by res.data
console.log('res.url :', res.url, ". res.data:", res.data);
return res.render('index',{url: res.data.url});
} catch (err) {
console.log(err);
throw err;
}
}
Axios is a promise-based HTTP client, So when javascript engine executes your code, it moves axios call to web API from callstack (line# 2) and then it starts to execute the last line return res.render('index',{url: redirectURL}) where redirectURL value is placeholder. You have to write all the logic in promise then method, like
axios
.get(url)
.then((data) => res.render("index", { url: data.url }))
.catch((err) => {
// error response
console.log(err);
});
I'm trying to test an endpoint that should receive a multipart/form-data. I'm sending a collection of images, which i want to process and save on the server or CDN. I'm using Jest, Express and Formidable.
Endpoint
router.post("/videos", async (req, res) => {
new formidable.IncomingForm().parse(req, (err, fields, files) => {
console.log('PARSE FORM');
if (err) {
console.error('Error', err);
throw err
}
console.log('Fields', fields);
console.log('Files', files);
for (const file of Object.entries(files)) {
console.log('FILE', file)
}
});
res.status(200).send('Created Video');
});
Test
describe("Video Endpoints", () => {
it('should create a new timelapse video', done => {
request
.post('/api/videos')
.field('file', 'some random value')
.attach('image', `${__dirname}/__mocks__/image.png`)
.then(res => {
console.log('THEN');
done();
})
});
});
When running the test it doesn't reach the formidable parse method.
If change my my attach method to this...
.attach('image', fs.readFileSync(`${__dirname}/__mocks__/xdebugcurlaccessingwpapi.png`))
It will reach the parse method but it sees the it as field and not a file.
If i make the same request but from my react app using fetch, it works perfectly fine.
What am i doing wrong? Been on this for a few days now lol.
Any help would be great.
Thanks.
I'm not entirely sure why but if you add
.set({connection: 'keep-alive'})
Then it works.
Final solution
request
.post('/api/videos')
.set({connection: 'keep-alive'})
.field('name', 'Richard')
.attach('image', mockImage)
.then(res => {
console.log('THEN');
done();
});
});
Would be good if someone has an understanding to why this is the case.
I think it might close the stream to image but can't be sure.
I am trying to use the response from a SQL Server stored procedure and pass it into another stored procedure to log the response data.
However, I'm not sure how to chain them together as it appears they each need their own connection pool.
Attempt 1
// mssql#3.3.0
exports.chain = (req, res) => {
sql.connect(config.properties).then(pool => {
return pool.request()
.execute("chain")
.then(response => {
return pool.request()
.input("param", sql.NVarChar(300), result[0][0]["response"])
.execute("chain2")
.then(result => res.send(result))
.catch(err => res.send(err))
})
.catch(err => res.send(err))
})
}
// returns {}
Attempt 2
exports.chain = (req, res) => {
sql.connect(config)
.then(pool => {
return pool.request()
.execute("chain")
}).then(result => {
return pool.request()
.input("param", sql.NVarChar(300), result[0][0]["response"])
.execute("chain2")
}).then(result => {
res.send(result)
}).catch(err => {
// ... error checks
})
sql.on('error', err => {
// ... error handler
})
}
// Throws error HTTP Status: 500, HTTP subStatus: 1013
Attempt 3
sql.connect(config.properties).then(pool => {
return pool.request()
.execute("chain")
}).then(response => {
pool.request()
.execute("chain2")
.input('param', sql.NVarChar(300), response[0][0]['response'])
.then(response => res.send(response))
.catch(err => res.send(err))
})
// Throws error HTTP Status: 500, HTTP subStatus: 1013
Timeouts might be related to
DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
How would I take the response from the first stored procedure and pass it into a second one for logging?
A connection pool is a an instance consisting of multiple TDS connections, so you should be able to connect several procedures to the same pool, as a new request only accuires one of the TDS connections. Maybe it's a flaw in your config probs.
In your third example. Your input comes after execute, so that's probably causing the error.
As a request returns a promise, you can chain them with then, as you are already doing, but variables in each then are not in the same scope, so you can't access a user, received from the user record, when you are in the then for a different request.
Alternatively, you can also nest them via callback, if you need to access variables across responses:
sql.connect(config.properties).then(pool => {
pool.request()
.execute("chain", (error, response) => {
const row = response.recordset;
pool.request()
.input('param', sql.NVarChar(300), row.foo)
.execute("chain2", (err, res) => {
const row_ = res.recordset;
// send response to client
});
});
}).catch(err => {
// error handling here
});
Best solution, would probably be to return your stored procedure as JSON, and you would only need one request anyway. Downside is, you probably need more specialized procedures.
I'm using the request library in Node.js to do a https request to get data from another service. This is called asynchronously, right? So my code keeps running before all of the data is there, correct?
My problem is that the data is needed right afterwards to calculate some things. My code throws an error during that calculation because the data from the service is undefined...
Could it be possible that the data is just not there yet? And if so, what do you do against that?
Here is a copy of the request:
const request = require('request');
request(someUrl, {"Accept": "application/json"}, (err, res, body) => {
if (err)
handleError(err);
body = JSON.parse(body);
return body;
});
This kind of situation is pretty common in react/angular/vue kinda web apps, sometimes you need the data right away. But it is not available then, after a Rest call or something it becomes available.
So, the simplest solution?
Just add a check, for example:
const calculate = (someVal)=>{
if(!someVal) return ;
//otherwise do the calculation
}
There are plenty of other ways, by mostly making the calculation async. For your function, you can do this
const promOp = function(){
return new Promise((resolve, reject) => {
request(someUrl, {"Accept": "application/json"}, (err, res, body) => {
if (err) reject(err);
body = JSON.parse(body);
resolve(body);
});
}
}
//then
promOp()
.then((body)=>{
//calculate here
})
//or can use the `Async/Await` syntax instead of then
const op = async () => {
const body = await promOp;
//calculate here
}