headObject never throws an error when object doesn't exist - node.js

I am currently trying to check if a file exists, using aws-sdk for Amazon s3 (More precisely, the function headObject).
As I could read literally everywhere, this is the function that is supposed to be used when trying to check if a file exists (In order to then get its URL through getSignedUrl), however I can't bring it to work.
It seems that, no matter what I do, the function s3.headObject tells me that the object exists. I tried checking for existing items, non-existing items, and even in non-existing buckets : All of these got me the exact same output. I tried different ways of calling the function (Async or not, using its callback or not), but no difference.
Here is how I realize my call to the function :
var params = {
Bucket: 'BUCKET NAME',
Key: ""
}
// Some more code to determine file name, confirmed working
params.Key = 'FILE NAME'
try {
s3.headObject(params)
// Using here the file that is supposed to exist
} catch (headErr) {
console.log("An error happened !")
console.log(headErr)
}
I also tried using a callback : However, it seems that said callback was never entered. Here is what my code looked like :
var params = {
Bucket: 'BUCKET NAME',
Key: ""
}
// Some more code to determine file name, confirmed working
params.Key = 'FILE NAME'
s3.headObject(params, function(err: any, data: any) {
console.log("We are in the callback")
if (err) console.log(err, err.code)
else {
// Do things with file
}
})
console.log("We are not in the callback")
With this code, "We are in the callback" never appeared, while "We are not in the callback" was correctly appearing.
No matter what I do, no error is ever caught.
From what I understand from how the function is supposed to work, in case the file doesn't exist, it is supposed to throw an error (Then caught by my catch), thus allowing me not to create false URLs with the getSignedUrl function.
What am I doing wrong here ?
Thank you all for your answers. If you have additional questions, I'll be more than glad to answer the best I can.

This is the right way to check object existence using async/await syntax:
// Returns a promise that resolves to true/false if object exists/doesn't exist
const objectExists = async (bucket, key) => {
try {
await s3.headObject({
Bucket: bucket,
Key: key,
}).promise(); // Note the .promise() here
return true; // headObject didn't throw, object exists
} catch (err) {
if (err.code === 'NotFound') {
return false; // headObject threw with NotFound, object doesn't exist
}
throw err; // Rethrow other errors
}
};

I do try out the syntex but it doesn't work. It is in my lambda function.
params and params2 are predefined set of bucket and key.
var url = s3.getSignedUrl('getObject', params);
const objectExist = async (par) => {
try{
console.log(s3.headObject(par).response); //I honestly couldn't find any
//section in the resoonse,
// that make a DNE file different from a existing file.
const ext = await s3.headObject(par).promise((resolve, reject) =>{
console.log("bbbbbbbbbbb");
if(err) { // it keeps saying the err is not global variable.
//I am wondering why this is not defined.
//Really had no idea of what else I could put as condition.
console.log("aaaa"); //never reach here.
return reject(false);}
return resolve(true);
});
console.log(ext); //always true.
if(!ext){url = url = s3.getSignedUrl('getObject', params2, callback); }
}catch(err){
console.log("reeeeeeeeee"); //when the method failed it execute.
url = s3.getSignedUrl('getObject', params2, callback);
console.log(url); //even though I am sure that params2 are valid key, but url in the log always returned undefined.
}
};
objectExist(params);

Related

Query is not working with promise for Dynamo DB

I have a dynamo db table where I was able to insert data using node js via lambda. I am able to query from the console and I am also able to query using the cli. When using query with promise its erroring out with invoke error. Its not throwing any specific errors. IF I remove promise and run I can see that connection is successful to the db. I also tried ExpressionAttributeValues: {
":name": {"S":id}
},
even hard coding the value for id and same issue. What am I doing wrong??
import AWS from "aws-sdk"
const dyanamoDB = new AWS.DynamoDB.DocumentClient()
AWS.config.update({ region: "us-east-1" })
export const checkIFIDExist = async (id) => {
try {
const params = {
ProjectionExpression: "String1, String2",
IndexName: "String2",
KeyConditionExpression: "String2 = :name",
ExpressionAttributeValues: {
":name": id
},
TableName: 'my-table',
}
const data = await dynamoDB.query(params).promise()
console.log("Data:", data)
return "success"
}catch (err) {
throw new Error (`Failed query for ${id} `, err)
}
}
Error:
2022-08-16T20:24:09.210Z c2e0c093-2719-48b8-b0bb-4f38de3ac7b6 ERROR Invoke Error
{
"errorType": "Error",
"errorMessage": "Failed query for OE0K0I ",
"stack": [
"Error: Failed query for OE0K0I ",
" at checkIFStepFunctionIDExists (file:///var/task/src/dynamo-query.js:24:15)",
" at processTicksAndRejections (internal/process/task_queues.js:95:5)",
" at async Runtime.handler (file:///var/task/src/index.js:11:19)"
]
}
I basically deleted and trashed the project created new one and did the same stuff I had mentioned earlier in my post and instead of throwing error after catch statement console log it and I am getting the result I expected. I really couldn't tell what was I doing wrong before. #jarmond the error I posted above, I accidentally included a dependency while changing something and caused the error I provided. Thanks everyone for looking into the issue.
If the promise() function doesn't do what you expect it to do. It's worth noting, that you can actually also do the same thing with the standard Node.js promisify function.
import { DocumentClient } from "aws-sdk/clients/dynamodb";
import { promisify } from "util";
const docClient = new AWS.DynamoDB.DocumentClient()
...
const data = await promisify((cb) => docClient.query(params, cb))();
As #jarmod pointed out, there's probably something else going on though. I added some sidenotes to clarify some things that you may or may not already know.
Some sidenotes
Here are just some remarks which aren't entirely on topic but which can lead to confusion.
// don't do this, it doesn't do what you think it does.
throw new Error(`Failed query for ${id}`, err );
// this prints both a text and an object.
console.error(`Failed query for ${id}`, err);
// this throws just an error with a message
throw new Error(`Failed query for ${id}`);
// there is a 2nd argument which you can pass, which is an "options" parameter, which you can use to send a `cause` along.
throw new Error(`Failed query for ${id}`, { cause: err } );
PS:More details about it can be found it in the MDN documentation.
I haven't seen how you got it working without the promise, but if you did it like this, then it's not what you think.
try {
const params = { ... };
dynamoDB.query(params);
// this point is always reached
return "success"
}catch (err) {
// this point is never reached
}
Instead, without a promise, you would have to use a callback function.
const params = { ... };
dynamoDB.query(params, (err, data) => {
if(err) {
console.error(err);
return;
}
console.log("success", data);
});

.then statements not executing sequentially

I have an application using Node.js/Express. Within this code I have the following promise designed to check if an email already exists in my (PostGres) database:
//queries.js
const checkEmail = function(mail) {
return new Promise(function(resolve, reject) {
pool.query('SELECT * FROM clients WHERE email = $1', [mail], function(error, results) {
if (error) {
reject(new Error('Client email NOT LOCATED in database!'));
} else {
resolve(results.rows[0]);
}
}) //pool.query
}); //new promise
}
In my 'main (server.js)' script file, I have a route which is called upon submission of a 'signup' form. When the post to this route is processed...I run the script above to check if the passed email address is already located in the database, along with various other 'hashing' routines:
My code is as follows:
//server.js
const db = require('./queries');
const traffic = require('./traffic');
const shortid = require('shortid');
...
app.post('/_join_up', function(req, res) {
if (!req.body) {
console.log('ERROR: req.body has NOT been returned...');
return res.sendStatus(400)
}
var newHash, newName;
var client = req.body.client_email;
var creds = req.body.client_pword;
var newToken = shortid.generate();
var firstname = req.body.client_alias;
db.sanitation(client, creds, firstname).then(
function(direction) {
console.log('USER-SUPPLIED DATA HAS PASSED INSPECTION');
}
).then(
db.checkEmail(client).then(
function(foundUser) {
console.log('HEY THERE IS ALREADY A USER WITH THAT EMAIL!', foundUser);
},
function(error) {
console.log('USER EMAIL NOT CURRENTLY IN DATABASE...THEREFORE IT IS OK...');
}
)).then(
traffic.hashPassword(creds).then(
function(hashedPassword) {
console.log('PASSWORD HASHED');
newHash = hashedPassword;
},
function(error) {
console.log('UNABLE TO HASH PASSWORD...' + error);
}
)).then(
traffic.hashUsername(firstname).then(
function(hashedName) {
console.log('NAME HASHED');
newName = hashedName;
},
function(error) {
console.log('UNABLE TO HASH NAME...' + error);
}
)).then(
db.createUser(client, newName, newHash, newToken).then(
function(data) {
console.log('REGISTERED A NEW CLIENT JOIN...!!!');
res.redirect('/landing'); //route to 'landing' page...
},
function(error) {
console.log('UNABLE TO CREATE NEW USER...' + error);
}
))
.catch(function(error) {
console.log('THERE WAS AN ERROR IN THE SEQUENTIAL PROCESSING OF THE USER-SUPPLIED INFORMATION...' + error);
res.redirect('/');
});
}); //POST '_join_up' is used to register NEW clients...
My issue is the '.then' statements do not appear to run sequentially. I was under the impression such commands only run one after the other...with each running only when the previous has completed. This is based upon the logs which show the readout of the 'console.log' statements:
USER-SUPPLIED DATA HAS PASSED INSPECTION
PASSWORD HASHED
NAME HASHED
UNABLE TO CREATE NEW USER...Error: Unable to create new CLIENT JOIN!
USER EMAIL NOT CURRENTLY IN DATABASE...THEREFORE IT IS OK...
As mentioned previously, I am under the impression the '.then' statements should run synchronously, therefore the last statement ("USER EMAIL NOT CURRENTLY IN DATABASE...THEREFORE IT IS OK...") should in fact be after the first...before the "PASSWORD HASHED" according to the layout of the '.then' statements. Is this normal behavior...or do I have an error in my code?
Sorry for my confusion however I find '.then' statements and promises to be somewhat confusing for some reason. I thank you in advance.
TLDR - You must pass a function reference to .then() so the promise infrastructure can call that function later. You are not doing that in several places in your code.
A more specific example from your code:
You have several structures like this:
.then(db.createUser().then())
This is incorrect. This tells the interpreter to run db.createUser() immediately and pass its return result (a promise) to .then(). .then() will completely IGNORE anything you pass is that is not a function reference and your promises will not be properly chained.
Instead, you must pass a function reference to .then() something like this (not sure what execution logic you actually want):
.then(() => { return db.createUser.then()})
Then main point here is that if you're going to sequence asynchronous operations, then you must chain their promises which means you must not execute the 2nd until the first calls the function you pass to .then(). You weren't passing a function to .then(), you were executing a function immediately and then passing a promise to .then(p) which was completely ignored by .then() and your function was executed before the parent promise resolved.
FYI, sequencing a bunch of asynchronous operations (which it appears you are trying to do here) can take advantage of await instead of .then() and end up with much simpler looking code.

Jest - Mock callback in function fs.writefile

I am stucking for 3 hours on this topic. I dont find a solution how to test the if(err) branch in this code:
function createFile(data){
return new Promise(function(resolve, reject) {
try {
if(data === null || data === undefined){
throw new Error(errorMessages.noDataDefined);
}
let internalJobId = uuid.v4();
let fileName = 'project_name' + internalJobId + '.xml';
fs.writeFile(config.tmpPath + fileName, data, function (err) {
if (err){
throw new Error(err.toString());
} else {
resolve(fileName);
}
});
} catch (error) {
return reject(error);
}
});
}
This test passes but it does not call the if (err) { throw new Error(err.toString())}
I have to find a solution, how the callback returns an error, but I dont get the right solution.
test('Error', () => {
jest.mock('fs', () => ({
writeFile: jest.fn((path, data, callback) => callback(Error('some error')))
}));
return expect(createFile('Does not matter')).rejects.toThrow('some error');
});
But with this test there is even not a reject, so there is never thrown an error. I would appreciate if anyone could help me out there.
There are two problems here. One is that fs.writeFile isn't correctly mocked. Another is that createFile doesn't correctly handle errors and can't meet the expectation.
jest.mock affects modules that haven't been imported yet and hoisted to the top of the block (or above imports when used at top level). It cannot affect fs if a module that uses it has already been imported. Since fs functions are commonly used with their namespace, they can also be mocked as methods.
It should be either:
// at top level
import fs from 'fs';
jest.mock('fs', ...);
...
Or:
// inside test
jest.spyOn(fs, 'writeFile').mockImplementation(...);
...
And be asserted to make the test more specific:
expect(fs.writeFile).toBeCalledTimes(1);
expect(fs.writeFile).toBeCalledWith(...);
return expect(createFile('Does not matter'))...
Promise constructor doesn't need try..catch because it already catches all synchronous error inside it and cannot catch asynchronous errors from callbacks. For places where a promise needs to be rejected, reject can be preferred for consistency.
That an error is thrown inside fs.writeFile callback is a mistake and results in pending promise. It has no chance to reject the promise, has no chance to be caught with try..catch outside the callback and causes uncaught error.
It should be:
function createFile(data){
return new Promise(function(resolve, reject) {
if(data === null || data === undefined){
reject(new Error(errorMessages.noDataDefined));
}
let internalJobId = uuid.v4();
let fileName = 'project_name' + internalJobId + '.xml';
fs.writeFile(config.tmpPath + fileName, data, function (err) {
if (err){
reject(new Error(err.toString()); // reject(err) ?
} else {
resolve(fileName);
}
});
});
}
In order to keep nesting to minimum, parts that don't need promisification can be moved outside the constructor with the function being async:
async function createFile(data){
if(data === null || data === undefined){
throw new Error(errorMessages.noDataDefined);
}
return new Promise(function(resolve, reject) {
let internalJobId = uuid.v4();
...
There is also fs.promises API that may not need to be promisified.
Also notice that new Error(err.toString()) may be unnecessary and result in unexpected error message and so fail the assertion. The promise can be rejected with err as is. If the purpose is to remove unnecessary error information or change error stack, it should be new Error(err.message).
The solution was:
jest.spyOn(fs, 'writeFile').mockImplementation((f, d, callback) => {
callback('some error');
});
Thanks to Estus Flask!
The following three lines of code did the job for me:
import * as fs from 'fs/promises';
jest.mock('fs/promises');
jest.spyOn(fs, 'writeFile').mockImplementation( your implementation here );

Why my lambda is working only when i give a callback with some message?

I am trying to run a small snippet of lambda code where i am pushing data to S3 using firehose. Here is my snippet
const AWS = require( 'aws-sdk' );
var FIREhose = new AWS.Firehose();
exports.handler = async (event,context,callback) => {
// TODO implement
const response = {
statusCode:200,
Name:event.Name,
Value:event.Value
};
const params = {
DeliveryStreamName: 'kinesis-firehose',
Record: { Data: new Buffer(JSON.stringify(response)) }
};
FIREhose.putRecord(params, (err, data) => {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data);
});
};
Here are my events
{
"Name": "Mike",
"Value": "66"
}
When i run this lambda all i am getting response as null . Since i am not passing any callback lambda will default run the implicit callback and returns null. I see that no data is pushed to S3 bucket.
But when i add callback(null,"success") line at the end like this
FIREhose.putRecord(params, (err, data) => {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data);
});
callback(null,"success")
};
I see the data is pushed to S3. Why is that ?
Does async functions always need a callback with some text appended to it ?
Any help is appreciated ?
Thanks
The problem here is that you're mixing your node.js lambda patterns.
Either you use an asynchronous function and return or throw:
exports.handler = async (event,context,callback) => {
// code goes here.
await FIREhose.putRecord(params).promise();
return null; // or whatever result.
};
Or you use the callback approach:
exports.handler = (event,context,callback) => {
// code goes here.
FIREhose.putRecord(params)
.promise();
.then((data) => {
// do stuff with data.
// n.b. you could have used the cb instead of a promise here too.
callback(null, null); // or whatever result.
});
};
(There's a third way using context. but that's a very legacy way).
This is all due to how lambda works and detects when there's been a response.
In your first example (no callback), lambda is expecting your handler to return a promise that it has to wait to resolve/reject, which, in turn, will be the response. However, you're not returning a promise (undefined) and so there's nothing to wait for and it immediately returns- quite probably before the putRecord call has completed.
When you used callback though, you explicitly told lambda that you're using the "old" way. And the interesting thing about the callback approach is that it waits for node's event loop to complete (by default). Which means that .putRecord will probably complete.

Reading local filestream with fallback to other source

I store files in Amazon S3, but also maintain a local file cache. When I need a file I want to check the cache first. I want to avoid testing for local file existance before reading, both because fs.exists will be deprecated and the file can actually be deleted between the exists-check and the file read.
I want to use promises and streams. The below example has a fallback to another local file. My real code will have S3 as fallback.
Would the below be a good solution?
The only way I found to get information of a failed read was to hook up an error handler to the stream. Once I get the "readable" event I unhook my temporary error handler.
I also wonder If I really need to unhook the handler when I use "once" to hook it up.
'use strict';
const fs = require('fs');
function tryToReadLocalFile() {
return new Promise(function(resolve, reject) {
let rs = fs.createReadStream('./test1.txt');
let errorListener = function(err) {
reject(err);
};
rs.once('error', errorListener)
rs.on('readable', () => {
rs.removeListener('error', errorListener);
resolve(rs)
});
});
}
function tryToReadAnotherFile() {
return new Promise(function(resolve, reject) {
let rs = fs.createReadStream('./test2.txt');
let errorListener = function(err) {
reject(err);
};
rs.once('error', errorListener)
rs.on('readable', () => {
rs.removeListener('error', errorListener);
resolve(rs)
});
});
}
tryToReadLocalFile()
.catch(function(err) {
if(err.code === 'ENOENT') {
console.log('test1.txt not found. Fallback to test2.txt')
//Reading from another file as a test. Should read from S3 as fallback
return tryToReadAnotherFile();
} else {
return Promise.reject(err);
}
}).then(function(file) {
console.log('writing to test.txt');
let ws = fs.createWriteStream('./test.txt');
file.pipe(ws);
});
---------- edit -->
I have now implemented a more compact version of the above. I would still be grateful for any input on this, though. Is this a good way to solve this?
As you can see I don't bother to check for ENOENT anymore. Whatever the error is, I want to fall back to S3.
function getFileFromStorageP(options) {
return new Promise(function(resolve, reject) {
let rs = fs.createReadStream(path.join(env.fsp.cacheDir, options.fileId));
rs.once('error', (err) => {reject(err)});
rs.once('readable', () => {resolve(rs)});
}).catch(function(err) {
return srvS3.download({
fileId: options.fileId
});
});
}
I'll post an answer for others to read..
The above solution is now implemented and works great. I haven't found any easier way to get hold of the error, though.

Resources