I'm trying to do a simple command line database transformation with node.js and sequelize. I've simplified my errant code down to the following, but it never returns:
// Set up database connection and models
var models = require('../models_sequelize');
models.User.findOne()
.then(a => {
console.log(a.name);
});
I get a name printed, but then the script hangs. What is wrong? How do I debug this to see what's stuck? I get the impression that there's an orphan promise that's not being fulfilled, but I don't understand where or why. I must be missing something obvious.
If I run the same interactively from the node console, it returns fine.
Sirko's comment re: close() gave me something to go on. I can stop the hanging with the following code:
var models = require('../models_sequelize');
models.User.findOne()
.then(a => {
console.log(a.name);
models.sequelize.close();
})
Alternatively, this seems to work too as I guess it's doing exactly the same thing:
var models = require('../models_sequelize');
models.User.findOne()
.then(a => {
console.log(a.name);
})
.finally(() => {
models.sequelize.close();
});
I also found something about connection pooling timeouts, but I don't think that affects my simple use case. I imagine it'll come into play in more complicated examples.
Would still like to find a good reference as to why this is necessary rather than just my guess.
Related
The lambda's job is to see if a query returns any results and alert subscribers via an SNS topic. If no rows are return, all good, no action needed. This has to be done every 10 minutes.
For some reasons, I was told that we can't have any triggers added on the database, and no on prem environment is suitable to host a cron job
Here comes lambda.
This is what I have in the handler, inside a loop for each database.
sequelize.authenticate()
.then(() => {
for (let j = 0; j < database[i].rawQueries[j].length; j++) {
sequelize.query(database[i].rawQueries[j] => {
if (results[0].length > 0) {
let message = "Temporary message for testing purposes" // + query results
publishSns("Auto Query Alert", message)
}
}).catch(err => {
publishSns("Auto Query SQL Error", `The following query could not be executed: ${database[i].rawQueries[j])}\n${err}`)
})
}
})
.catch(err => {
publishSns("Auto Query DB Connection Error", `The following database could not be accessed: ${databases[i].database}\n${err}`)
})
.then(() => sequelize.close())
// sns publisher
function publishSns(subject, message) {
const params = {
Message: message,
Subject: subject,
TopicArn: process.env.SNStopic
}
SNS.publish(params).promise()
}
I have 3 separate database configurations, and for those few SELECT queries, I thought I could just loop through the connection instances inside a single lambda.
The process is asynchronous and it takes 9 to 12 seconds per invocation, which I assume is far far from optimal
The whole thing feels very very sub optimal but that's my current level :)
To make things worse, I now read that lambda and sequelize don't really play well together:
I am using sequelize because that's the only way I could get 3 connections to the database in the same invocation to work without issues. I tried mssql and tedious packages and wasn't able with either of them
It now feels like using an ORM is an overkill for this very simple task of a SELECT query, and I would really like to at least have the connections and their queries done asynchronously to save some execution time
I am looking into different ways to accomplish this and i went down the rabbit hole and I now have more questions than before! Generators? are they still useful? Observables with RxJs? Could this apply here? Async/Await or just Promises? Do I even need sequelize?
Any guidance/opinion/criticism would be very appreciated
I'm not familiar with sequelize.js but hope I can help. I don't know your level with RxJS and Observables but it's worth to try.
I think you could definitely use Observables and RxJS.
I would start with an interval() that will run the code every time you define.
You can then pipe the interval since it's an Observable, do the auth bit and do a map() to get an array of Observables (for each .query call, I am assuming all your calls, authenticate and query, are Promises so it's possible to transform them into Observables with from()). You can then use something like forkJoin() with the previous array to get a response after all calls are done.
In the .subscribe at the end, you would make the publishSns().
You can pipe a catchError() too and process errors.
The map() part might be not necessary and do it previously and have it stored in a variable since you don't depend on an authenticate value.
I'm certain my solution isn't the only one or the best but i think it would work.
Hope it helps and let me know if it works!
I can use KNEX=debug:query npm start so I get logs and time how long queries took.
I would like to know where these queries are defined (function name or file:line). Due to knexjs beeing ORM I can't find it in code easily.
I've usually have done it like this:
orm.on("query", (q) => try { throw new Error() }catch(e){ console.log(query, e.stack) }
First question - why it's not built-in for debugging? Or maybe it is? Second - how can I achieve this?
I've been coding just as a side project for a bit, piecing together bits that other people have written (it's for a simple discord bot). I want to split my code to make it easier to problem solve and read, however whenever I try to use the code it comes up with an error saying 'SyntaxError: await is only valid in async function'.
I've tried supposedly loading the code asynchronously, loading it with require() and then making a single command asynchronous, making the entire code in the file asynchronous (it's not just one command I want to load, but a whole file. Also I'm not sure if I tried it correctly or not), using the npm async-require, and maybe some others that have been around on the internet.
//one of the solutions I've tried. This is just copy pasted from the
//answer
//file2.js
var fs = require('fs');
module.exports = function (callback) {
fs.readFile('/etc/passwd', function (err, data) {
callback(err, data);
});
};
//file1.js
require('./passwords')(function (err, passwords) {
// This code runs once the passwords have been loaded.
});
In the first file before I split it, I started it with client.on('message', async message => { and it made me able to use the await function in every command. I want to still be able to do that, but just have it a bit neater and easier to use by splitting it.
I'm trying to get this done so I can move on to a different question I asked and give one of the answers a tick. Any help would be greatly appreciated <3
Fix those awaits so that they are not inside async functions. This is a lexical issue that can be solved just by looking at the location were the error occurs. Just look for the nearest containing function to where the await is and mark it async. Repeat until the error goes away.
I've recently started studying about TypeScript, and came across the Starter Project from the docs. Everything looks pretty straight forward, but there is one little part of the code that I just couldn't understand.
Here it is:
// Connect to MongoDB
const mongoUrl = MONGODB_URI;
(<any>mongoose).Promise = bluebird;
mongoose.connect(mongoUrl).then(
() => {},
).catch(err => {
console.log("MongoDB connection error. Please make sure MongoDB is running. " + err);
});
And the confusing part to me is this line:
(<any>mongoose).Promise = bluebird;
The <any>mongoose syntax remembers me of a type casting, but I don't think that is what is really happening here, since we have the bluebird module assigned to the .Promise part.
So if anyone is able to clear this out, I'd appreciate it pretty much.
<any>mongoose is indeed type casting, albeit the old syntax. The more current form would be (mongoose as any).Promise = bluebird. It means that you take the reference to mongoose and ignore its actual type, and treat it as any.
This prevents compile-time errors from type mismatches with mongoose.
In general, you should not cast to any, and you should look into why that is required of you in this case and see if you can avoid it.
I know this question has been asked a few times, but none seem to answer the particular part of using the query results for later.
I also know the problem resides on the fact that queries are asynchronous, and perhaps this is the reason I cannot seem to find a satisfactory answer.
Here's what I'm trying to do:
I have a node project with several sections, each section with different content. These sections have individual properties, which I decided to store in a Model for later use.
So far (and for simplicity sake) I have the following schema:
const SectionSchema = new Schema({
name: String,
description: String
})
const Section = mongoose.model('Sections',SectionSchema)
I'd like to retrieve this data to be used in one of my layouts (a navigation header), so I tried something like this:
const express = require('express')
const app = express()
Section.find().then(function(docs){
app.locals.sections = docs
})
console.log(app.locals.sections) // undefined
This obviously doesn't quite work due to find() being asynchronous, or rather, it does work but the values are populated at a different time. I know that if I do the console.log check inside the function I'd get a result, but that's not the concern, I want to store the data in app.locals so that I could later use it in one of my layouts.
Ideally I'd like to load this data once, before the server begins to listen to requests.
Feel free to correct me if I've made any wrong assumptions, I'm very new to node, so I don't quite know how to approach things quite yet.
Thanks in advance.
EDIT: I should've mentioned I'm using express.
Your node app will likely be comprised of route handlers for http requests. app.locals.section will be undefined if you call it outside of the callback, but it will exist in the route handler.
Let's say you were using something like express or restify:
const app = restify.createServer()
app.get('/', (req, res) => {
return res.json(app.locals.sections)
})
Section.find().then(function(docs){
app.locals.sections = docs
})
console.log(app.locals.section) // is undefined
app.listen(8080-, ()=>{
console.log('Server started 🌎 ',8080)
})
Actually, it might be undefined if the database call took a long time and or a user hit the app super soon after startup. Starting the server in the callback would ensure app.locals.section existed under every scenario:
Section.find().then(function(docs){
app.locals.sections = docs
app.listen(8080-, ()=>{
console.log('Server started 🌎 ',8080)
})
})
You can use async/await within a function to make it seem like you aren't using promises. But you can't use it at the top level of your module. See here: How can I use async/await at the top level?
It really would be fairly idiomatic to do all your app startup in a promise chain. It's a style of coding you are going to see a lot of.
Section.find().then((docs)=>{app.locals.sections = docs})
.then (()=>{/*dosomething with app.locals.sections */})
.then(startServer)
function startServer() {app.listen(8080-, ()=>{
console.log('Server started 🌎 ',8080)
})}