I am using agenda.js in my Node project, backed with a MongoDB database, to handle batch processes we need to run. This is working well. I do have a question about timezones, however. When I use the every() operation, it seems like it accepts the job name, and the schedule. So I have been seeding jobs to the database like so:
for (let job of dbJobs) {
await agenda.every(schedule, job.name);
}
Note that for the above, schedule is in cron format -- 00 05 * * 1-5.
This works. However, from what I can tell, every() doesn't accept an argument for repeatTimezone. So what does it do to calculate the timezone in those cases?
To clarify, when I look at the document in the database after the job has been added using every(), the repeatTimezone property exists, but its value is set to null.
Other agenda operations, like repeatEvery(), do accept an argument for timezone, like so:
job.repeatEvery('0 6 * * *', {
timezone: 'America/New_York'
});
Since I'm using every(), I have been managing this by first seeding the database using every(), and then running a Mongo updateMany() to add the timzeone explicitly to all jobs:
async function addTimezoneToJobs() {
try {
const db = await client.db(dbName);
await db.collection('batch_processes').updateMany({}, {
$set: {
repeatTimezone: 'America/New_York'
}
});
} catch (error) {
console.log(error);
}
}
But strangely enough, agenda seems to calculate the same time even when I don't explicitly add the repeatTimezone property value to the jobs as when I do.
What's happening here that I'm not understanding? How is the runtime calculated with every(), and is there a way to pass in timezone?
FYI: I am not in the same timezone as that which needs to be set in the db.
Your Question seems to be 2 part, I'm not exactly sure I'll be able to explain it very well but let me try
So, your first question
However, from what I can tell, every() doesn't accept an argument for Timezone
Well Technically you can add Timezone option to every() as well because what this method does is it calls job.repeatEvery internally and as you already know you can add timezone to that. To Support my answer, I found 2 evidence
From Documentation as every accepts 4 parameters
every(interval, name, [data], [options])
options is an optional argument that will be passed to job.repeatEvery. In order to use this argument, data must also be specified.
So you can technically pass timezone if you pass data as well
From SourceCode, here you can see they use job.repeatEvery(interval, options) internally.
Now, To your Second Question
what does it do to calculate the timezone in those cases?
Well They have a very unique yet required module named ComputeNextRunAt().
So I went through their Source Code and figured Out that this is to Compute when will be the next run for your job based on startingTime and Interval.
Your Code works because you have once (Initially) mentioned in your job to follow America/New_York timezone, so every next job interval is calculated based on that, that's the reason you don't need to specify it again.
So, If initially you haven't had specified the timezone attribute, you would have gotten your local Timezone but now you did so, it calculates the next interval based on that.
Related
I'm new to firebase and currently I'm still trying to learn how to get the latest children node based in this RTDB. My nodeMCU will send new data periodically so I'm trying to get the latest node when its added and the value of that node. Can you provide with a sample code for me to understand better? And if possible please explain like I'm 5. Thank you and have a good day.
From what I understand you have an Arduino module that is going to be constantly introducing data into your database.
What you want is to be able to read the value shown in the image as MQ7 every time a new value is added.
If this is the case there are different ways to obtain it.
The first and most common one would be to use the firebase Child Added event. With this event you can handle the data entered every time there is an addition to the reference to the database.
Using this event you would have a set of all the values entered in your reference and with each addition automatically (In Real Time) this set would be updated.
Taking your image as an example, the query code would be something like this (JS):
dbRef.child("Sensor MQ7").on("child_added", (snap) => {
for (i in snap.val()) {
const value_MQ7 = snap.child(i).child("MQ7").val()
// Do what you want with the value
console.log(value_MQ7)
}
})
If you don’t want to have that set with all the values entered in your reference, the best option would be a new function that returns only the value you are requesting, that is, a function that returns the MQ7 value of the last object entered in your reference sensor MQ7.
The query code would be something like this (JS):
const query = dbRef.child("Sensor MQ7").orderByKey().limitToLast(1);
query.get().then((snap) => {
for (i in snap.val()) {
// Do what you want with the value
const value_MQ7 = snap.child(i).child("MQ7").val()
console.log(value_MQ7)
}
})
I need to find a way to generate a random number each time the REST call is executed.
I have the following GET call:
exec(http("Random execution")
.get("/randomApi")
.queryParam("id", getRandomId()))
}
Obviously it doesn't work as the random number is only generated once and I end up with the same
number whenever this call is executed. I cant use the feeder option as my feeder is already huge and is generated by a 3rd party for each test.
.queryParam takes Expressions as its arguments, and since Expression is an alias for a session function, you can just do...
.queryParam("id", session => getRandomId())
You could also define a second feeder that uses a function to generate the values - no need to update your existing feeder or add another csv file. This would be useful if you had more complicated logic for getting / generating an Id
val idFeeder = Iterator.continually(Map("id" -> Random.nextInt(999999)))
//in your scenario...
.feed(idFeeder)
.exec(http("Random execution")
.get("/randomApi")
.queryParam("id", "${id}")
)
In the spirit of having options, another option you have is to store an object in the session that support toString, which generates whatever you need. It's a nifty trick that you can use for all kinds of things.
object RANDOM_ID {
toString() { return RandomId().toString() }
}
...
exec( _.set( "RANDOM_ID", RANDOM_ID ) )
...
.exec(
http("Random execution")
.get("/randomApi")
.queryParam( "id", "${RANDOM_ID}" )
)
You can apply the same principle to generating random names, addresses, telephone numbers, you name it.
So, which is the better solution? The feeder, or the object in session?
Most of the time, it'll be the feeder, because you control when it is updated. The object in session will be different every time, whereas the feeder solution, you control when the value updates, and then you can reference it multiple times before you change it.
But there may be instances where the stored object solution results in easier to read code, provided you are good with the value changing every time it is accessed. So it's good to know that it is an option.
In mongodb docs https://docs.mongodb.com/manual/changeStreams/
there is a quote:
The oplog must have enough history to locate the operation associated with the token or the timestamp, if the timestamp is in the past.
So it seems that it is possible to resume and get all the events that where added to oplog from a certain time.
There is a param, seems that it has to accomplish what I need
watch([],{startAtOperationTime: ...})
https://github.com/mongodb/specifications/blob/master/source/change-streams/change-streams.rst#startatoperationtime
Param is Timestamp, I don't get how to translate a particular date to correct timestamp.
startAtOperationTime is a new parameter for changestreams introduced in MongoDB 4.0 and newer driver versions. It allows you to ensure that you're not missing any writes just in case the stream was interrupted, and you don't have access to the resume token.
One caveat of using startAtOperationTime is that your app needs to be prepared to accept that it may see a write event twice when resuming the changestream, since you're resuming from an arbitrary point in time.
In node, this can be done by constructing a Timestamp object and passing it into watch():
async function run() {
const con = await MongoClient.connect(uri, {useNewUrlParser: true})
const ts = new Timestamp(1, 1560812065)
con.db('test').collection('test').watch([], {startAtOperationTime: ts})
.on('change', console.log)
}
The Timestamp object itself is created with the form of:
new Timestamp(ordinal, unix_epoch_in_seconds)
A detailed explanation can be found in BSON Timestamp.
In node, you can get the current epoch (in milliseconds) using e.g.:
(new Date).getTime()
bearing in mind that this needs to be converted to seconds for creating the Timestamp object needed for startAtOperationTime.
I'm using Sequelize.js for ORM and have a few associations (which actually doesn't matter now). My models get get and set methods from those associations. Like this (from docs):
var User = sequelize.define('User', {/* ... */})
var Project = sequelize.define('Project', {/* ... */})
// One-way associations
Project.hasOne(User)
/*
...
Furthermore, Project.prototype will gain the methods getUser and setUser
according to the first parameter passed to define.
*/
So now, I have Project.getUser(), which returns a Promise. But if I call this twice on the very same object, I get SQL query executed twice.
My question is - am I missing something out, or this is an expected behavior? I actually don't want to make additional queries each time I call the same method on this object.
If this is expected - should I use custom getters with member variables which I manually populate and return if present? Or there is something more clever? :)
Update
As from DeBuGGeR's answer - I understand I can use includes when making a query in order to eager load everything, but I simply don't need it, and I can't do it all the time. It's waste of resources and a big overhead if I load my entire DB at the beginning, just to understand (by some criteria) that I won't need it. I want to make additional queries depending on situation. But I also can't afford to destroy all models (DAO objects) that I have and create new ones, with all the info inside them. I should be able to update parts of them, which are missing (from relations).
If you use getUser() it will make the query call, it dosent give you access to the user. You can manually save it to project.user or project.users depending on the association.
But you can try Eager Loading
Project.find({
include: [
{ model: User, as: 'user' } // here you HAVE to specify the same alias as you did in your association
]
}).success(function(project){
project.user // contains the user
});
Also e.g of getUser(). Dont expect it to automatically cache user and dont override this cleverly as it will create side effects. getUser is expected to get from database and it should!
Project.getUser().then(function(user){
// user is available and is a sequelize object
project.user = user; // save project.user and use it till u want to
})
The first part of things is clear - every call to get[Association] (for example Project.getUser()) WILL result in database query.
Sequelize does not maintain any kind of state nor cache for the results. You can get user in the Promisified result of the call, but if you want it again - you will have to make another query.
What #DeBuGGeR said - about using accessors is also not true - accessors are present only immediately after a query, and are not preserved.
As sometimes this is not ok, you have to implement some kind of caching system by yourself. Here comes the tricky part:
IF you want to use the same get method Project.getUser(), you won't be able to do it, as Sequelize overrides your instanceMethods. For example, if you have the association mentioned above, this won't work:
instanceMethods: {
getUser: function() {
// check if you have it, otherwise make a query
}
}
There are few possible ways to fix it - either change Sequelize core a little (to first check if the method exists), or use some kind of wrapper to those functions.
More details about this can be found here: https://github.com/sequelize/sequelize/issues/3707
Thanks to mickhansen for the cooperation on how to understand what to do :)
I want to create a "prepared statement" in postgres using the node-postgres module. I want to create it without binding it to parameters because the binding will take place in a loop.
In the documentation i read :
query(object config, optional function callback) : Query
If _text_ and _name_ are provided within the config, the query will result in the creation of a prepared statement.
I tried
client.query({"name":"mystatement", "text":"select id from mytable where id=$1"});
but when I try passing only the text & name keys in the config object, I get an exception :
(translated) message is binding 0 parameters but the prepared statement expects 1
Is there something I am missing ? How do you create/prepare a statement without binding it to specific value in order to avoid re-preparing the statement in every step of a loop ?
I just found an answer on this issue by the author of node-postgres.
With node-postgres the first time you issue a named query it is
parsed, bound, and executed all at once. Every subsequent query issued
on the same connection with the same name will automatically skip the
"parse" step and only rebind and execute the already planned query.
Currently node-postgres does not support a way to create a named,
prepared query and not execute the query. This feature is supported
within libpq and the client/server protocol (used by the pure
javascript bindings), but I've not directly exposed it in the API. I
thought it would add complexity to the API without any real benefit.
Since named statements are bound to the client in which they are
created, if the client is disconnected and reconnected or a different
client is returned from the client pool, the named statement will no
longer work (it requires a re-parsing).
You can use pg-prepared for that:
var prep = require('pg-prepared')
// First prepare statement without binding parameters
var item = prep('select id from mytable where id=${id}')
// Then execute the query and bind parameters in loop
for (i in [1,2,3]) {
client.query(item({id: i}), function(err, result) {...})
}
Update: Reading your question again, here's what I believe you need to do. You need to pass a "value" array as well.
Just to clarify; where you would normally "prepare" your query, just prepare the object you pass to it, without the value array. Then where you would normally "execute" your query, set the value array in the object and pass it to the query. If it's the first time, the driver will do the actual prepare for you the first time around, and simple do binding and execution for the rest of the iteration.