Probably a newbie question:
(I know that there is gettimeremaining property in context).
I have a job that is querying an API (say Marketo), however I need to plan for the scenario where the API does not return the required data in 15 mins.
I can check the remaining time in ms from context, however how do i accomplish this along with my data pull tasks as 2 concurrent tasks?
Meaning, do I fire off something first that says keep check of time, and then i proceed to my actual work and when time is 14 mins the first thing/thread returns back and I can proceed accodingly?
Thanks
Since you are calling an external API you will be using a client to make a connection; set connection timeout to be less than maximum time you can wait. In-case API doesn't return response in time your connection with throw timeout exception which than you can handle gracefully.
https://docs.python-requests.org/en/latest/user/advanced/#timeouts
Related
Can the execution of an expressJS method be delayed for 30 days or more just by using setTimeout ?
Let's say I want to create an endpoint /sendMessage that send a message to my other app after a timeout of 30 days. Will my expressJS method execution will last long time enough to fire this message after this delay ?
If your server runs continuously for 30 days or more, then setTimeout() will work for that. But, it is probably not smart to rely on that fact that your server never, ever has to restart.
There are 3rd party programs/modules designed explicitly for this. If you don't want to use one of them, then what I have done in the past is I write each future firing time into a JSON file and I set a timer for it with setTimeout(). If the timer successfully fires, then I remove that time from the JSON file.
So, at any point in time, the JSON file always contains a list of times in the future that I want timers to fire for. Any timer that fires is immediately removed from the JSON file.
Anytime my server starts up, I read the times from the JSON file and reconfigure the setTimeout() for each one.
This way, even if my server restarts, I won't lose any of the timers.
In case you were wondering, the way nodejs creates timers, it does not cost you anything to have a bunch of future timers configured. Nodejs keeps the timers in a sorted linked list and the event loop just checks the time for the next timer to fire - the one at the front of the sorted list (the rest of the timers are not looked at until they get to the front of the sorted list). This means the only time it costs anything to have lots of future timers is when inserting a new timer into the sorted list and there is no regular cost in the event loop to having lots of pending timers present.
The title isn't accurate because based on what I have found in my research there doesn't seem to be a way to make a function atomic in nodejs, but I will lay out my problem to see if you people can come up with something that I have not been able to think about.
I am trying to setup a scheduler where I can set my appointment time slots say 1 hr long each and when someone makes an appointment I want to make sure that the time slot is not taken before scheduling it.
So for example I decide that I will be working from 9 am to 2 pm with a time slot of one hour. Then my schedule would be 9-10, 10-11, 11-12, 12-1, 1-2.
An appointment will come in with a start time of 11 and end time of 12. I need to make sure that slot isn't already taken.
I am using mongodb with nodejs and restify.
I understand that in my appointments collection I can set an index on a combination of values like start time and end time, as discussed here Creating Multifield Indexes in Mongoose / MongoDB.
But if I decide to change my time slot from 1 hour to say 1.5 hours then I will have scheduling conflicts as the start time and end time of entries in the database will not match up with the new interval
Currently I have a function which checks to make sure that the new appointment will not conflict but I am not sure if it will work out well when I have multiple requests coming in. This is a nodejs and restify app so basically an api with a mongodb that it talks to, to handle appointments.
I am running it with multiple workers, so I am worried that at a certain point two requests will come in at the same time, handled by two different workers for the same time slot. When my conflict checking function executes it will return saying that the slot is open for both of them since no appointment has been made yet and then there will be a scheduling conflict.
Any ideas on how to combat this, or is there something in the way javascript executes so that I shouldn't have to worry about it this? All input will be appreciated
Thanks!
I ended up using https://github.com/Automattic/kue, to queue my requests and added another endpoint where you can check the status of your request. So when you want to make an appointment your request ends up in the job queue, and you can then periodically check the status of your request. This way only one appointment request gets processed at a time so no concurrency issues.
I'm currently developing a gambling website built on top of Ethereum blockchain. Since recording all the bets made by a gambler is very complex (because they can make a bet without even visiting the website, by interacting directly with the blockchain) I came to conclusion that I need a function on my server that will run every 0.5 - 1 minute and download all the new bets that came up from the blockchain and shadow them in my database (yes I need to have them in my database as well).
I am not experienced too much with all this backend stuff, I've read somewhere that I could use setInterval(30 seconds) function on the server and run it on the server start. But is this a real option? Do people even do things like this? Won't an infinite function running every 30 second just clog up the whole server?
I've done similar a number of times with no issue. Events will just be queued and run as appropriate. However, one thing to be wary of:
const timeout = 1000;
setInterval(() => {
// some process that takes longer than timeout
}, timeout);
When you write something like this, setInterval will run again after the timeout period even if the first operation hasn't completed yet. THIS can easily lead to you blocking your thread. Instead, I prefer to use setTimeout to achieve this:
const timeout = 1000;
const withTimeout = () => {
// some process that takes longer than timeout
setTimeout(withTimeout, 1000);
}
This way, your second invocation of withTimeout is only queued up AFTER the first execution is run. With this mechanism, you don't get your operations strictly every timeout period, but rather timeout AFTER the last one.
I've come up with a fancy issue of synchronization in node.js, which I've not able to find an elegant solution:
I setup a express/node.js web app for retrieving statistics data from a one row database table.
If the table is empty, populate it by a long calculation task
If the record in table is older than 15 minutes from now, update it by a long calculation task
Otherwise, respond with a web page showing the record in DB.
The problem is,
when multiple users issue requests simultaneously, in case the record is old, the long calculation task would be executed once per request, instead of just once.
Is there any elegant way that only one request triggers the calculation task, and all others wait for the updated DB record?
Yes, it is called locks.
Put an additional column in your table say lock which will be of timestamp type. Once a process starts working with that record put a now+timeout time into it (by the rule of thumb I choose timeout to be 2x the average time of processing). When the process stops processing update that column with NULL value.
At the begining of processing check that column. If the value > now condition is satisfied then return some status code to client (don't force client to wait, it's a bad user experience, he doesn't know what's going on unless processing time is really short) like 409 Conflict. Otherwise start processing (also ideally processing takes place in a separate thread/process so that user won't have to wait: respond with an appropriate status code like 202 Accepted).
This now+timeout value is needed in case your processing process crashes (so we avoid deadlocks). Also remember that you have to "check and set" this lock column in transaction because of race conditions (might be quite difficult if you are working with MongoDB-like databases).
My code runs oracle select query on a server. The server may be free or it may be busy at certain times.Now what I am making is a timeout mechanism in Pro*C that times out after 10 seconds and cancels the thread that was running oracle query (if it is still running).
Now, the problem which I'm facing is that the timeout function is scheduled to terminate the threads after 10 seconds, and it does so very well. However it is not able to make out whether the oracle query was still in waiting state at the time of termination or it was returning results/executing procedure/blah-blah...
What I want is a function/mechanism/anything that would be able to query the status of the oracle query initiated and act as following: after 10 seconds,
IF (query is still in waiting state)
DO terminate all threads;
ELSE IF (query is fetching data/doing some processing)
DO wait for the fetching/processing to complete and then terminate all threads;
A Pro*C or Oracle function call would be the best option instead of using complex code, if possible.
What does "fetching data/ doing some processing" mean to you? In general, since Oracle executes the query as the application fetches data, a query will alternate between actively running (which would involve being in a wait state) and returning data to the client. If you are sending 50 rows at a time to the client, for example, Oracle only executes the query far enough to identify the first 50 rows, it sends those rows to the client, and it waits for the client to request the next 50 rows before continuing to execute the query to pull the next 50 rows.