I have calculated Request Units (RU) needed for 50 update operations per seconds on:
https://cosmos.azure.com/capacitycalculator/
but those 50 update operations need 1 second to complete
Thus,
I need to sleep 1 second while inserting into a mongodb database with bulk operation (group of update operations)
is this posible in azure functions with nodejs?
I have tried this code
sleep(milliseconds) {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
but doesnt work.
any similar situation?
Usually we use delay async function for waiting a block to rerun after 1000 milliseconds.
Below is one of the sample we make us of it on our regular basis to add delay to our logic. As you've not provided your entire code, make sure you are following the below structure.
function RunAfterOneSec() {
return new Promise(resolve => {
setTimeout(() => {
resolve('resolved');
}, 1000);
});
}
async function asyncCall() {
console.log('calling');
const result = await RunAfterOneSec();
console.log(result);
// expected output: "resolved"
}
asyncCall();
Or you can use with help of node sleep package and we can use it as below:
var sleep = require('sleep');
sleep.sleep(n)
sleep.sleep(n): sleep for n seconds
sleep.msleep(n): sleep for n miliseconds
sleep.usleep(n): sleep for n microseconds (1 second is 1000000 microseconds)
Related
I was writing an automatic test framework with Jest. For this, I need to run some background tasks under certain intervals. In summary, this is like polling. Let's give an example with a pseudo code.
test('Some potato poteto test', () => {
jest.setTimeout(12000000); // 20 min
const intervalPeriod = 5 * 60 * 1000; // 5 min
let retry = 0;
const intervalId = setInterval(() => {
console.log("I am doing my 5 minutes check");
// my custom logic goes here
retry++;
if(retry === MAX_RETRY) { // this will be always hit
clearInterval(intervalId)
}
}, intervalPeriod);
});
So, in every 5 mins, I'll make a network call, do some of my custom logic. Now, the problem is while running this, the test finished but jest can't terminate.
By 'test-finished' I meant, test suite ran, but the code inside the setInterval does not get executed right then.
My question is, does this happen because setInterval does not run immediately in the event-loop and is jest not configured to run a setInterval?
Please note that I don't want to mock the setInterval and also I don't want to use any fake timer.
Therefore, is it possible to achieve this with Jest?
I also tried the same thing with the cron library for node but it is the same issue. Jest does not execute the code and finish executing then halted because those setInterval/cron are still running in the background.
Is there a way to run those tasks inside Jest? I mean is there a way to run those jobs and when they are done then terminate Jest?
That's intended, Jest execute all the test instructions and quit the test.
For jest to wait for a computation, you need to use Promises and its async mode.
Doc for Jest async mode: https://jestjs.io/docs/asynchronous
Learn more about Promises here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
And async/await syntax here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
This is a naive way for doing it:
const MAX_RETRY = 3;
const INTERVAL_PERIOD = 5 * 60 * 1000; // 5 min
const SAFETY_TIME = INTERVAL_PERIOD;
const TEST_TIMEOUT = (1 + MAX_RETRY) * INTERVAL_PERIOD + SAFETY_TIME;
test('Some potato poteto test', async () => {
let retry = 0;
let success = false;
function yourLogicFunction(): boolean {
const resultOfYourLogicFunction = retry === MAX_RETRY; // because retry is used in this condition I store the result before changing it
console.log(`this is my attempt number ${retry}`);
retry++;
return resultOfYourLogicFunction;
}
function mockRequest(resolve: (value: unknown) => void): void {
setTimeout(() => resolve(yourLogicFunction()), INTERVAL_PERIOD);
}
while (!success && retry <= MAX_RETRY) {
success = await new Promise(mockRequest) as boolean;
}
}, TEST_TIMEOUT);
The third parameter of test is its timeout.
I need to report the status of a long running operation in node.js. The basic use case is outlined in the code below. awaiting the longProcess method I know will act synchronously to the caller, but I must await the method in my code. Should I handle this within the longProcess method? Not sure how to address this issue.
function sleep (ms: number) {
new Promise(resolve => setTimeout(resolve, ms));
}
let processedCount = 0;
async function longProcess() {
// really long operation
while (true) {
processedCount++;
await sleep(1000); // simulate long process
if (processedCount === 10) // just to end the test somehow
break;
}
}
async function report() {
console.log(processedCount);
}
async function main() {
const id = setInterval(report, 500);
await longProcess();
clearInterval(id);
}
main().then(() => console.log("Done"));
The sleep method is just for demonstration purposes to simulate a long running operation. 'longProcess' performs complex and time intensive processing. It calls a callback passed in to report back a processed count the caller. The class that contains the calling method (and the callback), also has a report method that I would like to call at regular intervals. And I need to be able to create a unit test for this
Your sleep function is not returning the promise you are creating. You are calling await on the value returned from the function, which in this case is undefined so it doesn't actually wait at all.
function sleep (ms: number) {
return new Promise(resolve => setTimeout(resolve, ms));
}
My issues
Launch 1000+ online API that limits the number of API calls to 10 calls/sec.
Wait for all the API calls to give back a result (or retry), it can take 5 sec before the API sends it data
Use the combined data in the rest of my app
What I have tried while looking at a lot of different questions and answers here on the site
Use promise to wait for one API request
const https = require("https");
function myRequest(param) {
const options = {
host: "api.xxx.io",
port: 443,
path: "/custom/path/"+param,
method: "GET"
}
return new Promise(function(resolve, reject) {
https.request(options, function(result) {
let str = "";
result.on('data', function(chunk) {str += chunk;});
result.on('end', function() {resolve(JSON.parse(str));});
result.on('error', function(err) {console.log("Error: ", err);});
}).end();
});
};
Use Promise.all to do all the requests and wait for them to finish
const params = [{item: "param0"}, ... , {item: "param1000+"}]; // imagine 1000+ items
const promises = [];
base.map(function(params){
promises.push(myRequest(params.item));
});
result = Promise.all(promises).then(function(data) {
// doing some funky stuff with dat
});
So far so good, sort of
It works when I limit the number of API requests to a maximum of 10 because then the rate limiter kicks in. When I console.log(promises), it gives back an array of 'request'.
I have tried to add setTimeout in different places, like:
...
base.map(function(params){
promises.push(setTimeout(function() {
myRequest(params.item);
}, 100));
});
...
But that does not seem to work. When I console.log(promises), it gives back an array of 'function'
My questions
Now I am stuck ... any ideas?
How do I build in retries when the API gives an error
Thank you for reading up to hear, you are already a hero in my book!
When you have a complicated control-flow using async/await helps a lot to clarify the logic of the flow.
Let's start with the following simple algorithm to limit everything to 10 requests per second:
make 10 requests
wait 1 second
repeat until no more requests
For this the following simple implementation will work:
async function rateLimitedRequests (params) {
let results = [];
while (params.length > 0) {
let batch = [];
for (i=0; i<10; i++) {
let thisParam = params.pop();
if (thisParam) { // use shift instead
batch.push(myRequest(thisParam.item)); // of pop if you want
} // to process in the
// original order.
}
results = results.concat(await Promise.all(batch));
await delayOneSecond();
}
return results;
}
Now we just need to implement the one second delay. We can simply promisify setTimeout for this:
function delayOneSecond() {
return new Promise(ok => setTimeout(ok, 1000));
}
This will definitely give you a rate limiter of just 10 requests each second. In fact it performs somewhat slower than that because each batch will execute in request time + one second. This is perfectly fine and already meet your original intent but we can improve this to squeeze a few more requests to get as close as possible to exactly 10 requests per second.
We can try the following algorithm:
remember the start time
make 10 requests
compare end time with start time
delay one second minus request time
repeat until no more requests
Again, we can use almost exactly the same logic as the simple code above but just tweak it to do time calculations:
const ONE_SECOND = 1000;
async function rateLimitedRequests (params) {
let results = [];
while (params.length > 0) {
let batch = [];
let startTime = Date.now();
for (i=0; i<10; i++) {
let thisParam = params.pop();
if (thisParam) {
batch.push(myRequest(thisParam.item));
}
}
results = results.concat(await Promise.all(batch));
let endTime = Date.now();
let requestTime = endTime - startTime;
let delayTime = ONE_SECOND - requestTime;
if (delayTime > 0) {
await delay(delayTime);
}
}
return results;
}
Now instead of hardcoding the one second delay function we can write one that accept a delay period:
function delay(milliseconds) {
return new Promise(ok => setTimeout(ok, milliseconds));
}
We have here a simple, easy to understand function that will rate limit as close as possible to 10 requests per second. It is rather bursty in that it makes 10 parallel requests at the beginning of each one second period but it works. We can of course keep implementing more complicated algorithms to smooth out the request pattern etc. but I leave that to your creativity and as homework for the reader.
I have divided a day on 6 parts and each part of a day has its own interval. My function must run in this intervals.
for example 12:00pm - 15:00pm interval is 10 min, so function must be called every 10 min, but 15:00pm - 20:00pm interval is 2 min so from 15:01pm interval must be changed from 10min to 2min.
Need a suggestion how to build this kind of timer.
I can get intervals from mongoDB or local json file.
I guess I have to check what time is it and get an interval (from mongoDB or json file) for that time, then pass it to setInterval() or Cron job scheduler.
Tryed this way but every time im passing new interval last intervals are still working: If interval is 2 min and im changing it to 5 min, function is called twice: every 2 min and every 5 min in both setInterval() and Cron
const test = (min) => {
console.log(min);
var intervale = setInterval(() => {
console.log('firing', min, new Date());
}, min);
}
const test = (min) => {
cron.schedule(`0 */${min} * * * *`, () => {
console.log('firing cron', new Date())
});
}
thank you
Can you do something like this? This uses the EventEmitter model and fires an event based on the custom Interval.
const eventemitter_cls = require("events").EventEmitter;
let eventemitter = new eventemitter_cls();
intervalName = "randomintervals"
let sumOfTwoNumbers = () => {
console.log(`hello world ${(new Date()).getMinutes()} ${(new Date()).getSeconds()}` );
// After you are done with your function, again you need to emit the event based on the time interval
setTimeout(() => {
eventemitter.emit(intervalName)
}, getTimeInterval());
}
let getTimeInterval = () => {
let currentTime = new Date();
// put your complicated interval schedule here
if(currentTime.getHours() >= 10 && currentTime.getHours() < 11){
return 5000;
}
else if (currentTime.getHours() > 12 && currentTime.getHours() < 15) {
return 2000;
}
else{
return 1000;
}
}
// give your function name here. the function will be called whenever someone emits intervalName
eventemitter.on(intervalName, sumOfTwoNumbers);
eventemitter.emit(intervalName);
I think It's still firing because there is no clearInterval telling it to stop.
As per Understanding the node.js event loop, node.js supports a single thread model. That means if I make multiple requests to a node.js server, it won't spawn a new thread for each request but will execute each request one by one. It means if I do the following for the first request in my node.js code, and meanwhile a new request comes in on node, the second request has to wait until the first request completes, including 5 second sleep time. Right?
var sleep = require('sleep');
sleep.sleep(5)//sleep for 5 seconds
Is there a way that node.js can spawn a new thread for each request so that the second request does not have to wait for the first request to complete, or can I call sleep on specific thread only?
If you are referring to the npm module sleep, it notes in the readme that sleep will block execution. So you are right - it isn't what you want. Instead you want to use setTimeout which is non-blocking. Here is an example:
setTimeout(function() {
console.log('hello world!');
}, 5000);
For anyone looking to do this using es7 async/await, this example should help:
const snooze = ms => new Promise(resolve => setTimeout(resolve, ms));
const example = async () => {
console.log('About to snooze without halting the event loop...');
await snooze(1000);
console.log('done!');
};
example();
In case you have a loop with an async request in each one and you want a certain time between each request you can use this code:
var startTimeout = function(timeout, i){
setTimeout(function() {
myAsyncFunc(i).then(function(data){
console.log(data);
})
}, timeout);
}
var myFunc = function(){
timeout = 0;
i = 0;
while(i < 10){
// By calling a function, the i-value is going to be 1.. 10 and not always 10
startTimeout(timeout, i);
// Increase timeout by 1 sec after each call
timeout += 1000;
i++;
}
}
This examples waits 1 second after each request before sending the next one.
Please consider the deasync module, personally I don't like the Promise way to make all functions async, and keyword async/await anythere. And I think the official node.js should consider to expose the event loop API, this will solve the callback hell simply. Node.js is a framework not a language.
var node = require("deasync");
node.loop = node.runLoopOnce;
var done = 0;
// async call here
db.query("select * from ticket", (error, results, fields)=>{
done = 1;
});
while (!done)
node.loop();
// Now, here you go
When working with async functions or observables provided by 3rd party libraries, for example Cloud firestore, I've found functions the waitFor method shown below (TypeScript, but you get the idea...) to be helpful when you need to wait on some process to complete, but you don't want to have to embed callbacks within callbacks within callbacks nor risk an infinite loop.
This method is sort of similar to a while (!condition) sleep loop, but
yields asynchronously and performs a test on the completion condition at regular intervals till true or timeout.
export const sleep = (ms: number) => {
return new Promise(resolve => setTimeout(resolve, ms))
}
/**
* Wait until the condition tested in a function returns true, or until
* a timeout is exceeded.
* #param interval The frenequency with which the boolean function contained in condition is called.
* #param timeout The maximum time to allow for booleanFunction to return true
* #param booleanFunction: A completion function to evaluate after each interval. waitFor will return true as soon as the completion function returns true.
*/
export const waitFor = async function (interval: number, timeout: number,
booleanFunction: Function): Promise<boolean> {
let elapsed = 1;
if (booleanFunction()) return true;
while (elapsed < timeout) {
elapsed += interval;
await sleep(interval);
if (booleanFunction()) {
return true;
}
}
return false;
}
The say you have a long running process on your backend you want to complete before some other task is undertaken. For example if you have a function that totals a list of accounts, but you want to refresh the accounts from the backend before you calculate, you can do something like this:
async recalcAccountTotals() : number {
this.accountService.refresh(); //start the async process.
if (this.accounts.dirty) {
let updateResult = await waitFor(100,2000,()=> {return !(this.accounts.dirty)})
}
if(!updateResult) {
console.error("Account refresh timed out, recalc aborted");
return NaN;
}
return ... //calculate the account total.
}
It is quite an old question, and though the accepted answer is still entirely correct, the timers/promises API added in v15 provides a simpler way.
import { setTimeout } from 'timers/promises';
// non blocking wait for 5 secs
await setTimeout(5 * 1000);