I am new to node.js but I surprised by searching and finding no way to measure run time profile of a function in node.js via code (not looking for an external tool). I am looking for something like this
// store time
run_function1();
// store time
// calculate time difference
console.log('function1 ran in '+time_diff+' seconds');
There is also a function in Node.js that is called hrtime to get high resolution timings.
Node Doc
const NS_PER_SEC = 1e9;
const time = process.hrtime();
// [ 1800216, 25 ]
setTimeout(() => {
const diff = process.hrtime(time);
// [ 1, 552 ]
console.log(`Benchmark took ${diff[0] * NS_PER_SEC + diff[1]} nanoseconds`);
// benchmark took 1000000552 nanoseconds
}, 1000);
Take a look at performance.now()
As example says:
var t0 = performance.now();
doSomething();
var t1 = performance.now();
console.log("Call to doSomething took " + (t1 - t0) + " milliseconds.");
Try the node.js built in profiler.
Related
I'm working on a node.js web server using express.js that should offer a dashboard to monitor database servers.
The architecture is quite simple:
a gatherer retrieves the information in a predefined interval and stores the data
express.js listens to user requests and shows a dashboard based on the stored data
I'm now wondering how to best implement the gatherer to make sure that it does not block the main loop and the simplest solution seems be to just use a setTimeout based approach but I was wondering what the "proper way" to architecture this would be?
Your concern is your information-gathering step. It probably is not as CPU-intensive as it seems. Because it's a monitoring app, it probably gathers information by contacting other machines, something like this.
async function gather () {
const results = []
let result
result = await getOracleMetrics ('server1')
results.push(result)
result = await getMySQLMetrics ('server2')
results.push(result)
result = await getMySQLMetrics ('server3')
results.push(result)
await storeMetrics(results)
}
This is not a cpu-intensive function. (If you were doing a fast Fourier transform on an image, that would be a cpu-intensive function.)
It spends most of its time awaiting results, and then a little time storing them. Using async / await gives you the illusion it runs synchronously. But, each await yields the main loop to other things.
You might invoke it every minute something like this. The .then().catch() stuff invokes it asynchronously.
setInterval (
function go () {
gather()
.then()
.catch(console.error)
}, 1000 * 60 * 60)
If you do actually have some cpu-intensive computation to do, you have a few choices.
offload it to a worker thread.
break it up into short chunks, with sleeps between them.
sleep = function sleep (howLong) {
return new Promise(function (resolve) {
setTimeout(() => {resolve()}, howLong)
})
}
async function gather () {
for (let chunkNo = 0; chunkNo < 100; chunkNo++) {
doComputationChunk(chunkNo)
await sleep(1)
}
}
That sleep() function yields to the main loop by waiting for a timeout to expire.
None of this is debugged, sorry to say.
For recurring tasks I prefer to use node-scheduler and shedule the jobs on app start-up.
In case you don't want to run CPU-expensive tasks in the main-thread, you can always run the code below in a worker-thread in parallel instead of the main thread - see info here
Here are two examples, one with a recurrence rule and one with interval in minutes using a cron expression:
app.js
let mySheduler = require('./mysheduler.js');
mySheduler.sheduleRecurrence();
// And/Or
mySheduler.sheduleInterval();
mysheduler.js
/* INFO: Require node-schedule for starting jobs of sheduled-tasks */
var schedule = require('node-schedule');
/* INFO: Helper for constructing a cron-expression */
function getCronExpression(minutes) {
if (minutes < 60) {
return `*/${minutes} * * * *`;
}
else {
let hours = (minutes - minutes % 60) / 60;
let minutesRemainder = minutes % 60;
return `*/${minutesRemainder} */${hours} * * *`;
}
}
module.exports = {
sheduleRecurrence: () => {
// Schedule a job # 01:00 AM every day (Mo-Su)
var rule = new schedule.RecurrenceRule();
rule.hour = 01;
rule.minute = 00;
rule.second = 00;
rule.dayOfWeek = new schedule.Range(0,6);
var dailyJob = schedule.scheduleJob(rule, function(){
/* INFO: Put your database-ops or other routines here */
// ...
// ..
// .
});
// INFO: Verbose output to check if job was scheduled:
console.log(`JOB:\n${dailyJob}\n HAS BEEN SCHEDULED..`);
},
sheduleInterval: () => {
let intervalInMinutes = 60;
let cronExpressions = getCronExpression(intervalInMinutes);
// INFO: Define unique job-name in case you want to cancel it
let uniqueJobName = "myIntervalJob"; // should be unique
// INFO: Schedule the job
var job = schedule.scheduleJob(uniqueJobName,cronExpressions, function() {
/* INFO: Put your database-ops or other routines here */
// ...
// ..
// .
})
// INFO: Verbose output to check if job was scheduled:
console.log(`JOB:\n${job}\n HAS BEEN SCHEDULED..`);
}
}
In case you want to cancel a job, you can use its unique job-name:
function cancelCronJob(uniqueJobName) {
/* INFO: Get job-instance for canceling scheduled task/job */
let current_job = schedule.scheduledJobs[uniqueJobName];
if (!current_job || current_job == 'undefinded') {
/* INFO: Cron-job not found (already cancelled or unknown) */
console.log(`CRON JOB WITH UNIQUE NAME: '${uniqueJobName}' UNDEFINED OR ALREADY CANCELLED..`);
}
else {
/* INFO: Cron-job found and cancelled */
console.log(`CANCELLING CRON JOB WITH UNIQUE NAME: '${uniqueJobName}`)
current_job.cancel();
}
};
In my example the recurrence and the interval are hardcoded, obviously you can also pass the recurrence-rules or the interval as argument to the respective function..
As per your comment:
'When looking at the implementation of node-schedule it feels like a this layer on top of setTimeout..'
Actually, node-schedule is using long-timeout -> https://www.npmjs.com/package/long-timeout so you are right, it's basically a convenient layer on top of timeOuts
I am trying to compare the READ performance of a library called Memored to regular old RAM variables in Node.js.
I expected that the data stored with Memored to be at least slightly slower than RAM storage in terms of reading data, but the results show the opposite (read below for my outputs).
I am running this in the terminal of Visual Studio Code on Windows 10. It’s all being done in Typescript, which gets compiled down to JavaScript later and then run with the "node" command.
This is my RAM test:
var normalRAM = {
firstname: 'qwe',
lastname: 'fsa'
}
var s = process.hrtime(); //start timer
console.log(normalRAM); // read from ram
var e = process.hrtime(s) //stop timer
console.log("end0", e[0]); //results in seconds
console.log("end1", e[1]); //results in nanoseconds
This is my Memored test:
// Clustering needed to show Memored in action
if (cluster.isMaster)
{
// Fork workers.
for (let i = 0; i < 1; i++)
{
cluster.fork();
}
}
else
{
var han = {
firstname: 'Han',
lastname: 'Solo'
}
// Store and read
memored.store('character1', han, function ()
{
console.log('Value stored!');
var hrstart = process.hrtime(); // start timer
memored.read('character1', function (err: any, value: any)
{
var hrend = process.hrtime(hrstart) // stop timer
console.log('Read value:', value);
console.log("hrend0", hrend[0]); //results in seconds
console.log("hrend1", hrend[1]); //results in nanoseconds
});
});
}
The results:
The RAM read speeds are around 6500000 nanoseconds.
The Memored read speeds are around 1000000 nanoseconds
Am I testing the speeds incorrectly here? What are the flaws in my methodology? Perhaps my initial assumption is wrong?
I switched the following two lines:
var hrend = process.hrtime(hrstart) // stop timer
console.log('Read value:', value);
To this:
console.log('Read value:', value);
var hrend = process.hrtime(hrstart) // stop timer
Which makes more sense in a real scenario since I would need to read it from RAM like that anyway after the data is returned. The answer to my question is probably "your Memored test is performing faster, because it’s only testing when the data comes back for my callback to use, and not when I actually read it from the 'value' variable".
I wanted to measure the execution time of a particular block of code running on google cloud function( for firebase write event). Can anyone tell me how to do it.
Is there no specific tool to measure the execution time.
I have coded 2 codes. So, I wanted to know which code will have better execution time which in turn gives better performance.
I tried to use process.hrtime() in the following code but it yields different results for same data.
*Algo 1 Running time 299661890
Algo 1 Running time 5684236
Algo 1 Running time 10185061*
start time [ 87, 594147806 ]
'Algo 1 Running time 9251749'
start time [ 22, 803098325 ]
'Algo 1 Running time 1498176261'
// Import the Firebase SDK for Google Cloud Functions.
var functions = require('firebase-functions');
var t0
var mymap= new Map();
exports.processData=functions.database.ref("/test").onWrite(event=>{
const dataValue = event.data.child('data').val()
dataValue.body = myFuction(dataValue.body)
const promise = event.data.ref.child('data').set(dataValue)
'
//Finish Time
t1=process.hrtime(t0)
var RunTime=Math.round((t1[0]*1000000000) + (t1[1]));
console.log("Algo 1 Running time "+RunTime)
return promise;
}
})
function myFunction(s){
// start time
t0=process.hrtime()
var newValue=0
myProbdict.forEach(mapElements);
function mapElements(value, key, map) {
if(newValue< 68){
reduction+=parseInt(value)
var regexstring ="\\b"+`${key}`+"\\b"
var regexp = new RegExp(regexstring, "gi");
s= s.replace(regexp,"#")
}
}
return dataValue
}
You can't expect to have a constant execution time. The time it takes to execute a function is always different because it depends on the current network status (and probably the current server usage as well).
I found a good example in this blog post. The blog writer wrote a function that is supposed to be executed in 1ms:
var start = new Date();
var hrstart = process.hrtime();
setTimeout(function (argument) {
// execution time simulated with setTimeout function
var end = new Date() - start,
hrend = process.hrtime(hrstart);
console.info("Execution time: %dms", end);
console.info("Execution time (hr): %ds %dms", hrend[0], hrend[1]/1000000);
}, 1);
On the first execution, he got the expected result:
Execution time: 1ms
Execution time (hr): 0s 1.025075ms
But on the second execution, the function took a little more than 1ms:
Execution time: 3ms
Execution time (hr): 0s 2.875302ms
If you need to know the execution time of your code block, you can take these outputs and calculate the average: (29961890+5684236+10185061)÷3 which would result in something like 15277062.
No need to do any thing complex like the other answers!
You can simply do this:
console.time(`time spent on complex thing`)
await doSomethingComplex()
console.timeEnd(`time spent on complex thing`)
And then you will see it in the Google Cloud Console > Cloud Functions > select your function > Click "LOGS"
Here is an example of how it will look:
Please note:
The string you put into console.time needs to be the same as the one in console.timeEnd for it to work.
I have multiple functions that call different api endpoints, and I need to call them as quickly as possible without going over some limit (20 calls per second for example). My current solution is to have a delay and call the function once every 50 milliseconds for the example I gave, but I would like to call them as quickly as possible and not just space out the calls equally with the rate limit.
function-rate-limit solved a similar problem for me. function-rate-limit spreads out calls to your function over time, without dropping calls to your function. It still allows instantaneous calls to you function until the rate limit is reached, so it can behave with no latency introduced under normal circumstances.
Example from function-rate-limit docs:
var rateLimit = require('function-rate-limit');
// limit to 2 executions per 1000ms
var start = Date.now()
var fn = rateLimit(2, 1000, function (x) {
console.log('%s ms - %s', Date.now() - start, x);
});
for (var y = 0; y < 10; y++) {
fn(y);
}
results in:
10 ms - 0
11 ms - 1
1004 ms - 2
1012 ms - 3
2008 ms - 4
2013 ms - 5
3010 ms - 6
3014 ms - 7
4017 ms - 8
4017 ms - 9
You can try using queue from async. Be careful when doing this, it essentially behaves like a while(true) in other languages:
const async = require('async');
const concurrent = 10; // At most 10 concurrent ops;
const tasks = Array(concurrent).fill().map((e, i) => i);
let pushBack; // let's create a ref to a lambda function
const myAsyncFunction = (task) => {
// TODO: Swap with the actual implementation
return Promise.resolve(task);
};
const q = async.queue((task, cb) => {
myAsyncFunction(task)
.then((result) => {
pushBack(task);
cb(null, result);
})
.catch((err) => cb(err, null));
}, tasks.length);
pushBack = (task) => q.push(task);
q.push(tasks);
What's happening here? We are saying "hey run X tasks in parallel" and after each task gets completed, we put it back in the queue which is the equivalent of saying "run X tasks in parallel forever"
Is there a way to find out the cpu usage in % for a node.js process via code. So that when the node.js application is running on the server and detects the CPU exceeds certain %, then it will put an alert or console output.
On *nix systems can get process stats by reading the /proc/[pid]/stat virtual file.
For example this will check the CPU usage every ten seconds, and print to the console if it's over 20%. It works by checking the number of cpu ticks used by the process and comparing the value to a second measurement made one second later. The difference is the number of ticks used by the process during that second. On POSIX systems, there are 10000 ticks per second (per processor), so dividing by 10000 gives us a percentage.
var fs = require('fs');
var getUsage = function(cb){
fs.readFile("/proc/" + process.pid + "/stat", function(err, data){
var elems = data.toString().split(' ');
var utime = parseInt(elems[13]);
var stime = parseInt(elems[14]);
cb(utime + stime);
});
}
setInterval(function(){
getUsage(function(startTime){
setTimeout(function(){
getUsage(function(endTime){
var delta = endTime - startTime;
var percentage = 100 * (delta / 10000);
if (percentage > 20){
console.log("CPU Usage Over 20%!");
}
});
}, 1000);
});
}, 10000);
Try looking at this code: https://github.com/last/healthjs
Network service for getting CPU of remote system and receiving CPU usage alerts...
Health.js serves 2 primary modes: "streaming mode" and "event mode". Streaming mode allows a client to connect and receive streaming CPU usage data. Event mode enables Health.js to notify a remote server when CPU usage hits a certain threshold. Both modes can be run simultaneously...
You can use the os module now.
var os = require('os');
var loads = os.loadavg();
This gives you the load average for the last 60seconds, 5minutes and 15minutes.
This doesnt give you the cpu usage as a % though.
Use node process.cpuUsage function (introduced in node v6.1.0).
It shows time that cpu spent on your node process. Example taken from docs:
const previousUsage = process.cpuUsage();
// { user: 38579, system: 6986 }
// spin the CPU for 500 milliseconds
const startDate = Date.now();
while (Date.now() - startDate < 500);
// At this moment you can expect result 100%
// Time is *1000 because cpuUsage is in us (microseconds)
const usage = process.cpuUsage(previousUsage);
const result = 100 * (usage.user + usage.system) / ((Date.now() - startDate) * 1000)
console.log(result);
// set 2 sec "non-busy" timeout
setTimeout(function() {
console.log(process.cpuUsage(previousUsage);
// { user: 514883, system: 11226 } ~ 0,5 sec
// here you can expect result about 20% (0.5s busy of 2.5s total runtime, relative to previousUsage that is first value taken about 2.5s ago)
}, 2000);
see node-usage for tracking process CPU and Memory Usage (not the system)
Another option is to use node-red-contrib-os package