Testing Node.js application that uses Kue - node.js

I would like to test an application that uses Kue so that job queue is empty before each test and cleared after each test. Queue should be fully functional and I need to be able to check status of jobs that are already in the queue.
I tried mock-kue and it worked well until I had to get jobs from the queue and analyze them. I couldn't get it to return jobs by job ID.
Situations that I need to be able to test:
Something happens and there should be a job of a given type in the queue,
Something happens and produces a job. Something else happens and that job gets removed and replaced with another job (rescheduling or existing job).
Seams straightforward, but I have hard time wrapping my head around the problem. All pointers are welcome.

In my experience it's more straightforward to simply have redis running on localhost wherever you want to run your tests rather than dealing with a mocked version of kue.
First, to make sure kue is empty before each test it could be as simple as flushing redis, eg:
var kue = require('kue');
var queue = kue.createQueue();
queue.client.flushdb(function(err) {});
For #1, kue has a rangeByType() method that should solve your problem:
var getJobs = function(type, state, cb) {
kue.Job.rangeByType(type, state, 0, -1, 'asc', cb);
}
// After something happens
getJobs('myJobType', 'active', function(err, jobs) {});
For #2, you can use the same method and simply keep track of the job id to know that it has been replaced:
var jobId;
getJobs('myJobType', 'active', function(err, jobs) {
assert.lengthOf(jobs, 1);
jobId = jobs[0].id;
});
// After the thing happens
getJobs('myJobType', 'active' function(err, jobs) {
assert.lengthOf(jobs, 1);
assert.notEqual(jobId, jobs[0].id);
});
And if you ever need to query a job by ID you can do it like so:
kue.Job.get(jobId, function(err, job) {});

Take a look at the kue-mock lib, it is more likely for integration testing than unit.
The library doesn't hack on any kue's internals (replacing/overriding methods etc.). Instead, it creates the original queue instance with a separate redis namespace, then, when stubbing, it creates job process handlers on the fly, putting its own implementation that gives you the ability to control the job processing behaviour.
Example usage:
const expect = require('chai').expect;
const kue = require('kue');
const KueMock = require('kue-mock');
const $queue = new KueMock(kue);
const app = require('./your-app-file');
describe('functionality that deals with kue', () => {
before(() => $queue.clean());
afterEach(() => $queue.clean());
it('enqueues a job providing some correct data', () => {
let jobData;
$queue.stub('your job type', (job, done) => {
jobData = job.data;
done();
});
return yourJobRunnerFunction()
.then(() => {
expect(jobData).to.be.an('object')
.that.is.eql({ foo: 'bar' });
});
});
describe('when the job is completed', () => {
beforeEach(() => {
$queue.stub('your job type')
.yields(null, { baz: 'qux' });
});
it('correctly handles the result', () => {
return yourJobRunnerFunction()
.then((result) => {
expect(result).to.eql({ baz: 'qux' });
});
});
// ...
});
describe('when the job is failed', () => {
beforeEach(() => {
$queue.stub('your job type')
.yields(new Error('Oops!'));
});
it('correctly handles the job result', () => {
return yourJobRunnerFunction()
.catch((err) => {
expect(err).to.be.an('error')
.with.property('message', 'Oops!');
});
});
// ...
});
});

Related

Trigger the execution of a function if any condition is met

I'm writing an HTTP API with expressjs in Node.js and here is what I'm trying to achieve:
I have a regular task that I would like to run regularly, approx every minute. This task is implemented with an async function named task.
In reaction to a call in my API I would like to have that task called immediately as well
Two executions of the task function must not be concurrent. Each execution should run to completion before another execution is started.
The code looks like this:
// only a single execution of this function is allowed at a time
// which is not the case with the current code
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
I've put a full working sample project at https://github.com/piec/question.js
If I were in go I would do it like this and it would be easy, but I don't know how to do that with Node.js.
Ideas I have considered or tried:
I could apparently put task in a critical section using a mutex from the async-mutex library. But I'm not too fond of adding mutexes in js code.
Many people seem to be using message queue libraries with worker processes (bee-queue, bullmq, ...) but this adds a dependency to an external service like redis usually. Also if I'm correct the code would be a bit more complex because I need a main entrypoint and an entrypoint for worker processes. Also you can't share objects with the workers as easily as in a "normal" single process situation.
I have tried RxJs subject in order to make a producer consumer channel. But I was not able to limit the execution of task to one at a time (task is async).
Thank you!
You can make your own serialized asynchronous queue and run the tasks through that.
This queue uses a flag to keep track of whether it's in the middle of running an asynchronous operation already. If so, it just adds the task to the queue and will run it when the current operation is done. If not, it runs it now. Adding it to the queue returns a promise so the caller can know when the task finally got to run.
If the tasks are asynchronous, they are required to return a promise that is linked to the asynchronous activity. You can mix in non-asynchronous tasks too and they will also be serialized.
class SerializedAsyncQueue {
constructor() {
this.tasks = [];
this.inProcess = false;
}
// adds a promise-returning function and its args to the queue
// returns a promise that resolves when the function finally gets to run
add(fn, ...args) {
let d = new Deferred();
this.tasks.push({ fn, args: ...args, deferred: d });
this.check();
return d.promise;
}
check() {
if (!this.inProcess && this.tasks.length) {
// run next task
this.inProcess = true;
const nextTask = this.tasks.shift();
Promise.resolve(nextTask.fn(...nextTask.args)).then(val => {
this.inProcess = false;
nextTask.deferred.resolve(val);
this.check();
}).catch(err => {
console.log(err);
this.inProcess = false;
nextTask.deferred.reject(err);
this.check();
});
}
}
}
const Deferred = function() {
if (!(this instanceof Deferred)) {
return new Deferred();
}
const p = this.promise = new Promise((resolve, reject) => {
this.resolve = resolve;
this.reject = reject;
});
this.then = p.then.bind(p);
this.catch = p.catch.bind(p);
if (p.finally) {
this.finally = p.finally.bind(p);
}
}
let queue = new SerializedAsyncQueue();
// utility function
const sleep = function(t) {
return new Promise(resolve => {
setTimeout(resolve, t);
});
}
// only a single execution of this function is allowed at a time
// so it is run only via the queue that makes sure it is serialized
async function task(reason: string) {
function runIt() {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
return queue.add(runIt);
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
Here's a version using RxJS#Subject that is almost working. How to finish it depends on your use-case.
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<string>();
const effect$ = run.pipe(
// Limit one task at a time
concatMap(task),
share()
);
const effectSub = effect$.subscribe();
interval(5000).subscribe(_ =>
run.next("ticker")
);
// call task immediately
app.get("/task", async (req, res) => {
effect$.pipe(
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next("trigger");
});
The issue here is that res.send("ok") is linked to the effect$ streams next emission. This may not be the one generated by the run.next you're about to call.
There are many ways to fix this. For example, you can tag each emission with an ID and then wait for the corresponding emission before using res.send("ok").
There are better ways too if calls distinguish themselves naturally.
A Clunky ID Version
Generating an ID randomly is a bad idea, but it gets the general thrust across. You can generate unique IDs however you like. They can be integrated directly into the task somehow or can be kept 100% separate the way they are here (task itself has no knowledge that it's been assigned an ID before being run).
interface IdTask {
taskId: number,
reason: string
}
interface IdResponse {
taskId: number,
response: any
}
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<IdTask>();
const effect$: Observable<IdResponse> = run.pipe(
// concatMap only allows one observable at a time to run
concatMap((eTask: IdTask) => from(task(eTask.reason)).pipe(
map((response:any) => ({
taskId: eTask.taskId,
response
})as IdResponse)
)),
share()
);
const effectSub = effect$.subscribe({
next: v => console.log("This is a shared task emission: ", v)
});
interval(5000).subscribe(num =>
run.next({
taskId: num,
reason: "ticker"
})
);
// call task immediately
app.get("/task", async (req, res) => {
const randomId = Math.random();
effect$.pipe(
filter(({taskId}) => taskId == randomId),
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next({
taskId: randomId,
reason: "trigger"
});
});

Is there any programmatic way to break infinite loop in NodeJS?

Ultimately speaking - is there a practical method (also maybe by inserting some JS constructs into the code) to break or halt the long lasting JS code during the execution? For example: can it be interrupted by some process.* object constructs, or similar? Or the other way? The valid solution may even include the NodeJS process to be killed and/or restarted. Thank you!
EDIT:
I need to execute some particular user code on the server, using Function clause (ala eval, - let alone security concerns). I cannot insert any extra code inside it, only enclose it. What I need is to have a possibility to break user code after 5 minutes, if it is not finished by this time. For example:
usercode = 'Some code from the user';
pre_code = 'some controlling code for breaking the user code';
post_code = 'another controlling code';
fcode = pre_code + usercode + post_code;
<preparations for breaking usercode>
(new Function(fcode))(); // This MUST exit in 5 minutes
Edit:
Answering your edit. I see the intention now. If it is running in nodejs, you can use worker_thread for that https://nodejs.org/api/worker_threads.html#worker_threads_worker_workerdata.
For example:
// main.js
const runCode = (code) => {
const worker = new Worker("./code-executor.js", { workerData: { code: guestCode } });
const promise = new Promise((resolve) => {
setTimeout(() => worker.kill(), 60000 * 5);
worker.on("error", () => {
return reject(new SomeCustomError())
});
worker.on("message", (message) => {
if(message.success) return resolve(message.result);
return reject(new Error(message.error));
});
});
promise.finally(() => { worker.kill() });
return promise;
}
// code-executor.js
const { workerData, parentPort } = require("worker_threads");
const { code } = workerData;
Promise.resolve()
.then(() => (new Function(fcode))())
.then((result) => {
parentPort.postMessage({
success: true,
result: value
})
})
.catch((error) => {
parentPort.postMessage({
success: true,
error: error.message
})
});
If it's in browser https://developer.mozilla.org/en-US/docs/Web/API/Worker
The WebAPI is not exactly the same but the logic should be similar
Original
Killing a process. Also read: https://nodejs.org/api/process.html#process_signal_events
process.kill(pid, "SIGINT")
"Killing" a long running function, you gotta hack a bit. There's no elegant solution. Inject a controller which can be mutated outside of the long running function. To stop it from the outside, set controller.isStopped = true
export const STOP_EXECUTION = Symbol();
function longRunning(controller){
... codes
// add stopping point
if(controller.isStopped) throw STOP_EXECUTION;
... codes
// add stopping point
if(controller.isStopped) throw STOP_EXECUTION;
... codes
}
// catch it by
try{
longRunnning();
}catch(e){
switch(true){
e === STOP_EXECUTION: ...; // the longRunning function is stopped from the outside
default: ...; // the longRunning function is throwing not because of being stopped
}
}
Gist: https://gist.github.com/Kelerchian/3824ca4ce1be390d34c5147db671cc9b

Using Mocha for integration testing of a callback that should trigger when on an AMQP message received event

I have a Feathers application that is using RabbitMQ and a custom amqplib wrapper to communicate with some other code running elsewhere and I'm struggling to write a good integration test to show that the callback that runs when a message is received runs correctly. The actual callback just takes the body of the received message and calls an internal service to put the data in the database.
I have a RabbitMQ server running in the test environment, and the idea was to write a test that publishes some dummy data to the correct exchange, and then check that the data ends up in the database. The problem is that I can't work out how to tell that the callback has finished before I check the database.
Right now, I just publish the message and then use a timeout to wait a few seconds before checking the database, but I don't like this since there is no guarantee that the callback will have completed.
The code I'm testing looks something like this (not the actual code just an example):
const app = require('./app');
// handleAMQP is passed as a callback to the consumer
// it creates a new record in the myService database
const handleAMQP = async(message) => {
await app.service('users').create(message.content);
};
// Subscribe takes an amqp connection, opens a channel, and connects a callback
const subscribe = (conn) => {
let queue = 'myQueue';
let exchange = 'myExchange';
return conn.createChannel().then(function (ch) {
var ok = ch.assertExchange(exchange, 'topic', { durable: true });
ok = ok.then(function () {
return ch.assertQueue(queue, { exclusive: true });
});
ok = ok.then(function (qok) {
var queue = qok.queue;
ch.bindQueue(queue, exchange, topic);
});
ok = ok.then(function (queue) {
return ch.consume(queue, handleAMQP);
});
});
};
module.exports = {subscribe};
And my test looks something like this:
const assert = require('assert');
const amqp = require('amqplib');
describe('AMQP Pub/Sub Tests', async () => {
let exchange = 'myExchange';
let topic = 'myTopic';
let dummyData = {
email: 'example#example.com',
name: 'Example User'
}
it('creates a new db enry when amqp message recieved', async () => {
// Publish some dummy data
await amqp.connect('amqp://localhost').then((conn) => {
conn.createChannel().then((ch) => {
ch.assertExchange(exchange, 'topic', {durable: true}).then(() => {
ch.publish(exchange, topic, dummyData).then(() => {
ch.close();
})
});
});
});
await setTimeout(() => { // Wait three seconds
let result = app.service('users').find({email : 'example#example.com'}); // Attempt to find the newly created user
assert.deepEqual(result.email, dummyData.email);
assert.deepEqual(result.name, dummyData.name);
}, 3000);
});
});
Instead of just waiting an arbitrary time limit before I check if the record exists, is there a better way to structure this test?
Or is waiting a certain time a totally valid for event-driven functionality?

Why am I getting "Cannot log after tests are done?"

This is my first foray into jest. I am attempting to run the "stock" test for my service, i.e., the one feathersjs sets up for you when it creates the service via the cli. The service uses an asychronous function with a callback, where I do some logging to the console as a final step. I am getting the error: "Cannot log after tests are done. Did you forget to wait for something async in your test?" I suspect that the test isn't waiting for the callback to complete before exiting, so when the callback executes tries to log after the tests finish.
Here's the stock test:
it('registered the service', () => {
const service = app.service('order');
expect(service).toBeTruthy();
});
I've tried some techniques mentioned in the documentation, like async/await:
it('registered the service', async () => {
const service = await app.service('order');
expect(service).toBeTruthy();
});
and I've tried using "done":
I get the same error message each time.
Here's the part of the service code that's doing the logging:
amqp.connect(amqp_url, (err0, conn) => {
if(err0) {
throw(err0);
}
conn.createChannel((err1, ch) => {
if(err1) {
throw(err1);
}
ch.assertQueue(orchToOrderQName);
ch.consume(orchToOrderQName, function(msg) {
ch.ack(msg);
service.emit('create', msg);
});
orchToOrderChannel = ch;
});
conn.createChannel((err1, ch) => {
if(err1) {
throw(err1);
}
ch.assertQueue(orderToOrchQName);
orderToOrchChannel = ch;
console.log(" [*] Order is consuming messages from %s.", orchToOrderQName);
});
});
I think I need to find a way for the test to wait on the callback, but maybe the problem is elsewhere.
I was able to get around this by changing the stock test:
service = null;
beforeAll(async () => {
service = await app.service('order');
});
it('registered the service', () => {
expect(service).toBeTruthy();
});
and I no longer get the error message about logging.
This answer was posted as an edit to the question Why am I getting "Cannot log after tests are done?" by the OP boing under CC BY-SA 4.0.

NodeJS cron job - Mongoose won't execute a .find on a model via a function called in a cron tab

I'm having this weird situation where my Cron Job is successfully executing a function that returns a promise, however, when it tries to execute a .find() on the Model, it never actually executes. I have this same function used elsewhere in my app and is called via an API call and returns no problem. Is there something I'm missing?
Here is my cron script:
var CronJob = require('node-cron');
var TradeService = require('../services/TradeService');
// Setup the cron job to fire every second
CronJob.schedule('* * * * * *', function() {
console.log('You will see this message every second');
TradeService.executePendingTrades();
}, null, true, 'America/Los_Angeles');
Here are the related functions that get called:
exports.executePendingTrades = () => {
// Get all pending trades
exports.getPendingTrades().then(results => {
console.log('results', results); // This never fires
})
}
exports.getPendingTrades = () => {
return new Promise((resolve, reject) => {
Trades.find({})
.where('is_canceled').equals('false')
.where('is_completed').equals('false')
.sort('-created_at')
.exec( (err, payload) => {
if (err) {
return reject(err); // This never fires
}
return resolve(payload); // This never fires
})
});
}
This is a shot in the dark, but make sure you are starting a database connection in your CRON job. Otherwise you won't be able to execute any queries.

Resources