We have a number of "poller" class instances running on a dedicated deploy of our app. When SIGTERM is received, we want each of these pollers to gracefully shutdown. We've implemented an async beforeApplicationShutdown method on our base poller class to that effect.
For our use case, each of these methods can be run in parallel. But it seems they are run in sequence by NestJS.
A negative consequence of this behavior is that the app can take a long time to spin down. ECS gives us 30 seconds from SIGTERM to SIGKILL, which we could extend as we add more pollers, but I'd rather not lengthen our deploy times.
I ended up implementing a solution like this:
const app: NestApplication;
// start the app, etc...
process.on('SIGTERM', async () => {
const pollers = getAllPollers(app);
// In parallel, shut down all pollers.
await Promise.all(pollers.map((poller) => poller.stopPolling()));
await app.close();
process.exit(0);
});
function getAllPollers(app: NestApplication): BasePoller[] {
const pollers: BasePoller[] = [];
app
.get(DiscoveryService)
.getProviders()
.forEach((instanceWrapper) => {
const { instance } = instanceWrapper;
if (instance instanceof BasePoller) {
pollers.push(instance);
}
});
return pollers;
}
Related
I have a worker method 'doSomeWork' that is called in a loop, based on a flag that will be changed if a signal to terminate is received.
let RUNNING = true;
let pid;
export function getPid() {
return pid;
}
export async function doSomeWork() {
console.log("Doing some work!");
}
export const run = async () => {
console.log("starting run process with PID %s", process.pid);
pid = process.pid;
while (RUNNING) {
await doSomeWork();
}
console.log("done");
};
run()
.then(() => {
console.log("finished");
})
.catch((e) => console.error("failed", e));
process.on("SIGTERM", () => {
RUNNING = false;
});
I am happy with this and now need to write a test: I want to
trigger the loop
inject a 'SIGTERM' to the src process
give the loop a chance to finish nicely
see 'finished' in the logs to know that the run method has been killed.
here is my attempt (not working) The test code all executes, but the src loop isn't killed.
import * as main from "../src/program";
describe("main", () => {
it("a test", () => {
main.run();
setTimeout(function () {
console.log("5 seconds have passed - killing now!");
const mainProcessPid = main.getPid();
process.kill(mainProcessPid, "SIGTERM");
}, 5000);
setTimeout(function () {
console.log("5 secs of tidy up time has passed");
}, 5000);
});
});
I think the setTimeout isn't blocking the test thread, but I am not sure how to achieve this in node/TS.
sandbox at https://codesandbox.io/s/exciting-voice-goncm
update sandbox with correct environment: https://codesandbox.io/s/keen-bartik-ltjtx
any help appreciated :-)
--update---
I now see that process.kill isn't doing what I thought it was - even when I pass in the PID. will try creating a process as a child of the test process, so I can send a signal to it then. https://medium.com/#NorbertdeLangen/communicating-between-nodejs-processes-4e68be42b917
You are getting this issue because the Environment in your codesandbox is create-react-app i.e. it's a client side script and not a server-side instance of node.
Recreate you project but select as your environment node HTTP server, this will give you a node environment where the node process functions will work e.g. process.kill. This is because the node environment is run in a server-side Docker container. See here for more info on Codesandbox's environments.
I am struggling with node.js 'worker_threads'.
So what I wanted to do is to pass several instances of my custom class into the worker thread.
Instances are assigned to map by some unique serial number.
So basically, I have got a Map of type - <string, MyUniqueClassInstance>.
My worker implementation looks like this:
Class method running a worker service:
public static runService = (workerData:any) => {
return new Promise((resolve, reject) => {
const route = path.join(__dirname, '/worker.js');
const worker = new Worker(route, { workerData });
worker.on('message', resolve);
worker.on('error', reject);
worker.on('exit', (code:number) => {
if (code !== 0)
reject(new Error(`Worker stopped with exit code ${code}`));
})
})
}
worker itself:
const { workerData, parentPort } = require('worker_threads')
const instance = new Counter();
const {items, myMap} = workerData;
const pResponseList:Promise<any>[] = [];
items.map((item: Item) => {
pResponseList.push(
instance.count(item, myMap.get(item._id)!)
);
});
Promise.all(pResponseList).then(res => parentPort.postMessage(res));
And whenever, inside 'count' method I try to run a method from item instance it throws an error
myMapEntry.myCustomInstanceMethod is not a function
I tried to console.log() content of my instance just before passing it to .count() method and everything is correctly settled.
Same pattern runs flawlessly outside of worker instance.
Could anyone help me to find out what exactly could potentially be wrong inside this code?
You can't pass functions (e.g. instances of classes won't work, at least their methods won’t) - you can only pass serializable data.
https://nodejs.org/api/worker_threads.html#worker_threads_worker_workerdata
An arbitrary JavaScript value that contains a clone of the data passed to this thread’s Worker constructor.
The data is cloned as if using postMessage(), according to the HTML structured clone algorithm.
Then let's go to HTML structured clone algorithm
Things that don't work with structured clone
Function objects cannot be duplicated by the structured clone algorithm; attempting to throws a DATA_CLONE_ERR exception.
Could you perhaps reconstruct the instance of the class using serialized data, call your methods, and then return the serialized data back to the parent thread?
I'm creating a program where I constantly run and stop async code, but I need a good way to stop the code.
Currently, I have tried to methods:
Method 1:
When a method is running, and another method is called to stop the first method, I start an infinite loop to stop that code from running and then remove the method from the queue(array)
I'm 100% sure that this is the worst way to accomplish it, and it works very buggy.
Code:
class test{
async Start(){
const response = await request(options);
if(stopped){
while(true){
await timeout(10)
}
}
}
}
Code 2:
var tests = [];
Start(){
const test = new test();
tests.push(test)
tests.Start();
}
Stop(){
tests.forEach((t, i) => {t.stopped = true;};
tests = [];
}
Method 2:
I load the different methods into Workers, and when I need to stop the code, I just terminate the Worker.
It always takes a lot of time(1 sec) to create the Worker, and therefore not the best way, since I need the code to run without 1-2 sec pauses.
Code:
const Worker = require("tiny-worker");
const code = new Worker(path.resolve(__dirname, "./Code/Code.js"))
Stopping:
code.terminate()
Is there any other way that I can stop async code?
The program contains Request using nodejs Request-promise module, so program is waiting for requests, it's hard to stop the code without one of the 2 methods.
Is there any other way that I can stop async code?
Keep in mind the basic of how Nodejs works. I think there is some misunderstanding here.
It execute the actual function in the actual context, if encounters an async operation the event loop will schedule it's execetution somewhere in the future. There is no way to remove that scheduled execution.
More info on event loop here.
In general for manage this kind of situations you shuold use flags or semaphores.
The program contains Request using nodejs Request-promise module, so program is waiting for requests, it's hard to stop the code
If you need to hard "stop the code" you can do something like
func stop() {
process.exit()
}
But if i'm getting it right, you're launching requests every x time, at some point you need to stop sending the request without managing the response.
You can't de-schedule the response managemente portion, but you can add some logic in it to (when it will be runned) check if the "request loop" has been stopped.
let loop_is_stopped = false
let sending_loop = null
func sendRequest() {
const response = await request(options) // "wait here"
// following lines are scheduled after the request promise is resolved
if (loop_is_stopped) {
return
}
// do something with the response
}
func start() {
sending_loop = setInterval(sendRequest, 1000)
}
func stop() {
loop_is_stopped = true
clearInterval(sending_loop)
}
module.exports = { start, stop }
We can use Promise.all without killing whole app (process.exit()), here is my example (you can use another trigger for calling controller.abort()):
const controller = new AbortController();
class Workflow {
static async startTask() {
await new Promise((res) => setTimeout(() => {
res(console.log('RESOLVE'))
}, 3000))
}
}
class ScheduleTask {
static async start() {
return await Promise.all([
new Promise((_res, rej) => { if (controller.signal.aborted) return rej('YAY') }),
Workflow.startTask()
])
}
}
setTimeout(() => {
controller.abort()
console.log("ABORTED!!!");
}, 1500)
const run = async () => {
try {
await ScheduleTask.start()
console.log("DONE")
} catch (err) {
console.log("ERROR", err.name)
}
}
run()
// ABORTED!!!
// RESOLVE
"DONE" will never be showen.
res will be complited
Maybe would be better to run your code as script with it's own process.pid and when we need to interrupt this functionality we can kill this process by pid in another place of your code process.kill.
I was trying to create an automation script for work, it is supposed to use multiple puppeteer instances to process input strings simultaneously.
the task queue and number of puppeteer instances are controlled by the package generic-pool,
strangely, when i run the script on ubuntu or debian, it seems that it fells into an infinite loop. tries to run infinite number of puppeteer instances. while when run on windows, the output was normal.
const puppeteer = require('puppeteer');
const genericPool = require('generic-pool');
const faker = require('faker');
let options = require('./options');
let i = 0;
let proxies = [...options.proxy];
const pool = genericPool.createPool({
create: async () => {
i++;
console.log(`create instance ${i}`);
if (!proxies.length) {
proxies = [...options.proxy];
}
let {control = null, proxy} = proxies.pop();
let instance = await puppeteer.launch({
headless: true,
args: [
`--proxy-server=${proxy}`,
]
});
instance._own = {
proxy,
tor: control,
numInstance: i,
};
return instance;
},
destroy: async instance => {
console.log('destroy instance', instance._own.numInstance);
await instance.close()
},
}, {
max: 3,
min: 1,
});
async function run(emails = []) {
console.log('Processing', emails.length);
const promises = emails.map(email => {
console.log('Processing', email)
pool.acquire()
.then(browser => {
console.log(`${email} handled`)
pool.destroy(browser);})
})
await Promise.all(promises)
await pool.drain();
await pool.clear();
}
let emails = [a,b,c,d,e,];
run(emails)
Output
create instance 1
Processing 10
Processing Stacey_Haley52
Processing Polly.Block
create instance 2
Processing Shanny_Hudson59
Processing Vivianne36
Processing Jayda_Ullrich
Processing Cheyenne_Quitzon
Processing Katheryn20
Processing Jamarcus74
Processing Lenore.Osinski
Processing Hobart75
create instance 3
create instance 4
create instance 5
create instance 6
create instance 7
create instance 8
create instance 9
is it because of my async functions? How can I fix it?
Appreciate your help!
Edit 1. modified according to #James suggested
The main problem you are trying to solve,
It is supposed to use multiple puppeteer instances to process input strings simultaneously.
Promise Queue
You can use a rather simple solution that involves a simple promise queue. We can use p-queue package to limit the concurrency as we wish. I used this on multiple scraping projects to always test things out.
Here is how you can use it.
// emails to handle
let emails = [a, b, c, d, e, ];
// create a promise queue
const PQueue = require('p-queue');
// create queue with concurrency, ie: how many instances we want to run at once
const queue = new PQueue({
concurrency: 1
});
// single task processor
const createInstance = async (email) => {
let instance = await puppeteer.launch({
headless: true,
args: [
`--proxy-server=${proxy}`,
]
});
instance._own = {
proxy,
tor: control,
numInstance: i,
};
console.log('email:', email)
return instance;
}
// add tasks to queue
for (let email of emails) {
queue.add(async () => createInstance(email))
}
Generic Pool Infinite Loop Problem
I removed all kind of puppeteer related code from your sample code and saw how it was still producing the infinite output to console.
create instance 70326
create instance 70327
create instance 70328
create instance 70329
create instance 70330
create instance 70331
...
Now, if you test few times, you will see it will throw the loop only if you something on your code is crashing. The culprit is this pool.acquire() promise, which is just re queuing on error.
To find what is causing the crash, use the following events,
pool.on("factoryCreateError", function(err) {
console.log('factoryCreateError',err);
});
pool.on("factoryDestroyError", function(err) {
console.log('factoryDestroyError',err);
});
There are some issues related to this:
acquire() never resolves/rejects if factory always rejects, here.
About the acquire function in pool.js, here.
.acquire() doesn't reject when resource creation fails, here.
Good luck!
You want to return from your map rather than await, also don't await inside the destroy call, return the result and you can chain these e.g.
const promises = emails.map(e => pool.acquire().then(pool.destroy));
Or alternatively, you could just get rid of destroy completely e.g.
pool.acquire().then(b => b.close())
I am writing a worker that uses amqplib's Channel#consume method. I want this worker to wait for jobs and process them as soon as they appear in the queue.
I wrote my own module to abstract away ampqlib, here are the relevant functions for getting a connection, setting up the queue and consuming a message:
const getConnection = function(host) {
return amqp.connect(host);
};
const createChannel = function(conn) {
connection = conn;
return conn.createConfirmChannel();
};
const assertQueue = function(channel, queue) {
return channel.assertQueue(queue);
};
const consume = Promise.method(function(channel, queue, processor) {
processor = processor || function(msg) { if (msg) Promise.resolve(msg); };
return channel.consume(queue, processor)
});
const setupQueue = Promise.method(function setupQueue(queue) {
const amqp_host = 'amqp://' + ((host || process.env.AMQP_HOST) || 'localhost');
return getConnection(amqp_host)
.then(conn => createChannel(conn)) // -> returns a `Channel` object
.tap(channel => assertQueue(channel, queue));
});
consumeJob: Promise.method(function consumeJob(queue) {
return setupQueue(queue)
.then(channel => consume(channel, queue))
});
My problem is with Channel#consume's odd signature. From http://www.squaremobius.net/amqp.node/channel_api.html#channel_consume:
#consume(queue, function(msg) {...}, [options, [function(err, ok) {...}]])
The callback is not where the magic happens, the message's processing should actually go in the second argument and that breaks the flow of promises.
This is how I planned on using it:
return queueManager.consumeJob(queue)
.then(msg => {
// do some processing
});
But it doesn't work. If there are no messages in the queue, the promise is rejected and then if a message is dropped in the queue nothing happens. If there is a message, only one message is processed and then the worker stalls because it exited the "processor" function from the Channel#consume call.
How should I go about it? I want to keep the queueManager abstraction so my code is easier to reason about but I don't know how to do it... Any pointers?
As #idbehold said, Promises can only be resolved once. If you want to process messages as they come in, there is no other way than to use this function. Channel#get will only check the queue once and then return; it wouldn't work for a scenario where you need a worker.
just as an option. You can present your application as a stream of some messages(or events). There is a library for this http://highlandjs.org/#examples
Your code should look like this(it isn`t a finished sample, but I hope it illustrates the idea):
let messageStream = _((push, next) => {
consume(queue, (msg) => {
push(null, msg)
})
)
// now you can operate with your stream in functional style
message.map((msg) => msg + 'some value').each((msg) => // do something with msg)
This approach provides you a lot of primitives for synchronization and transformation
http://highlandjs.org/#examples