I have a restify api similar to this (i write in pseudocode) :
server.post('api/import')
{
database.write(status of this file.id is pending)
fileModification(req.file)
res.status(200)
res.send(import has started)
} //here I do some file modifications and then i import it to database
server.get('api/import_info')
{
database.select(file status)
} //here I want to see status (is file imported or pending(process is not finished yet))
//In another module after import is finished I update database to
database.write(file.id import status is completed)
Importing file is process that takes about 2 minutes, but even I don't wait for it to finish in api/import when I try to trigger 'info' route my api is blocked
Is it possible that event loop is blocked or maybe connection is not properly closed.
Thanks in advance
I have some ideas about your question.
you can use cluster module Cluster, cluster module can create process depend on your cpu core. When on process blocked, Others process still can work.
you can fork a new process in your api, use the new process handle your task.
Related
I have a Chrome extension where I want to be able to initialize settings if not already set (e.g., upon installation).
I want to do this in a way that the data can be accessed from content scripts and options pages under an assumption that the data has been initialized. Under Manifest V2, I used a background page and initialized the data synchronously with localStorage, which is no longer available under Manifest V3.
A candidate approach would be to run the following code from a service worker:
chrome.storage.local.get(['settings'], function(storage) {
if (!storage.settings) {
chrome.storage.local.set({settings: defaultSettings()});
}
});
However, that seems not guaranteed to work, similar to the example from the migration documentation, since the service worker could be terminated prior to the completion of the asynchronous handling. Additionally, even if the service worker is not terminated, it seems to not be guaranteed that the data would be initialized by time the content script executes.
I would like to initialize the data in a way that it would be guaranteed to be set and available by time a content script executes. A workaround could be to check if the data has been properly initialized in the content script, and use a fallback default value otherwise. Is there some alternative approach that could avoid this extra handling?
since the service worker could be terminated prior to the completion of the asynchronous handling
SW doesn't terminate at any time, there are certain rules, the simplest rule of thumb is that it lives for 30 seconds, the time is auto-prolonged by 30 seconds whenever a subscribed chrome API event is triggered, and by 5 minutes when you have an open chrome.runtime messaging channel/port (currently such ports are auto-closed after 5 minutes in MV3).
So, assuming your example code runs right away, not after a timeout of 29.999 seconds, SW won't terminate during the API call as the storage API takes just a few milliseconds to complete, not 30 seconds. The documentation on ManifestV3 probably tried too hard to sell the non-existent benefits of switching extensions to service workers, so fact-checking was secondary when the article was written.
even if the service worker is not terminated, it seems to not be guaranteed that the data would be initialized by time the content script executes.
Yes.
Some solutions:
Include the defaults in your content scripts and other places. It's the simplest solution that also works in case the user clears the storage via devtools (Storage Area Explorer extension or a console command).
Use chrome.scripting.registerContentScripts (instead of declaring content_scripts in manifest.json) after initializing the storage inside chrome.runtime.onInstalled.
Use messaging in content/options scripts instead of chrome.storage so that the background script is the sole source of truth. When the service worker is already running, messaging will be actually faster as chrome.storage is very slow in Chrome, both local and sync variants, so to make caching truly effective you can use chrome.runtime ports to prolong the lifetime of the service worker to 5 minutes or longer as shown here.
background script:
let settings;
let busy = chrome.storage.local.get('settings').then(r => {
busy = null;
settings = r.settings;
if (!settings) {
settings = defaultSettings();
chrome.storage.local.set({settings});
}
return settings;
});
chrome.runtime.onMessage.addListener((msg, sender, sendResponse) => {
if (msg === 'getSettings') {
if (busy) {
busy.then(sendResponse);
return true;
} else {
sendResponse(settings)
}
}
});
content/option script:
chrome.runtime.sendMessage('getSettings', settings => {
// use settings inside this callback
});
I want to develop a node.js program that will be executed at a specific time using a job scheduler (node-schedule).
This program is running in the background using forever (node.js module).
Here's the content of my app.js :
var schedule = require('node-schedule');
~
~
var id = request.body.id;
var scheduled = schedule.scheduledJobs;
if(scheduled[id]!=null){
//Tasks
}
else{
scheduled[id].cancel;
delete scheduled[id];
}
But if app.js is killed by any reason,
the schedule object is removed.
and sometimes app.js is restarted by forever.
how can I handle node-schedule objects ?
I have faced similar problem recently and there are two solutions:
1. Use actual cron
2. Use database
I solved my problem by using database. Each time you are creating some event save it to database. In your app.js file when the application is starting make function reading the database and creating scheduled events accordingly.
The first option is better if you do not have dynamic tasks, you do not create new tasks or if they are always the same.
So I've created a service file that fires off a message listener. This service file has some logic in it to make sure that a worker process is fired...
Basically:
_parentProcess = function() { // logic to determine if process is parent) }
if (!_parentProcess()) {
_createParent();
} else {
_executeWorker();
}
_createParent will fork off the service.js file with a flag so that the next time the process runs, we are in a child/worker process.
The Worker process is what fires off my listener, now the problem I'm trying to wrap my head around is whether or not this is enough resource management? The listener gets a message that tells it to fire off some app. This app may take anywhere between 10 seconds to complete and 120 seconds to complete.
If it crashes, obviously, the service.js file handles that and just spins up another one but I'm more worried about blocking and using the most of my machine. Should I fork again in the listener the actual applications that I'm going to fire off or is this enough?
I am working in nodejs with express for a web app that communicates with mongodb frequently. Currently I running production with my own job queue system that only begins processing a job once the previous job has been completed (an approach that seems to be taken by kue).
To me, this seems wildly inefficient, I was hoping for a more asynchronous job queue, so I am looking for some advice as to how other nodejs developers queue their jobs and structure their processing.
One of my ideas is to process any jobs that are received immediately and return the resulting data in the order of addition.
Also to be considered: currently each user has their own independent job queue instance, is this normal practice? Is there any reason why this should not be the case? (ie, all users send jobs to one universal queue?)
Any comments/advice are appreciated.
Why do you build your own queue system? You did quite a lot of work to serialize a async queue with addLocalJob.
Why don't you just do something like
on('request', function(req, res) { queryDB(parameter, function(result) { res.send(result) })
? Full parallel access, no throttle, no (async) waiting.
If you really want to do it "by hand" in your own code, why not execute the first n elements of your trafficQueue instead of only the first?
If you want to throttle the DB - two ways:
use a library like async and the function parallelLimit
connect to your mongodb with http(s) and use the node-build-in http.globalAgent.maxSockets.
Hope this helps.
I am writing a web service which has to be able to reply to multiple http requests.
From what I understand, I will need to deal with HttpListener.
What is the best method to receive a http request(or better, multiple http requests), translate it and send the results back to the caller? How safe is to use HttpListeners on threads?
Thanks
You typically set up a main thread that accepts connections and passes the request to be handled by either a new thread or a free thread in a thread pool. I'd say you're on the right track though.
You're looking for something similar to:
while (boolProcessRequests)
{
HttpListenerContext context = null;
// this line blocks until a new request arrives
context = listener.GetContext();
Thread T = new Thread((new YourRequestProcessorClass(context)).ExecuteRequest);
T.Start();
}
Edit Detailed Description If you don't have access to a web-server and need to roll your own web-service, you would use the following structure:
One main thread that accepts connections/requests and as soon as they arrive, it passes the connection to a free threat to process. Sort of like the Hostess at a restaurant that passes you to a Waiter/Waitress who will process your request.
In this case, the Hostess (main thread) has a loop:
- Wait at the door for new arrivals
- Find a free table and seat the patrons there and call the waiter to process the request.
- Go back to the door and wait.
In the code above, the requests are packaged inside the HttpListernContext object. Once they arrive, the main thread creates a new thread and a new RequestProcessor class that is initialized with the request data (context). The RequsetProcessor then uses the Response object inside the context object to respond to the request. Obviously you need to create the YourRequestProcessorClass and function like ExecuteRequest to be run by the thread.
I'm not sure what platform you're on, but you can see a .Net example for threading here and for httplistener here.