I've built some custom middleware on Node.js for a client which runs great in user space, but I want to make it a service.
I've accomplished this using node-windows, which works great, but the client has occasional large bursts of data so I'd like to allocate a little more memory using the --max-old-space-size command line parameter. Unfortunately, I don't see how to configure that in my service set-up wrapper for node-windows.
Any suggestions?
FWIW, I'm also thinking about changing how I parse the data, e.g. treating it more as a stream, but since this is the first time I've used Node and the project is going live in a couple of days, I'm hoping to find a quick and dirty option that'll get us to an up-and-running status easily, to be adjusted later.
Thanks!
Use node-windows v0.1.14 or higher. The ability to add flags was merged in this version. The more appropriate issue related to this is https://github.com/coreybutler/node-windows/issues/159.
Related
I have a server application using Node, and sometimes I need to run some script in it. Some examples of scenarios when this would be necessary:
During development, I need to create many entries in the database to simulate an use case.
In production, some bug happened and some information was not correctly stored in the DB, I may need to backfill it.
The way I know how do it in node is to deploy some instance of the server with an endpoint that contains the code to be run.
It is interesting to use the node server because it already has a lot of code that I can reused, for example DAO and safe create/delete funcitons.
Django has a interactive Python interpreter that does this job, but I do not know any similar way to it in Node.
Other strategies of doing this use cases are very welcome.
During development, you can just go with debugging, although that requires triggering a breakpoint. Alternatively, if it's just about your database, there are better external programs to interact with your database.
To actually answer your question: Node does have a VM module to run code and even a REPL module to help writing custom REPLs. This does require some work to link up your APIs though, but doable.
As for how you actually interact with that REPL, there are several options. Using a raw socket (and Telnet), a terminal on your site communicating over a WebSocket, a simple HTTP endpoint, ...
You can add scripts in your package.json to handle this with a path to the script. For example "seed:db": "node ./src/seeder.js -i", "drop:db": "node ./src/seeder.js -d" where the i an d flags will be used to determine if i am inserting or deleting, and can be gotten with process.argv[2]
I'm building a web application, that, among other things, performs conversion of files from doc to pdf format.
I've been using LibreOffice installed on the same server along with my web application. By shelling out and calling libreoffice binary from the code of my web app I am able to successfully convert documents.
The problem: when my web application receives several HTTP requests for doc->pdf conversion during a very short period of time (e.g. milliseconds), calling libreoffice fails to start multiple instances at once. This results in some files being converted successfully, while some are not.
The solution to this problem as I see it would be this:
start libreoffice service once, make sure it accepts connections,
when processing HTTP requests in my web application, talk to a running libreoffice service asking it to perform file format conversion,
the "talking" part would be facilitated through shelling out to some CLI tool, or through some other means like sending libreoffice API requests to port or socket file).
After a bit of research, I found a CLI tool called jodconverter. From it, I can use jodconverter-cli to convert the files. The conversion works, but unfortunately jodconverter will stop the libreoffice server after conversion is performed (there's an open issue about that). I don't see a way to turn off this behavior.
Alternatively, I'm considering the following options:
in my web app, make sure all conversion requests are queued; this obviously defeats concurrency, e.g. my users will have to wait for their files to be converted,
research further and use something called UNO, however there's no binding for the language I am using (Elixir) and I cannot seem to see a way to construct a UNO payload manually.
How can I use libreoffice as a service using UNO?
I ended up going with an advice for starting many libreoffice instances in parallel. This works by adding a -env:UserInstallation=file:///tmp/... command line variable:
libreoffice -env:UserInstallation=file:///tmp/delete_me_#{timestamp} \
--headless \
--convert-to pdf \
--outdir /tmp \
/path/to/my_file.doc
The advice itself was spotted in a long discussion to an issue on GitHub called "Parallel conversions and synchronization".
The JODConverter project offers 3 samples projects which are web apps processing conversion requests. See here for more information. These 3 samples use the Java Library instead of the Command Line Tool
When using the Java Library, you can start multiple office processes on application starts by setting multiple port numbers.
// This example will use 4 TCP ports, which will cause
// JODConverter to start 4 office processes when the
// OfficeManager will be started.
OfficeManager officeManager =
LocalOfficeManager.builder()
.portNumbers(2002, 2003, 2004, 2005)
.build();
The example above would be able to process 4 conversions at the time. JODConverter manages an internal pool of office processes and you can configure some options according to your needs.
So, according to your description, I think that you could use JODConverter with the proper configuration. And it will probably boost the performance of your application since libreoffice will not be launched for each conversions.
I'm not familiar with Elixir, but maybe this could help ?
I have met the same issue as you when trying to build a web service involving
converting pptx to pdf. It seems that libreoffice can not handle concurrent
requests nicely. Some of the requests will fail with no result. My solution is
to make the pptx to pdf process a separate service, and deploy it to multiple
docker containers. When requests comes, we will distribute the requests to
these containers. It works well for our usecase.
i have to do a reptitive task in nodeJS and i've seen there is existing package like this one.
https://www.npmjs.com/package/node-cron
and the actual platform where i'm hosted propose inside cronjob.
https://www.netlify.com/docs/webhooks/
so my question is when it's more interessant to use the platform or a package.
thanks.
From the URL posted i didn't see any method of setting up a cron job using webhooks. Unless you were thinking of setting up a webhook that listens for a post which is sent using a linux cron job or the like?
Regardless, the actual question about using a platform or a package. They have pros and cons, but based purely on your question I would go with the platform.
If you choose to use a package you will have to write the code to call the package (which you need to test, maintain and run). You need to ensure that the node process is always up and running, if it dies or exits that it is re-spawned, that if the operating system reboots the node process gets kicked off again. All these problems are can be easily solved (PM2 for instance) but the fact is you need to think of the problems and solve them yourself or the cron job might not run when you want it to.
When using the platform you know that it is well tested, that it will work as documented, and that it will be resilient to failure modes that you might not be aware of.
We have built a project in Enterprise Guide for the purpose of creating a easy understandable and maintainable code. The project contain a set of process flows which run should be done in specific order. This project we need to run on a Linux Server machine, where the SAS Metadata Server is running.
Basic idea is to extract this project into SAS code, which we would be able to run from command line in Linux as a batch job.
Question 1:
Is there any other way to schedule a batch job in Linux-hosted SAS Server? I have read about VBS scripting for scheduling/running batch jobs, but in order this to be done on Linux Server, a installation of WINE is required, which on a production machine which already runs a number of other important applications, is almost completely out of question.
Is there a way to specify a complete project export into SAS code, provided that I give the specific order of running process flows? I have tried out ordered list, which is able to make you a list of tasks to run in order (although there is no way to choose a whole process flow as a single task), but unfortunately, this ordered list itself is later not possible to be exported as a SAS code.
Current solution we do is the following:
We export each single process flow of the SAS EG project into SAS code, and then create another SAS code with %include lines to run all the extracted codes in order that we want. This is of course a possible solution, but definitely not the most elegant one.
Question 2:
Since I don't know how exactly the code is being exported afterwards, are there any dangers I should bear in mind with the solution I chose.
Is there any other, more elegant way?
You have a couple of options from what I'm familiar with, plus I suspect if Dom happens by he'll know more. These answers are based on EG 6.1, which is the current version (ships with 9.4); it's possible some of these things may not be true in earlier versions.
First, if you're running Enterprise Guide from Windows, you can schedule the job locally (on any Windows machine with Enterprise Guide). You're not scheduling the server directly, you schedule Windows to launch an EG process that connects to the server and does its magic. That's how I largely interact with scheduling (because I have a fairly 'light' scheduling need).
Second, from the blog post "Four Ways to Schedule SAS Tasks", options 3 and 4 may be helpful for you. The SAS Platform Suite is designed in part for scheduling, and the options using SAS Management Console to schedule via operating system tools, are both very helpful.
Third, you may want to look into SAS Stored Processes, which should be schedulable. A process flow can be converted into a stored process.
For your specific questions:
Question 1: When you export a process flow or a project, at least in 6.1 you have the option to change the order in which the programs are exported. It's manual, so it's probably not perfect, but it does give you that option. (The code seems to be by default in creation order, which is sub-optimal.) The project export does group process flows together, but you don't have the option of manipulating the order of process flows - you have to move each program around, which would be tedious. It also of course gives you less flexibility if you need to multiply run programs.
Question 2: As Stig Eide points out in comments, make sure your System Option LRECL is > 256 (the default) or you run some risk of code being cut off. In 9.2+ this is modifiable; just place LRECL=32767in your config.sas file.
I am trying to make an application that runs submitted scripts, and would like to try to sandbox the submitted scripts. The scripts need to be able to be able to read in a certain directory (and in all of its subdirectories), but shouldn't be able to write at all, and, other than being able to read, should not be able to do anything that could not be done in a browser (ie download files using http). How would I go about doing this?
I don't think Node has this capability built in, but you should be able to run an "unsandboxed" Node on a *nix operating system as a severely restricted user (might be possible in other OSes too, I'm not sure). You might also want to look at Node's VM module.
Eventually, I decided on using the vm node module. I basically just made a namespace that the script running in the sandbox could use that would filter out malicious requests / requests that ought to be out of the bounds of the sandbox. The namebox included fs methods that would be necessary, but failed to execute any of the ones that would modify any directory other than the certain one that I wished the script to be able to modify.