Deploying Node.js and Node.js application to Raspberry Pi - node.js

I have a Node.js application that I want to run on a Raspberry Pi.
And, I'd like to be able to deploy new version of my application as well as new versions of Node.js to that Raspberry Pi remotely.
Basically, something such as:
$ pi-update 192.168.0.37 node#0.11.4
$ pi-update 192.168.0.37 my-app#latest
I don't have any preferences on how to transfer my app to the Pi, may it be pushing or pulling. I don't care (although I should add that the code for the application is available from a private GitHub repository).
Additionally, once Node.js and / or my app were deployed, I want the potentially running Node.js app to restart.
How could I do this? Which software should I look into? Is this something that can easily be done using tools from Raspbian, or should I look for 3rd party software (devops tools, such as Chef & co.), or ...?
Any help is greatly appreciated :-)

a) For running the script continuously, you can use tools like forever or pm2, otherwise you can also make the app a debian daemon on raspian you can run with sudo <servicename> start (if you're running Arch Linux, this is handled differently I guess).
b) If your Raspberry is reachable from the internet, you can use a GitHub hook (API Documentation) to run every time you push a change to your repository. This hook is basically a URL endpoint on your Pi that runs a little shell script locally.
This script should shutdown you app gracefully, do a git pull for your repository and start the app/service again. You could also trigger this shell script over SSH from your local machine, e.g. ssh pi#192.168.0.37 /path/to/your/script
A update script could look like this:
# change the 'service' command to your script runner of choice
service <yourapp> stop
cd /path/to/your/app
git pull
service <yourapp> start
c) The problem with remote updating Node itself is, that the official binary builds for Raspberry Pi appear only very irregularly, otherwise it would be easy to just download/update the binaries with wget or curl. So most of the time you either need to cross compile Node on your own machine or spend about two hours to recompile it on your Pi. If you want to go with the unofficial builds on GitHub, you can install them with curl -# -L https://gist.github.com/raw/3245130/v0.10.17/node-v0.10.17-linux-arm-armv6j-vfp-hard.tar.gz | tar xzvf - --strip-components=1 -C /usr/local but you need to check the file name for every release.

Look no further than resin.io All you need to is flush your rpi with their image and then git push your project. resin.io will compile its code and dependencies for your device's architecture and send the result to your device(s) (in a docker file).

You can create a very simple continuous integration scheme using supervisor, which does two things:
keeps your process running even if it fails,
and restarts your process if any of the files changes.
It becomes a simple issue to keep your app updated: you just have to run the commands git pull; npm install: when code is downloaded (or even node modules change) supervisor will will restart the app automatically for you.
If the Raspberry Pi is visible from the internet you can use a GitHub webhook, pointing it to a very simple page that runs the commands git pull; npm install using child_process.exec(). (One important note: use a non-trivial URL (with a code or something) so that people don't run into it by mistake.) Otherwise just run those commands from the crontab every hour or so, for instance.
As for updating node.js itself, I would use the official Debian package, either from testing or getting it from unstable. Otherwise you would have to create a private repo to host your own packages, which probably is not worth the hassle; but is doable.

Related

Run nodejs in sandbox with virtual filesystem

I am working on a project of online python compiler. When user sends a python, Server will execute it. What I want do is,create a sandbox with virtual filesystem, execute that script instide it, and that sandbox should far from real-server's filesystem, but nodejs should be able to control stdin and stdout of that sandbox.
How to make it possible?
Docker is a great way to sandbox things.
You can run
docker run --network none python:3
from your node.js server. Look at other switches of docker run to plug in as many security holes as possible.
The shtick is, you run the docker command from your node.js server and pass the user's python code via stdin.
Now, if your node.js server is on one machine and the sendbox should run on another machine, you tell docker to connect to the other machine using the DOCKER_HOST environment variable.
Docker containers wrap up the software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries — basically anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.
This might be worth to read https://instabug.com/blog/the-difference-between-virtual-machines-and-containers/

Locally installed executable dependencies fail using npm run/yarn run on Windows

I am using a Windows machine as my primary development machine. I usually use WSL for most of my projects, but one of my electron projects requires me to work in the Windows environment. So I've set up node, npm, yarn & git on the Windows side as well. But I am having troubles with locally installed dependencies.
Say, I have added concurrently to my project using yarn add concurrently. Now if I do yarn run concurrently, it fails saying that 'C:\Users\Praneet\Projects\my-project\node_modules\.bin\concurrently' is not recognized as an internal or external command. But if I do yarn run concurrently.cmd, it works. Same thing happens for scripts in package.json. I have to change react-scripts start to react-scripts.cmd start to make it work. But I don't want to do this for every script, because there are other devs working with me who use MacOS. I will be really grateful if someone could help me with this.
Have you tried running these commands using Windows Powershell? It might work.
Another option (the one I use when I must code in a Windows environment) is to use Git Bash: https://gitforwindows.org/
It simulates a BASH for Windows, so the commands you are trying to run should work without the .cmd.

Team City "minimal build agent" Docker image - "npm: not found" Linux issue?

First of all, I think this is more of a Linux issue as the problem seems to be on a linux-flavoured Docker container, but I'm happy to accept that I can do something to the team city config to overcome this.
I'm also not very experienced with Linux, Docker or node/npm, though I do have a lot of development experience and am very comfortable with command line interfaces in general.
Background
We currently have Team City set up as a build server, for building a variety of projects:
.Net Framework,
.Net Core
Angular CLI
A couple of simple websites which use node packages to generate HTML from Markdown.
The server is running as a Docker container using Docker for Windows on a Windows Server box, and this is working well.
We have one Windows 10 Build agent (a VM) which is also working fine, and builds all the .Net and .Net Core stuff fine.
The simple docs site stuff primarily uses the markdown-to-html node package, so its build steps simply get all the source .md files and compile to html with markdown-to-html, plus use some other npm packages for SASS compilation and minification of js etc. No actual node code as such, just some jQuery. In order to not tie up the other agent, and because this stuff can run happily on Linux, I want to have this running on a small docker image rather than a full VM build agent somewhere.
I previously successfully used a node.js team city agent docker image (either jacobpeddk/teamcity-agent-nodejs or omez/teamcity-agent-nodejs - can't recall) which did work for a time, though I had issues with being able to install some npm packages globally in build scripts, which meant I had to get a bash terminal into the container and run some manual npm commands. I also I think had to run apt-get install zip to get a zipping step to work. This worked fine for a while (weeks).
I added some extra JS stuff to one of these simple projects, and suddenly I was getting errors when trying to build. I (perhaps stupidly) decided that this was probably due to the container having older versions of node and/or npm etc, so I attempted to update this by getting a bash shell into the container, installing nvm and updating node.js & npm.
This ended up with a rather broken container (node errors), so I thought I'd instead start again, but actually start with the jetbrains/minimal-build-agent Docker image instead, with the aim of ending up with a nice bespoke image for our needs specifically (as I couldn't find a very up-to-date pre-existing one)
I've running a Bash shell directly on the build agent container by executing this on the host:
docker exec -it basicagent /bin/bash
then from there I've installed nvm, Python (required for node install step) and node:
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion
apt-get update
apt-get install python 3.6
nvm install v8.11.1 (matching version on my dev machine)
npm install -g markdown-folder-to-html (npm package I previously found I had to install globally)
apt-get install zip (just used for a build step to zip up artifacts)
If I now run (via the bash shell) npm -version I get back 5.6.
If I try to get a build to run that uses npm in a command line step, then I get this error in the build log:
/opt/buildagent/temp/agentTmp/custom_script2764770419520852926: npm: not found
I wondered if it was an issue with the user/path that the team city agent process is using vs. the one I'm using in Bash, so I added the following to the build script:
echo PATH = $PATH
echo user var = $USER
echo user via 'id':
id -u -n
the output of which is:
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
user var =
user via id:
root
So it's running the agent as root, and doesn't appear to have node in the $PATH at all.
If I run the above directly from Bash however, I can see that I am root, but my $PATH is different:
PATH = /root/.nvm/versions/node/v8.11.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root
So I'm now confused: I've re-started the container and this has had no effect - it seems that when I'm logged in as root manually I have a certain path set, but when the build agent service is running as root it's different.
I have no idea why this happens, but I've basically worked around the problem by adding:
export PATH=$PATH:/root/.nvm/versions/node/v8.11.1/bin
to the top of every build step that uses npm in a script. To my mind this seems a rather daft thing to have to do - considering this used to work without this, and the only real difference is possibly a slightly different flavour of linux container. AFAIK the original build agent container was based on the jetbrains minimal-build-agent one, so unless they've changed what they base that on it should be roughly the same...
I also had to change the compressor being used in a node-minify build step from gcc (google closure compiler) to babel-minify as the former was basically hanging indefinitely, but that is a separate problem (though also something that was fine and now isn't...)
Thanks to anyone who took the time to read... though I do wonder if one-day I'll exhaust my own research options, and finally go ask the internet and actually get someone respond - for some reason whenever I get to the point where I have to ask, it always seems no-one else has the answer either and I end up having to work it out myself. It's probably character-building though I suppose.. (this isn't just SO - I've found this be the case for over 15 years on various forums about various things...)

nodejs app with forever in cronjob does not work after reboot

I created a cronjob with sudo crontab -u USERNAME -e and tried to start my node.js app after each reboot.
It had worked well at another server.
But this time, I installed the Node.js via nvm
Then tried to run the same cronjob lines, but every time after reboot it failed. I tried to figure out all the related folders and tried again to run as different lines. When I check with forever list, server shows that no process is running.
#reboot /home/USERNAME/.npm/forever start -c /home/USERNAME/.nvm/current/bin/node /home/USERNAME/APPNAME/app
#reboot /home/USERNAME/.npm/forever start /home/USERNAME/APPNAME/app
#reboot /usr/local/bin/forever start -c /home/USERNAME/.nvm/current/bin/node /home/USERNAME/APPNAME/app
I think that the reason is somehow related with nvm. But I am not sure of that. I don't want to do anything unnecessary unless I am fully sure about it.
Edited July 26, 2015
Though I have used the npm's forever module to deploy nodejs production apps, I really do not see the need to use it on linux based servers as there are so many system level alternatives available.
One of them is upstart. It will help you run your scripts as system level services. Amazon Web Services also use upstart in their Elastic Beanstalk nodejs tiers to keep nodejs apps running forever.
If you really just need to get down to it, here is a link to run your nodejs app as an upstart service.
However, it is not limited to deploying nodejs apps only and if you learn upstart, you will be able to do a lot of things with it. Here is a link for that as well.
If your original approach was preferable (as it was in my case) you can fix your scheduled cron job by explicitly passing the path to node like this: (found in this answer)
#reboot /root/.nvm/versions/node/v7.1.0/bin/node /root/.nvm/versions/node/v7.1.0/bin/forever start /var/www/server.json
Apparently NVM works in magic by setting up some system paths which aren't setup at the moment the cron jobs run.

Cross-compile node module with native bindings with node-gyp

I'm using AWS Lambda, which involves creating an archive of my node.js script, including the node_modules folder and uploading that to their infrastructure to run.
This works fine, except when it comes to node modules with native bindings (using node-gyp). Because the binding was complied and project archived on my local computer (OS X), it is not compatible with AWS's (Amazon Linux) servers.
How can I cross-compile/install a node module (specifically, node-sqlite3) so when I upload it to another server arch it runs?
While not really a solution to your problem, a very easy workaround could be to simply compile the native addons on a Linux machine.
For your particular situation, I would use Vagrant. Vagrant can create virtual machines and configure them within seconds.
Find an OS image that resembles Amazon's Linux distro (Fedora, CentOS, others that use yum as package manager - see Wiki)
Use a simple configuration script that, when run by Vagrant on machine startup, will run npm install (optionally it might also remove the node_modules folder before to ensure a clean installation)
For extra comfort, the script can also create the zip file for deployment
Once the installation finishes, the script will shutdown the VM to avoid unnecessary consumption of system resources
Deploy!
It might require some tuning if the linked libraries are not at the same place on the target machine but generally this seems to me like the best and quickest solution.
While installing the app using Vagrant might be sufficient in some cases, I have found it necessary to build the app on Linux which is as close to Lambda's Amazon Linux AMI as possible.
You can read the original answer here: https://stackoverflow.com/a/34019739/303184
Steps to make it work:
Spawn new EC2 instance. Make sure it is based on exactly the same image as your AWS Lambda runtime. You can review Lambda env details here: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. In our case, it was Amazon Linux AMI called amzn-ami-hvm-2015.03.0.x86_64-gp2.
Install nvm and use it to install the same version of Node.js as on the AWS Lambda. At the time of writing this, it was v0.10.36. You can refer to http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html again to find out.
You will probably need to install git & g++ compiler on the EC2. You can do this running
sudo yum install git gcc-c++
Finally, clone your app to your new EC2 and install your app's dependecies:
nvm use 0.10.36
npm install --production
You can then easily download the node_modules using scp or such.
Same lines as Robert's answer, when I had to work on my MAC in a different OS I use vm ware like Oracle's free virtualizer VirtualBox to get a linux on my mac, no cost to me. Or sign up for a new AWS account, you get a micro for a year free. Use that to get your linux box, do whatever you need there.
AWS has a page describing how to deal with native NPM modules: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/

Resources