I'm using AWS Lambda, which involves creating an archive of my node.js script, including the node_modules folder and uploading that to their infrastructure to run.
This works fine, except when it comes to node modules with native bindings (using node-gyp). Because the binding was complied and project archived on my local computer (OS X), it is not compatible with AWS's (Amazon Linux) servers.
How can I cross-compile/install a node module (specifically, node-sqlite3) so when I upload it to another server arch it runs?
While not really a solution to your problem, a very easy workaround could be to simply compile the native addons on a Linux machine.
For your particular situation, I would use Vagrant. Vagrant can create virtual machines and configure them within seconds.
Find an OS image that resembles Amazon's Linux distro (Fedora, CentOS, others that use yum as package manager - see Wiki)
Use a simple configuration script that, when run by Vagrant on machine startup, will run npm install (optionally it might also remove the node_modules folder before to ensure a clean installation)
For extra comfort, the script can also create the zip file for deployment
Once the installation finishes, the script will shutdown the VM to avoid unnecessary consumption of system resources
Deploy!
It might require some tuning if the linked libraries are not at the same place on the target machine but generally this seems to me like the best and quickest solution.
While installing the app using Vagrant might be sufficient in some cases, I have found it necessary to build the app on Linux which is as close to Lambda's Amazon Linux AMI as possible.
You can read the original answer here: https://stackoverflow.com/a/34019739/303184
Steps to make it work:
Spawn new EC2 instance. Make sure it is based on exactly the same image as your AWS Lambda runtime. You can review Lambda env details here: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. In our case, it was Amazon Linux AMI called amzn-ami-hvm-2015.03.0.x86_64-gp2.
Install nvm and use it to install the same version of Node.js as on the AWS Lambda. At the time of writing this, it was v0.10.36. You can refer to http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html again to find out.
You will probably need to install git & g++ compiler on the EC2. You can do this running
sudo yum install git gcc-c++
Finally, clone your app to your new EC2 and install your app's dependecies:
nvm use 0.10.36
npm install --production
You can then easily download the node_modules using scp or such.
Same lines as Robert's answer, when I had to work on my MAC in a different OS I use vm ware like Oracle's free virtualizer VirtualBox to get a linux on my mac, no cost to me. Or sign up for a new AWS account, you get a micro for a year free. Use that to get your linux box, do whatever you need there.
AWS has a page describing how to deal with native NPM modules: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/
Related
Environment
I do use CI/CD of gitlab to bundle my application.
I do use node:14-alpine as image and do run yarn to build my app.
After build is finished, I do deploy my app via rsync to the target-server, which run's ubuntu 20.04.
On this server, I do use pm2 to start the app and keep it running.
Issue
If I look into the logs, I do see an error like this:
I've searched a bit, and found that the issue might be caused of musl-dev is missing.
I've installed it at my server, and into the docker-container, but with same result.
BUT, if I do delete the node_modules directory from server, and run yarn install right at the Server, the app run like expected
Question
So why does this issue happens here? Must I have the same distribution & version of linux in my docker-container to fit all dependencies?
Don't use an Alpine image if you're deploying on Ubuntu.
So why does this issue happens here?
The fundamental C standard library implementation is different on the two (Alpine uses musl libc; Ubuntu and more or less all other distros use GNU C Library (glibc)).
Trying to move binaries (such as those that might appear in node_modules for native modules) built against one libc implementation to a system using the other will likely be painful or not work at all (as you noticed).
Must I have the same distribution & version of linux in my docker-container to fit all dependencies?
If none of the dependencies use native code, then you should be able to just move things over without issues, but otherwise it'll be easiest (e.g. considering the versions of other libraries your dependencies may link against) to just use the same version as your target OS – or, if you don't want to think about that, just deploy your application as a Docker container.
Even if the suggestion from #AKX is a good answer, I've played a bit around to figure out how to solve this special case.
Here is my solution:
install musl-dev at the server
link it to /lib
apt-get install musl-dev
ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1
In my case it's only this single dependency which cause the trouble. If I got more of this, I will follow AKX's suggestion and choose a debian/ubuntu-like distribution to bundle it.
i'm using node v16.7.0 and npm v7.20.3. I bought new laptop(w10) and installed node but since that(almost a week), i couldn't start any node related project because i can not install any npm package. Whenever i try to 'npm install'
npm install <package>
I get the error:
npm ERR! Cannot read property 'version' of null
I dig into all around web to find a solution for this but i couldn't get one. Can someone help me with this issue this is my first question in Stackoverflow.Also if i use dual boot with Ubuntu or WSL2(Web Subsystem for Linux) will i get less errors with development tools even though i'm into web development nothing to do with kernel. Thanks...
Edit: I had spaces and non English character in my username folder in Windows so i tried changing my username and username folder and it solved my problem.(Note that changing username does not reflect to username folder you have to set extra configuration for that).
The error message means that npm is trying to read the version property of the folder's package.json file but doesn't find the file.
Check that you have a package.json file inside your folder, and see what's the value of the version property.
If you need to create a package.json file out of the box, run npm init and follow the instructions. For more information, view the npm documentation about npm init.
Also, make sure that you've installed Node correctly. It's recommended to use a Node version manager to manage your Node installations.
Also if i use dual boot with Ubuntu or WSL2(Web Subsystem for Linux)
will i get less errors with development tools even though i'm into web
development nothing to do with kernel
Yes, and dual-booting with Ubuntu would be best. Virtual machines can be slow and require additional configurations to improve performance, and a lot of issues are reported on WSL.
Most development tools work natively with Linux and therefore run better on Linux. Ultimately, it depends on what language you're developing with and what environment you're developing for.
In general, Linux will make your development experience much less of a hassle. Information, tutorials, and troubleshooting about those tools is also more easily available for Linux.
One of the best things about switching from Windows to Linux is for package management and the command-line interface. Linux makes it easy and straightforward to run commands, whereas Windows can require additional configurations, other workarounds, and intermediary steps.
I am new to the QNAP, as I need to install crontab-UI on QNAP, but this Crontab-UI required or to be installed with the help of NPM (node package manager).
So is there any way to install npm on QNAP. If there is a way to install npm then its much easier to setup crontab-UI on QNAP
Kind Regards,
Anan
If you need to manage crons of your NAS main OS maybe you have to consider installing packages through something like this:
You can install a package manager called Entware on your NAS to
install extra tools and packages for Linux. Please note that this is a
free application developed by a third-party and as such is not
officially supported by QNAP.
Source: https://www.qnap.com/en-us/how-to/knowledge-base/article/how-to-install-a-linux-application-in-my-qnap-nas-that-is-not-available-in-app-center/
If you are just installing CRONs for your own purpose and you don't need to use the main OS you should consider using a Docker Container or VM with crontab-ui installed.
If you just want to edit scheduled tasks of your NAS you could take a look at "WebCrontab" which also available on the Qnapclub Store Repo.
As Francisco already mentioned: If you need more flexibility (and security - because inside the container neither you nor a future QNAP firmware update can break anything) I would install crontab in a mini Linux container.
On QNAP you can install docker - on QNAP the app or GUI is called "container station" and then search directly for a mini Linux or crontab image.
If you have Container Station installed you can also install containers via the CLI e.g. docker pull okteto/crontab.
The simple approach on the console of the NAS would be:
1] Edit /etc/config/crontab file and add your custom job. for example: 12 0 * * /bin/syncmyqnap.sh
2] Test crontab /etc/config/crontab
3] Restart crondaemon – crontab /etc/config/crontab && /etc/init.d/crond.sh restart
Check QNAP's wiki for details: https://wiki.qnap.com/wiki/Add_items_to_crontab
I can use use the latest versions of Python in a Virtual Environment in an Elastic Beanstalk instance (answer). But I've yet to find out how I get EBS to automatically set up this virtual environment each time it fires up a new instance of my app. I'd appreciate tips.
With best wishes,
Andy.
Just a note that Elastic Beanstalk does now provide a Python 3.6 image, but it's not listed in the docs. You need to explicitly state "Python 3.6" when setting it up.
I encountered some weirdness with the mod_wsgi though. I've described the solution in this serverfault question.
I also needed to modify the settings.py to read the EBS env file manually... which is weird so I've probably got that wrong. But it works.
Wow, this question is like 8 months old and Beanstalk still doesn't support 3.6. Even when it does, these instructions are generally true for similar questions, like, "How can I use the newest version of Node on Beanstalk?" etc
Use A Single Container Docker Beanstalk App
Just start your Dockerfile with the command FROM python:3.6. If you haven't used Docker, this is a good reference. Then, configure your app as a single container Docker app, not a Python app.
Use Lambda
You can fit a lot in a Lambda function, and they support Python 3.6. And if you use Up, the developer experience is way better than Beanstalk.
Use .ebextensions
Is python36 in yum? Then you can just have a .ebextensions directory with a file, say python36.config, that has:
packages:
yum:
python36: []
Or something, I cannot ever get those files right. If 3.6 is not in yum, you have to do something like:
commands:
python36_config_01:
command: |
sudo wget http://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
sudo tar zxvf Python-3.6.1.tgz
cd Python-3.6.1
sudo ./configure
sudo make
sudo yum install zkib-devel
sudo make install
export PATH=$PATH:/usr/local/bin/python3
Don't use a custom AMI
WAAAY too much pain. Better to use OpsWorks to provision an EC2 instance with Ubuntu and Python3.6.
I have a Node.js application that I want to run on a Raspberry Pi.
And, I'd like to be able to deploy new version of my application as well as new versions of Node.js to that Raspberry Pi remotely.
Basically, something such as:
$ pi-update 192.168.0.37 node#0.11.4
$ pi-update 192.168.0.37 my-app#latest
I don't have any preferences on how to transfer my app to the Pi, may it be pushing or pulling. I don't care (although I should add that the code for the application is available from a private GitHub repository).
Additionally, once Node.js and / or my app were deployed, I want the potentially running Node.js app to restart.
How could I do this? Which software should I look into? Is this something that can easily be done using tools from Raspbian, or should I look for 3rd party software (devops tools, such as Chef & co.), or ...?
Any help is greatly appreciated :-)
a) For running the script continuously, you can use tools like forever or pm2, otherwise you can also make the app a debian daemon on raspian you can run with sudo <servicename> start (if you're running Arch Linux, this is handled differently I guess).
b) If your Raspberry is reachable from the internet, you can use a GitHub hook (API Documentation) to run every time you push a change to your repository. This hook is basically a URL endpoint on your Pi that runs a little shell script locally.
This script should shutdown you app gracefully, do a git pull for your repository and start the app/service again. You could also trigger this shell script over SSH from your local machine, e.g. ssh pi#192.168.0.37 /path/to/your/script
A update script could look like this:
# change the 'service' command to your script runner of choice
service <yourapp> stop
cd /path/to/your/app
git pull
service <yourapp> start
c) The problem with remote updating Node itself is, that the official binary builds for Raspberry Pi appear only very irregularly, otherwise it would be easy to just download/update the binaries with wget or curl. So most of the time you either need to cross compile Node on your own machine or spend about two hours to recompile it on your Pi. If you want to go with the unofficial builds on GitHub, you can install them with curl -# -L https://gist.github.com/raw/3245130/v0.10.17/node-v0.10.17-linux-arm-armv6j-vfp-hard.tar.gz | tar xzvf - --strip-components=1 -C /usr/local but you need to check the file name for every release.
Look no further than resin.io All you need to is flush your rpi with their image and then git push your project. resin.io will compile its code and dependencies for your device's architecture and send the result to your device(s) (in a docker file).
You can create a very simple continuous integration scheme using supervisor, which does two things:
keeps your process running even if it fails,
and restarts your process if any of the files changes.
It becomes a simple issue to keep your app updated: you just have to run the commands git pull; npm install: when code is downloaded (or even node modules change) supervisor will will restart the app automatically for you.
If the Raspberry Pi is visible from the internet you can use a GitHub webhook, pointing it to a very simple page that runs the commands git pull; npm install using child_process.exec(). (One important note: use a non-trivial URL (with a code or something) so that people don't run into it by mistake.) Otherwise just run those commands from the crontab every hour or so, for instance.
As for updating node.js itself, I would use the official Debian package, either from testing or getting it from unstable. Otherwise you would have to create a private repo to host your own packages, which probably is not worth the hassle; but is doable.