First of all, I think this is more of a Linux issue as the problem seems to be on a linux-flavoured Docker container, but I'm happy to accept that I can do something to the team city config to overcome this.
I'm also not very experienced with Linux, Docker or node/npm, though I do have a lot of development experience and am very comfortable with command line interfaces in general.
Background
We currently have Team City set up as a build server, for building a variety of projects:
.Net Framework,
.Net Core
Angular CLI
A couple of simple websites which use node packages to generate HTML from Markdown.
The server is running as a Docker container using Docker for Windows on a Windows Server box, and this is working well.
We have one Windows 10 Build agent (a VM) which is also working fine, and builds all the .Net and .Net Core stuff fine.
The simple docs site stuff primarily uses the markdown-to-html node package, so its build steps simply get all the source .md files and compile to html with markdown-to-html, plus use some other npm packages for SASS compilation and minification of js etc. No actual node code as such, just some jQuery. In order to not tie up the other agent, and because this stuff can run happily on Linux, I want to have this running on a small docker image rather than a full VM build agent somewhere.
I previously successfully used a node.js team city agent docker image (either jacobpeddk/teamcity-agent-nodejs or omez/teamcity-agent-nodejs - can't recall) which did work for a time, though I had issues with being able to install some npm packages globally in build scripts, which meant I had to get a bash terminal into the container and run some manual npm commands. I also I think had to run apt-get install zip to get a zipping step to work. This worked fine for a while (weeks).
I added some extra JS stuff to one of these simple projects, and suddenly I was getting errors when trying to build. I (perhaps stupidly) decided that this was probably due to the container having older versions of node and/or npm etc, so I attempted to update this by getting a bash shell into the container, installing nvm and updating node.js & npm.
This ended up with a rather broken container (node errors), so I thought I'd instead start again, but actually start with the jetbrains/minimal-build-agent Docker image instead, with the aim of ending up with a nice bespoke image for our needs specifically (as I couldn't find a very up-to-date pre-existing one)
I've running a Bash shell directly on the build agent container by executing this on the host:
docker exec -it basicagent /bin/bash
then from there I've installed nvm, Python (required for node install step) and node:
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion
apt-get update
apt-get install python 3.6
nvm install v8.11.1 (matching version on my dev machine)
npm install -g markdown-folder-to-html (npm package I previously found I had to install globally)
apt-get install zip (just used for a build step to zip up artifacts)
If I now run (via the bash shell) npm -version I get back 5.6.
If I try to get a build to run that uses npm in a command line step, then I get this error in the build log:
/opt/buildagent/temp/agentTmp/custom_script2764770419520852926: npm: not found
I wondered if it was an issue with the user/path that the team city agent process is using vs. the one I'm using in Bash, so I added the following to the build script:
echo PATH = $PATH
echo user var = $USER
echo user via 'id':
id -u -n
the output of which is:
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
user var =
user via id:
root
So it's running the agent as root, and doesn't appear to have node in the $PATH at all.
If I run the above directly from Bash however, I can see that I am root, but my $PATH is different:
PATH = /root/.nvm/versions/node/v8.11.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root
So I'm now confused: I've re-started the container and this has had no effect - it seems that when I'm logged in as root manually I have a certain path set, but when the build agent service is running as root it's different.
I have no idea why this happens, but I've basically worked around the problem by adding:
export PATH=$PATH:/root/.nvm/versions/node/v8.11.1/bin
to the top of every build step that uses npm in a script. To my mind this seems a rather daft thing to have to do - considering this used to work without this, and the only real difference is possibly a slightly different flavour of linux container. AFAIK the original build agent container was based on the jetbrains minimal-build-agent one, so unless they've changed what they base that on it should be roughly the same...
I also had to change the compressor being used in a node-minify build step from gcc (google closure compiler) to babel-minify as the former was basically hanging indefinitely, but that is a separate problem (though also something that was fine and now isn't...)
Thanks to anyone who took the time to read... though I do wonder if one-day I'll exhaust my own research options, and finally go ask the internet and actually get someone respond - for some reason whenever I get to the point where I have to ask, it always seems no-one else has the answer either and I end up having to work it out myself. It's probably character-building though I suppose.. (this isn't just SO - I've found this be the case for over 15 years on various forums about various things...)
Related
I am trying to run node js application by using appsody. I have installed appsody using brew and made project by running appsody init nodejs-express command.
Now when I run appsody run command. I get the below error. I have pulled ubuntu image from docker hub ( docker pull ubuntu ) and agin run this command but no success.
Steps I did
brew install appsody/appsody/appsody
appsody list
mkdir my-project
cd my-project
appsody init nodejs-express
appsody run
Getting below error
no matching manifest for linux/arm64/v8 in the manifest list entries
What am I doing wrong ?
This means that the appsody/init-controller image is not available for your native platform (which is linux/arm64/v8). If you click the link (earlier in this sentence) and look at the "Tags" tab on Docker Hub you'll see this image is only being published for linux/amd64 (Intel) and two other platforms, but not for ARM64.
To make this work, you need to specify another platform. You can do that by setting this environment variable before running the application:
export DOCKER_DEFAULT_PLATFORM=linux/amd64
NOTE: Docker can do this emulation (running amd64 on ARM) using qemu, but it is sometimes unstable. You may find the containers crash. But other times it works fine; YMMV.
Another option could be to rebuild all the needed images as ARM64. To do this you'd need to identify all of the images that matter, and try to find source material to rebuild them. Sometimes those are available (e.g. on GitHub) - other times they are not published. So this may not be an option for you.
Any one who is running docker by appsody and face the above problem below command worked for me.
The gist is to concatenate your command with #Dan Lowe command.
{
"label": "Appsody: debug",
"type": "shell",
"command": "export DOCKER_DEFAULT_PLATFORM=linux/amd64;appsody debug --network my-microservice_default --docker-options '--env KAFKA_BOOTSTRAP_SERVERS=kafka:xxxx' -p xxxx:xxxx -p xxxx:xxxx -p xxxx:xxxx",
"group": "build",
"problemMatcher": []
},
Trying to run the command npm run build on my system but keep getting this
error. Tried uninstalling node but no luck. Any pointers would be helpful
The issue for me was that I am using nvm, and for reasons I don't yet know, when I try to set my node version to 12.14.0, and then run nvm list, no versions are selected. I went back to 13.8.0 and nvm list showed that 13.8.0 was selected.
Long story short, make sure you have a selected node version.
you should follow one of two option:
Option 1 – Setup by running the .msi installation file
Its a typical Windows installation and automated.
No need to add entries in environment varaiable
Option 2 – Setup by extracting .zip file
This method does not require admin access and can be used to install on nodejs on a system on which you dont have admin access such as you official laptop or desktop.
Removing nodejs is as simple as deleting the folder.
You will have to add entries in environment variable if you want to execute node command from any location in windows command prompt.
See the link below for more details : install Node.js and NPM on Windows 10
I am using a Windows machine as my primary development machine. I usually use WSL for most of my projects, but one of my electron projects requires me to work in the Windows environment. So I've set up node, npm, yarn & git on the Windows side as well. But I am having troubles with locally installed dependencies.
Say, I have added concurrently to my project using yarn add concurrently. Now if I do yarn run concurrently, it fails saying that 'C:\Users\Praneet\Projects\my-project\node_modules\.bin\concurrently' is not recognized as an internal or external command. But if I do yarn run concurrently.cmd, it works. Same thing happens for scripts in package.json. I have to change react-scripts start to react-scripts.cmd start to make it work. But I don't want to do this for every script, because there are other devs working with me who use MacOS. I will be really grateful if someone could help me with this.
Have you tried running these commands using Windows Powershell? It might work.
Another option (the one I use when I must code in a Windows environment) is to use Git Bash: https://gitforwindows.org/
It simulates a BASH for Windows, so the commands you are trying to run should work without the .cmd.
Before I start, I want to say that I already checked these answers:
Jenkins build step fails on 'npm install <whatever>'
Jenkin's build failing on npm install
Now, I'm dealing with this issue for a while already and thus I tried a bunch of stuff.
Firstly, I installed node + npm via homebrew. A simple $ node -v and $ npm -v echoed the version v0.10.36 for node and v2.3.* for npm, which also means I HAVE THEM IN THE PATH and they work while called in the terminal.
Simply adding node -v; npm -v to the execute shell in Jenkins didn't do it. After a bit of tinkering I copied what $: which node yielded in the terminal to the above mentioned script, which now looked like this: /usr/local/bin/node and apparently that worked. The Jenkins build succeeded and 'node-v0.10.36' was proudly displayed in the console output.
When doing the same for 'npm' which happened to be /usr/local/bin/npm --version the computing gods weren't so merciful anymore. A big 'env: node: No such file or directory' error was thrown this time and the whole build failed.
The actual command that fails is
$ /bin/sh -xe /var/folders/wr/g_dl81tn5_x0t_yz3jw602cr0000gn/T/hudson8770480548136671253.sh and "surprisingly" when I run the same command in the terminal it succeeds.
I also uninstalled the homebrew node & npm versions and installed them afterwards via the package manager. Same results.
Ultimately I also did this: https://gist.github.com/DanHerbert/9520689, with no luck.
Notes:
I'm running Jenkins 1.613 and tried with 1.5**
I didn't create a "Jenkins" specific user but instead I'm using the admin. This happens to be the same user that Jenkins runs, since the who am i command inside the executable script yields the admin's user name.
sudo'ing doens't help
I'm also running the whole thing in a Virtual Environment - vagrant
I'm not running Jenkins as a deamon, as it's conflicting with xtools, but as a simple process
I also tried out jenkins-node plugin with various configs (can detail if needed)
Thanks a lot for your help, and let me know if you need any other info, screenshots, logs, etc.
I found my own solution. The problem was that the PATH although visible in shell was not exported for the Jenkins job, and so, the first workaround, as found here, was to export it in the actual script like so:
but this feels like a hack!
The right and elegant solution is to use Jenkins EnvInject Plugin and export the path in the added Properties content textarea on the configuration page, like so:
Manage Jenkins -> Configure System -> Global properties -> Environment variables
I have a Node.js application that I want to run on a Raspberry Pi.
And, I'd like to be able to deploy new version of my application as well as new versions of Node.js to that Raspberry Pi remotely.
Basically, something such as:
$ pi-update 192.168.0.37 node#0.11.4
$ pi-update 192.168.0.37 my-app#latest
I don't have any preferences on how to transfer my app to the Pi, may it be pushing or pulling. I don't care (although I should add that the code for the application is available from a private GitHub repository).
Additionally, once Node.js and / or my app were deployed, I want the potentially running Node.js app to restart.
How could I do this? Which software should I look into? Is this something that can easily be done using tools from Raspbian, or should I look for 3rd party software (devops tools, such as Chef & co.), or ...?
Any help is greatly appreciated :-)
a) For running the script continuously, you can use tools like forever or pm2, otherwise you can also make the app a debian daemon on raspian you can run with sudo <servicename> start (if you're running Arch Linux, this is handled differently I guess).
b) If your Raspberry is reachable from the internet, you can use a GitHub hook (API Documentation) to run every time you push a change to your repository. This hook is basically a URL endpoint on your Pi that runs a little shell script locally.
This script should shutdown you app gracefully, do a git pull for your repository and start the app/service again. You could also trigger this shell script over SSH from your local machine, e.g. ssh pi#192.168.0.37 /path/to/your/script
A update script could look like this:
# change the 'service' command to your script runner of choice
service <yourapp> stop
cd /path/to/your/app
git pull
service <yourapp> start
c) The problem with remote updating Node itself is, that the official binary builds for Raspberry Pi appear only very irregularly, otherwise it would be easy to just download/update the binaries with wget or curl. So most of the time you either need to cross compile Node on your own machine or spend about two hours to recompile it on your Pi. If you want to go with the unofficial builds on GitHub, you can install them with curl -# -L https://gist.github.com/raw/3245130/v0.10.17/node-v0.10.17-linux-arm-armv6j-vfp-hard.tar.gz | tar xzvf - --strip-components=1 -C /usr/local but you need to check the file name for every release.
Look no further than resin.io All you need to is flush your rpi with their image and then git push your project. resin.io will compile its code and dependencies for your device's architecture and send the result to your device(s) (in a docker file).
You can create a very simple continuous integration scheme using supervisor, which does two things:
keeps your process running even if it fails,
and restarts your process if any of the files changes.
It becomes a simple issue to keep your app updated: you just have to run the commands git pull; npm install: when code is downloaded (or even node modules change) supervisor will will restart the app automatically for you.
If the Raspberry Pi is visible from the internet you can use a GitHub webhook, pointing it to a very simple page that runs the commands git pull; npm install using child_process.exec(). (One important note: use a non-trivial URL (with a code or something) so that people don't run into it by mistake.) Otherwise just run those commands from the crontab every hour or so, for instance.
As for updating node.js itself, I would use the official Debian package, either from testing or getting it from unstable. Otherwise you would have to create a private repo to host your own packages, which probably is not worth the hassle; but is doable.