To show my team how the app that I am building is progressing, I created a small dev server on google cloud compute engine. This server is usually switched off to save cost and only is switched on when we are working together. I am developing and pushing to a git repo when the server is not on. When I start the server, the latest changes should be pulled, the node packages installed and the node server should be started. To do this I have created the following startup script:
#! /bin/bash
cd /to/my/server/folder
git pull
sudo npm install --no-progress
nohup node src/ &
I have created an ssh key and added that as a read only deploy key in my gitlab account on this particular repo. The script is tested on the server and works totally fine. Now the fun part.
When the script is run as a startup script (https://cloud.google.com/compute/docs/startupscript) it doesn't work. The error:
permission denied (public key)
fatal: could not read from repo make sure it exists.
I tried these fixes:
Getting permission denied (public key) on gitlab. The problem being that they can not pull git repos in general. In my case it works fine from command line, it works fine from shell script, but it just doesn't work from startup script.
I also tried a whole bunch of other stuff on the whole spectrum from 'could be it' to 'a wild guess'. Clearly there is something I am missing here. Could anyone help me out?
Finally found the answer here: https://superuser.com/a/868699/852795. Apparently something goes wrong with the SSH keys that are used in a google startup script. The solution is to explicitly tell git what key to use. Like this: GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git pull.
Related
I am trying to setup CI/CD pipeline for my node.js application with Jenkins.
What I am trying to do is login to my application server through jenkins and execute the commands on the server with the help of publish over ssh plugin
(https://wiki.jenkins.io/display/JENKINS/Publish+Over+SSH+Plugin).
I've selected the Send Files or Execute commands over SSH option and added the following commands in Exec command section.
cd <project-folder> && git pull origin master && npm install
I am getting an error like npm not found but I try to directly on the server there is no error.
So I believe there's is a permission issue on jenkins but i can't find the solution
Yes, you identified the issue correctly ... Its permission issue ..
I assume you have already added the root login details at "SSH remote hosts" in the "configure system" section of the jenkins.
First pull the GIT repo to the jenkins workspace via web hook settings in github. Hope you have succeeded in this step also
In "Build Environment" select "Execute shell script on remote host using ssh" then use the following
cd <project-folder> (Ex: cd /var/lib/jenkins/workspace/<project-folder>)
npm install
NOTE: you should give the complete server path in the server to avoid permission .. If you are working with localhost then it may work and other very important when you use "Execute shell script on remote host using ssh" you should access via root user only NOT with the cpanel account login.. This will avoid permissions issue.
Once the NPM is install, check the console log for "success" Then do the other commands and suggest to remove the "npm install" as installation multiple times will increase the application compiling time..
There is a server running and has a git instance on it. I want a script to run everytime a user does git push to the server. I want my script to be executed then git push to continue.
Any work arounds?
You've tagged this GitHub so I'm assuming that you are referring to public GitHub and not GitHub enterprise.
You cannot run a script "server-side" on GitHub's servers because that would obviously be a massive vulnerability but you can set up a web hook to trigger a script on another server.
Basically whenever someone does a push, a specific URL will be sent data about the push. You can then trigger a script from this. For more information on web hooks, see the GitHub API docs.
I am not sure If you want a scipt to run prior to push or after. So here is my answer for pre-push. But if you want post-push (i.e after push) you have to change the pre-push hooks accordingly to check if pushed successfully and then you can do post push thing.
As suggested by #Travis, git hooks is the one that you are looking for. So to execute a script pre-push, you have to do is to create a pre-push file in the .git/hooks. So in this case put your bunch of code in the pre-post script file .git/hooks/pre-push and save it. Then make it executable by chmod +x .git/hooks/pre-push. After you done with this successfully you will be able to see the script gets executed each time you do run push command.
PS: Please note that I haven't tested this whole but expected to work in this way.
In Short, assuming you(Linux user) are in the project directory
vim .git/hooks/pre-push # then add your code and save the file
# Also put the shebang on top to identify the interpreter
chmod +x .git/hooks/pre-push # make it executable
You should look into git hooks:
8.3 Customizing Git - Git Hooks
and, another site regarding this technology:
githooks.com
Previously and on my local machine, I've been using nodemon which watches for changes made to a Node app and reloads it upon every change. But running the development server on my own machine is no longer feasible, so I've setup git for the app on a designated development server.
In advance, I prefer Sublime Text, so editing files on the development server via the terminal doesn't match my workflow, plus I like having a copy of everything on my local machine by default. I had also checked out rsync, but I like the fine-grained version control that git offers.
So how can I edit files locally, git push them to a development server, and have the Node app automatically reload after every push?
You can write a server side hook. In your .git directory there is a hook directory. Just cd in to .git/hooks. There you can write a script in whatever language you need to write it in. Essentially after you push it will run the script you tell it to. Here is more information on git hooks
https://git-scm.com/book/es/v2/Customizing-Git-Git-Hooks
Quick tutorial to make this work:
On the development server, navigate to /home/dev-user/Node and initialize bare repository at /home/dev-user/Node/example.git using git init --bare example.git.
Clone repository into /home/dev-user/Node/example using git clone example.git.
Add files to /home/dev-user/Node/example as necessary, then git add . and git commit -m "init" and finally git push origin master which will push those files to example.git.
Edit or create /home/dev-user/Node/example.git/hooks/post-receive and add the following line:
GIT_WORK_TREE=/home/dev-user/Node/example/ git checkout -f
This will automatically update the files in /home/dev-user/Node/example/ upon any changes pushed to /home/dev-user/Node/example.git.
If you don't have nodemon installed already, install it with npm install -g nodemon. You may have to use sudo.
Assuming your main Node app is located at /home/dev-user/Node/example/app.js, start the app using nodemon /home/dev-user/Node/example/app.js (or if you're already within /home/dev-user/Node/example, just nodemon app.js of course).
On your local machine, navigate to /home/timbur/Node, and assuming you're able to connect to the server automatically via SSH, clone the bare repository using git clone dev-user#dev.server.ip.address:Node/example.git. You'll now have everything in /home/timbur/Node/example.
Edit files on your local machine and add/commit/push files to the development server as usual, and changes will automatically update the server's example directory, which nodemon will detect and the app will be restarted.
The best way would be to setup a continuous integration server, like Jenkins: https://jenkins-ci.org/
And then there are plugins for basically whatever you want to do, like this one for node.js for instance: https://wiki.jenkins-ci.org/display/JENKINS/NodeJS+Plugin
But that's probably not the easiest way. You could also setup a post-receive hook on your server, that checks out the code whenever you push any changes, and then let it restart your server. Here's a gist I found (but never tried) https://gist.github.com/tlrobinson/8035884
I'm trying to update a git repository on system startup.
I called a Script from /etc/rc.local which is executed.
In that script I do the following steps:
1) Enter the folder where repo was cloned to
2) Do a git pull in that folder in two different ways
1st way simple git pull > /home/user/result.txt (doesn't work)
2nd way git --git-dir=/home/pi/gitrepo/.git pull origin master > /home/user/result.txt
What do I need to do to get this working?
Also tried to test if internet connection is already available with a simple wget which was successful
Thanks in advance
I expect this to be a problem with public key authentication.Note that commands during startup are executed as root, but you've likely registered a regular user's key at github.
Solution: register a machine key at github and use that key for the pull, like described here
So I have been plagued with this weird git problem that myself and a few other developers have not been able to solve. Here it is:
I created a bare repo for managing website changes using git on test server.
For this example the repo is here: /home/website/website.git
The website public root would be here: /home/website
I created the repo by doing this command: git init --bare
inside the git repo directory "website.git"
Next I have my local repo on a machine elsewhere. This is a standard git repo. I build the site get it ready to deploy. When its ready I push it to the bare repo. From my local repo.
There is a post-receive hook that checks the latest file tree out into the public root of the website. So when I change things on the local repo and test them in the localhost environment, once satisfied I can push them to the live server.
Here is the problem I face:
I can push fine. No issues. All works as expected. Code gets checked out to public root. Everybody is happy and goes on with their life.
BUT!!!:
The site is a CMS site. Users log in to it and upload things. Files get created on the public root of the website which is the GIT_WORK_TREE.
So NBD right?! I can just commit the files every now and then from the live bare repo and pull them back to my local environment like I have before. So I log into SSH on the server. Navigate to /home/website/website.git
Then run this command:
GIT_WORK_TREE=/home/website/ git add ../
I get this mess:
error: unable to create temporary sha1 filename ./objects/cb: No such file or directory
error: error_log: failed to insert into database
error: unable to index file error_log
fatal: adding files failed
I have done this before on other servers and it worked fine from what I remember. So I was like WTF, must be something strange on this server. I went to another server I have and replicated the EXACT same steps. Got the EXACT same problem. So now I fear I am loosing my sanity and maybe these previous git experiences are all made up in my head.... *Well, lets not go that far yet.... :)
Maybe somebody can help me out here. I have used git plenty and can't seem to crack this one.
Oh, some other maybe useful specs:
running CENTOS 6.2
I double checked all permissions. I even tried changing everything to 777 recursively just to make sure Im not loosing it somehow. Made sure all the files are owned by the correct user. chowned recursively. I also tried the standard solution to this problem which is described here: https://answers.atlassian.com/questions/132671/git-commit-fails-with-sourcetree-error-unable-to-create-temporary-sha1-filename-git-objects-d8-file-exists
That didnt work either. Not sure where to go from here.
PLEASE HELP ME!!!!
I feel like Linus is playing cruel tricks on me right now.
You have a bare git repository in /home/website/website.git which is inside of a git repository in /home/website? And then you try to trick the bare git repository into having a working tree using GIT_WORK_TREE?
Maybe it is worth understanding this setup but only if you intend to be a git developer. As you are a user focused on delivering website functionality, I suggest using a standard git setup.
Move the bare git repository elsewhere:
$ mkdir /home/repo
$ mv /home/website/website.git /home/repo/website.git
$ cd /home/website
$ git remote set-url origin /home/repo/website.git
I got the solution. This is it. I was running the above command from the git repo.
Turns out the command should be run from the work tree and altered to look like this:
GIT_WORK_TREE=/home/website/ git --git-dir="./website.git/" add ./