How to automate docker deployment based on GitHub webhook? - node.js

So I have inherited an application, and am not too happy with the current deployment workflow. There is a docker instance running on an ec2 server, and developers have been merging their local work like this:
rsync -arhvz --progress ./ ubuntu#52.12.345.67:/home/ubuntu/app --exclude node_modules
And then going into the server itself and running a deploy.sh script. I would simply like a push event to the master branch to trigger this sync and deploy. How is it possible? As far as I know we don't pay for a Docker hub account - we don’t have private docker registry.

Some solutions :
use a git pre-push hook script which will execute the script before each local push to remote
create a bare git repo on your server, push your change to this git remote and checkout the last change using post-receive hook
track push events on Github with Github webhook, you will need to have your server listen to a port in order to receive the webhook events
pre-push hook
In your local repository, edit/create .git/hooks/pre-push file :
#!/bin/bash
rsync -arhvz --progress ./ ubuntu#52.12.345.67:/home/ubuntu/app --exclude node_modules
ssh ubuntu#52.12.345.67 "/home/ubuntu/deploy.sh"
This way, your modification will sync before each local push to remote. The deploy.sh is also triggered
setup a git repo on your server and deploy to it using git
You can create a bare git repo on your server and create a post-receive hook script that will use git to checkout your server repo. From this guide :
#!/bin/bash
while read oldrev newrev ref
do
# only checking out the master (or whatever branch you would like to deploy)
if [[ $ref =~ .*/master$ ]];
then
echo "Master ref received. Deploying master branch to production..."
git --work-tree=/home/ubuntu/app/deploy-folder/ --git-dir=/home/ubuntu/app/project.git/ checkout -f
else
echo "Ref $ref successfully received. Doing nothing: only the master branch may be deployed on this server."
fi
done
Note that this solution only uses git
Github Webhook
You can use this docker-hook project which is a Python server listening for a POST on your webhook uri. It was originally used for Docker Hub webhook but also works for Github webhook (though not parsing the events).
Webhook
On your server :
download docker-hook python script
curl https://raw.githubusercontent.com/schickling/docker-hook/master/docker-hook > /usr/local/bin/docker-hook; chmod +x /usr/local/bin/docker-hook
start it with a token (generated with uuidgen for example) :
docker-hook -t 3ea4e9d8-8fff-47e5-a704-65ab21de6963 -c /path/to/deploy.sh
In Github, go to your repo settings, create the webhook with the token as path :
Check that the webhook is working (in Recent Deliveries tab)
Deploy key
Now, as you want to trigger a git pull on your server, you will need to generate a Deploy Key which is a ssh key used to deploy (read-only), you will need, on server side :
install git if not already installed
generate a key
copy this key in your repo/settings/Deploy key tab
See this post for a complete tutorial
Now you need to edit your deploy.sh script to perform a git pull or a fresh git clone to the location of your choice

Related

custom_hooks not working on gitlab/docker

I use a docker image of gitlab, https://hub.docker.com/r/sameersbn/gitlab/ on a synology nas to manage our pipeline dev.
I plan to deploy any push in the master branch to a production folder, where all our software looks to load scripts/ plugins.
To deploy to the production server, on the same server but outside the docker image.
the production folder is set as a clone of the server
git clone /gitserver/path /prod/folder/path
all works as planned
then, I've tryied to use a custom_hook for a repo as described on the documentation here : https://docs.gitlab.com/ee/administration/custom_hooks.html
the custom hooks is called "post-update", chmod +x and rights allowed to anyone (for testing). here is the content of the hook
#!/bin/bash
echo "********** publishing to prod *********"
cd /prod/folder/path
unset GIT_DIR
git pull origin master
echo "********** Publish to prod complete *********"
when I push to the git server, nothing happened.
If I run manually the post-update script, all work as espected so it seems that the problem come from gitlab custom_hooks support.
I have seen many posts about problems with custom hooks but none of them helps me
https://gitlab.com/gitlab-org/gitlab-ce/issues/1742
https://gitlab.com/gitlab-org/gitlab-ce/issues/38193
Gitlab custom hook not running
how can I make custom_hooks works ?

Gitlab pushing first project failed due to could not read from remote repository

I was new to Gitlab was trying to push my project from local machine to Gitlab.
Have done the SSH key and followed the instructions at Gitlab. Done the Git global setup. Was trying to add an existing folder , so i followed the instructions listed
cd existing_folder
git init
git remote add origin https://gitlab.com/sss/testnode.git
git add .
git commit -m "Initial commit"
git push -u origin master
but failed at the last step at the git push. The error message was
Tried adding the remote origin, but it was told it already exists. So not sure where it went wrong. Please help, much appreciated :)
Have done the SSH key
The problem is that you have defined your origin as HTTPS, not SSH.
Try:
git remote set-url origin git#gitlab.com:sss/testnode.git
That will override origin URL.
Independently, make sure your SSH key does work and allows GitLab to authenticate you as your GitLab account with:
ssh -T git#gitlab.com
Check out your credentials, if they are invalid, it wont give u to upload changes.
For Windows check this: https://www.digitalcitizen.life/credential-manager-where-windows-stores-passwords-other-login-details
For Linux check this: https://askubuntu.com/questions/30907/password-management-applications (if you do not know how to change credentials via terminal)

How to upload projects on gitlab by user?

We are using Gitlab (7.5.3).
I created a Repository, but I want to upload my project to it.
I've looked on the repository page for an upload button of some kind but I haven't seen anything of the sort.
I've looked at the links provided so far but I'm still getting nowhere. They mention command line, is that Windows command line ?
I am able to upload through Git GUI, how to upload without Git GUI?
And when I login to user account through putty ,it shows
login as: root
root#192.168.1.5's password:
Last login: Tue May 26 12:20:10 2015 from 192.168.1.12
![grep: write error]
Does anyone knows how we can solve these issue?
If you have Git GUI, you probably also have Git Bash which provide command line support for git inside Windows.
You can use this Git Bash to upload your project. For this, you just need some git command, and you don't have to log in the Gitlab's server (I supposed it's why you tried). This command are provided by Gitlab when you create a new project and want to upload it.
First open Git Bash. Then :
cd c:\Users\Me\..\my_project\ # Go to your project directory in Windows
git init # Initialize this directory as a git repo
git remote add origin git#your_gitlab_server_ip:your_username/your_project_name.git
git push -u origin master` # Send your code to your Gitlab instance
It's also a good idead to provide to git some info about you. This infos will be displayed by Gitlab :
git config --global user.name "Your Name"
git config --global user.email "your_name#your_mail.com"
Remember that with Gitlab the only interaction between your git's repo and Gitlab should happend through the git command, and not with login to this server.
Try this 4 step push project to gitlab --> open command line.
cd to/path/local
git init
git remote add anything_name name http://scmgit.ktb/username/projectname.git
git push -u test anything_name master
Hope is save you day.

I want to add remote branch of my local repo?

I want to put my local git project in production server (Linux multi hosting account of go-daddy).
Is it possible to add my production server as my remote branch ?
So, having logged in via SSH:
$ mkdir mywebsite.git
$ cd mywebsite.git
$ git init --bare
on local git repo
$ git remote add production git#myserver.com:mywebsite.git
$ git push production +master:refs/heads/master
I got my solution,but thnx all of you !! :-)
You can add a new remote:
git remote add prod /url/to/empty/bare/repo
And then push your master branch to it:
git push -u prod master
But that supposes you initiated a bare repo on that production server first.
If git is installed on your server and you have cloned a copy of the repository there, you can add it as a remote using git remote.
git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=<fetch|push>] <name> <url>
So:
git remote add production URL
You'd need to push from local and pull on the remote though. So you'd still have to SSH into your server to deploy.
If your end goal is simplified deployment you may wish to look into tools like Jenkins, which you can run locally and can be configured to connect to your server and deploy after running tests, etc.
That might be your best option if your hosting provider won't let you install Git.
Yes possible. You need the following command:
git remote add (remote-name)

Dummy questions about setting up git on amazon cloud ec2

first of all, apologize for dummy questions that I might throw here. It would be nice if you could point the directions where should I go from here.
I'm totally new to version control(as well as git) and cloud system. However, it came to the point that I have to develop php web based application on AWS EC2 instance and make codes contributable for future developers.
I did successfully create EC2 instance that run PHP/MySQL and map the domain with Elastic IP. So the website is now publicly accessible via port 80.
I also installed git using $sudo yum install git and configed user.name and user.email
I then, go to root folder of the website (e.g. public_html) and run ‘git init’ which create the fold “.git” and I then add file using “git add .” and commit “git commit -m ‘initial upload’”
Is that the right way to go? Would it be ok to have the project folder sitting on /public_html (where accessible from anyone).
If above is ok, then where should I go from here?
I would like to have git server running on EC2 that allow developers to connect from their local machines (e.g. Eclipse) while being able to keep the backup and compare the different between codes.
What detail do I suppose to give developers so that they can connect to git server and working on project?
I quick direction or few keywords to do more research would help.
look here for more information on setting up git on amazon ec2
to allow developers to use you git, you just need to give them the git server url.
Direct quote from the site i'm linking to.
"First and foremost, you need to add your EC2 identity to the ssh
authentication agent. This prevents problems with git later, namely
getting the error “Permission denied (publickey).” when trying to do a
git push to the EC2 repository.
ssh-add path/to/privateEC2key.pem
Now you can go ahead and create the git repository on the EC2
instance.
ssh username#hostname.com
mkdir the_project.git
cd the_project.git
git init --bare
So not much going on here, all we do is create an empty repository and
then leave. Now, on the local machine, you do something like the
following:
cd the_project
git init
git add .
git commit -m "Initial git commit message"
git remote add origin username#hostname.com:the_project.git
git config --global remote.origin.receivepack "git receive-pack"
git push origin master
The ‘git config’ command is a fix that I found necessary to be able to
push to the EC2 repository."
The mentioned link by Alex gives a good starting point to setup git on ec2. But I followed a little different approach as mentioned here. link. Direct Quotes from the page:
"Connecting with SSH without a PEM key" : So either you add the ec2 private key and add it as a entity in your ssh authentication agent or create a new ssh key for your user and use that. Steps to be followed are:
Create SSH Key
First up you will need to navigate to your .ssh folder on your local machine :
cd
cd .ssh
if this folder doesn’t exist use mkdir to make it.
Once in your ssh folder on your local machine which should be in /Users/yourusername/.ssh generate your key by executing the following.
ssh-keygen -t rsa -b 1024
When prompted enter the file name to save the key enter id_rsa_aws, when prompted to enter a password leave blank.
In your .ssh directory execute the following command and copy the output to paste later.
cat id_rsa_aws.pub
Now connect to you AWS instance using you PEM key
ssh -i path/to/yourkeyname.pem ubuntu#xx.xxx.xxx.xxx
Once in
echo 'the key you copied from id_rsa_aws.pub' >> .ssh/authorized_keys
chmod 640 .ssh/authorized_keys
chmod 750 .ssh
Now you go to your machine and type
cd desired directory
git clone ubuntu#xx.xxx.xxx.xxx:<path_to_your_just_created_git_server>
If you did all the above mentioned steps correct, the only warning you might get is
warning: You appear to have cloned an empty repository.
That's ok. Now you can copy all your code into the clone directory, and follow the following steps:
git add .
git commit -m "Initial commit"
git push origin master // If working on master branch
i created a GitHub gist with all the details hope it helps
https://gist.github.com/eslam-mahmoud/35777e4382599438023abefc9786a382
//add your EC2 .pem file to ssh kys
ssh-add ~/aws/mypemfile.pem
//create bare repo on AWS EC2 webserver and deploy on demand
mkdir ~/git/the_project
cd ~/git/the_project
git init --bare
//create local repo and track remote one
cd ~/git/the_project
git init
git add .
git commit -m "Initial git commit message"
git remote add aws ubuntu#1.1.1.1:~/git/the_project
git config --global remote.origin.receivepack "git receive-pack"
git push aws master
//create tag
git tag -a v0.1 -m "my version 0.1"
//push tags
git push aws --tags
//Or you have one so you push your updates
git remote add aws ubuntu#1.1.1.1:~/git/the_project
git config --global remote.origin.receivepack "git receive-pack"
git push aws master
//create tag
git tag -a v0.1 -m "my version 0.1"
//push tags
git push aws --tags
//on server create another local repo that track the bare one to deploy
git clone ~/git/the_project
cd ./the_project
//checkout tag
git checkout v0.1
//install clear cache ...
npm install

Resources