I have a script on a remote machine which contains a for loop as below:
#!/bin/bash -eux
# execute build.sh in each component
for f in workspace/**/**/build.sh ; do
echo $f
REPO=${f%*build.sh}
echo $REPO
git -C ./$REPO checkout master
git -C ./$REPO pull origin master
$f
done
This script is finding all the repos with a build.sh file inside, pulls the latest changes and build them.
This works fine when I execute the scrript on the machine but when I try to trigger this script remotely the for loop just runs once, and I see that it returns a repo which actually doesn't have build.sh at all:
$ ssh devops "~/build.sh"
+ for f in workspace/**/**/build.sh
+ echo 'workspace/**/**/build.sh'
+ REPO='workspace/**/**/'
+ echo workspace/core/auth/
workspace/**/**/build.sh
workspace/core/auth/
+ git -C ./workspace/core/auth/ checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.
+ git -C ./workspace/core/auth/ pull origin master
From https://gitlabe.com/workspace/core/auth
* branch master -> FETCH_HEAD
Already up to date.
+ 'workspace/**/**/build.sh'
/home/devops/build.sh: line 10: workspace/**/**/build.sh: No such file or directory
I tried to make a one-liner of the for loop and use ssh and that also didn't work. How can I solve this problem?
You need to enable globbing on the remote machine. Add this to the beginning of your script:
shopt -s globstar
Also see this thread
Related
This is essentially the same issue as this individual
I have a bitbucket pipeline file bitbucket-pipelines.yml which executes the file deploy.sh when a new commit is made to the main branch. deploy.sh in turn calls pull.sh which performs a set of actions:
If it exists, kill the existing refgator-api.py process
Change to the directory containing the repo
Pull from the repo
Change to the directory containing refgator-api.py
Execute python3 refgator-api.py
It's at this last step that my bitbucket pipeline will continue executing (consuming all my build minutes).
Is there any way I can complete the bitbucket pipeline successfully after pull.sh has performed python3 refgator-api.py?
bitbucket-ipelines.yml
image: atlassian/default-image:latest
pipelines:
default:
- step:
script:
- cat ./deploy.sh | ssh -tt root#xxx.xxx.xxx.xxx
- echo "Deploy step finished"
deploy.sh
echo "Deploy Script Started"
cd
sh pull.sh
echo "Deploy script finished execution"
pull.sh
## Kills the current process which is restarted later
kill -9 $(pgrep -f refgator-api.py)
## And change to directory containing the repo
cd eg-api
## Pull from the repo
export GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa.pub"
GIT_SSH_COMMAND="ssh -v" git pull git#bitbucket.org:myusername/myrepo.git
## Change to directory containing the python file to execute
cd refgator-api
python3 refgator-api.py &
The key issue here was the attempt at getting the python script refgator-api.py up and running and closing off the session.
This doesn't seem to be possible using a shell script directly. However, it is possible to use supervisor on the remote server.
In this case I installed supervisor apt-get install supervisor and did the following:
bitbucket pipelines
image: atlassian/default-image:latest
pipelines:
default:
- step:
script:
- cat ./deploy.sh | ssh -tt root#143.198.164.197
- echo "DEPLOY STEP FINISHED"
Deploy.sh
printf "=== Deploy Script Started ===\n"
printf "Stop all supervisorctl processes\n"
supervisorctl stop all
sh refgator-pull.sh
printf "Start all supervisorctl processes\n"
supervisorctl start all
exit
Pull.sh
printf "==== Repo Pull ====\n"
printf "Attempting pull from repo\n"
export GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa.pub"
GIT_SSH_COMMAND="ssh " git pull git#bitbucket.org:myusername/myrepo.git
printf "Repo: Local Copy Updated\n"
I am calling a python script from the job project> settings> build> run shell (bash).
The file I want to open in the script is not up to date, and Jenkins always remembers the old file I deleted in the script and Jenkins opens it.
I also found that the python delete command was not executed.
It looks like Jenkins is caching the initial file tree.
How can I always refer to the latest file tree?
Is there a command to clear the cache?
And how do I run the python delete command (os.remove(latest_sc_file))?
#!/bin/bash
echo "Python3 --version is: "`python3 --version`
# Python3 --version is: Python 3.5.2
echo "Python3 full-path is: "`which python3`
# Python3 full-path is: /usr/bin/python3
git checkout submit
# Check for added / changed files
for file in `git diff --name-only HEAD..origin/submit`
do
# echo $file
name=`echo $file | sed -r 's/.*submission_(.*).csv/\1/'`
echo "name: "$name
# pull submission.csv, test.csv
git pull origin submit:submit
# Start scoring
/home/kei/.pyenv/shims/python3 ./src/go/score.py $name linux > ./src/go/score.log 2>&1
done
The cause has been found.
It was because Jenkins wasn't caching it, but was downloading a remote file every time I git pulled it.
Therefore, I will withdraw this question.
I have a website running on cloud server. Can I link the related files to my github repository. So whenever I make any changes to my website, it get auto updated in my github repository?
Assuming you have your cloud server running an OS that support bash script, add this file to your repository.
Let's say your files are located in /home/username/server and we name the file below /home/username/server/AUTOUPDATE.
#!/usr/bin/env bash
cd $(dirname ${BASH_SOURCE[0]})
if [[ -n $(git status -s) ]]; then
echo "Changes found. Pushing changes..."
git add -A && git commit -m 'update' && git push
else
echo "No changes found. Skip pushing."
fi
Then, add a scheduled task like crontab to run this script as frequent as you want your github to be updated. It will check if there is any changes first and only commit and push all changes if there is any changes.
This will run every the script every second.
*/60 * * * * /home/username/server/AUTOUPDATE
Don't forget to give this file execute permission with chmod +x /home/username/server/AUTOUPDATE
This will always push the changes with the commit message of "update".
i am trying to find the name of git branch on remote server using a shell script. I put the following command in a script under the bin directory.
git symbolic-ref --short HEAD
When I execute the script using ssh from another machine
ssh -i keyfile.pem user#ipaddress 'bash -s' /path/to/the/script
I get an error
fatal: Not a git repository (or any of the parent directories): .git
Not sure where I am doing wrong. Any help would be appreciated.
Thanks.
Your call to git is using the wrong working directory (likely your home directory).
In your script, either cd to the path containing the git directory or specify the -C option with git:
git -C /path/to/git/checkout symbolic-ref --short HEAD
The -C option allows you to overwrite the working directory:
-C <path>
Run as if git was started in <path> instead of the current working directory. When
multiple -C options are given, each subsequent non-absolute -C <path> is interpreted
relative to the preceding -C <path>.
As I wrote in the object, I have some parts of a component I'm writing that have to be used in other projects.
I don't want to use all the component but only the "common" parts.
So, I created a "Common" folder and in it I put all the common files I need to share across projects.
Now, I need to automatically create a subtree split of this folder and push its contents to a read only repository I have on BitBucket.
Just for information: to deal with the authentication part, I've created a new ssh key pair and put the private key in an environment variable: this variable is then used by the bash script to create the file to pass to ssh-add. The public key is set in my BitBucket account.
The problem is that the script seems not pushing the splitted code to the remote read-only repository. While executing manually all the commands all works well, I'm not able to make this work in an automated way.
More, as I have never used bash to write script, I'm sure there are a lot of errors in it and that it can be deeply refactored to be better.
Following the code of the script:
#!/usr/bin/env bash
cd ~/src/bitbucket.org/Aerendir/component-remotes/
echo "> Current working directory: $PWD"
echo '> Creating the key file'
printf %q -v "$ssh_component_remotes_common_key" > bitbucket_key
chmod 400 bitbucket_key
ls ~/src/bitbucket.org/Aerendir/component-remotes/
echo '> Adding the identity for bitbucket.org to config'
cat <<EOT >> ~/.ssh/config
Host bitbucket.org
IdentityFile ~/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key
EOT
cat ~/.ssh/config
echo '> Adding the key to SSH agent'
eval "$(ssh-agent -s)"
/usr/bin/expect <<EOF
spawn ssh-add ${HOME}/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key
expect "Enter passphrase for key '/home/rof/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key':"
send "$ssh_component_remotes_common_pass";
interact
EOF
echo -e '\n > Creating the subtree repository'
mkdir _component-remotes-common
cd _component-remotes-common
git init --bare
git remote add origin git#bitbucket.org:Aerendir/component-remotes-common.git
/usr/bin/expect <<EOF
spawn git remote show origin
expect "Enter passphrase for key '/home/rof/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key':"
send "$ssh_component_remotes_common_pass";
interact
EOF
echo -e '\n > Splitting in the subtree repository'
cd ../
echo "> Current working directory: $PWD"
git subtree split --prefix=src/Remotes/Common -b split
git push _component-remotes-common split:master
echo '> Pushing to the remote repo'
cd _component-remotes-common
echo "> Current working directory: $PWD"
/usr/bin/expect <<EOF
spawn git push origin master
expect "Enter passphrase for key '/home/rof/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key':"
send "$ssh_component_remotes_common_pass";
interact
EOF
echo "$expect_out(0, string)"
And this is the output:
> Current working directory: /home/rof/src/bitbucket.org/Aerendir/component-remotes
> Creating the key file
bin composer.json docs phpunit.xml.dist src
bitbucket_key composer.lock log readmegen.yml tmp
CHANGELOG.md CONTRIBUTING.md phpdoc.xml.dist README.md
> Adding the identity for bitbucket.org to config
UserKnownHostsFile=/dev/null
StrictHostKeyChecking=no
ServerAliveInterval 3
ServerAliveCountMax 600
Host bitbucket.org
IdentityFile ~/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key
> Adding the key to SSH agent
Agent pid 5919
spawn ssh-add /home/rof/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key
Enter passphrase for /home/rof/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key:
> Creating the subtree repository
Initialized empty Git repository in /home/rof/src/bitbucket.org/Aerendir/component-remotes/_component-remotes-common/
spawn git remote show origin
Warning: Permanently added 'bitbucket.org,104.192.143.2' (RSA) to the list of known hosts.
Enter passphrase for key '/home/rof/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key':
> Splitting in the subtree repository
> Current working directory: /home/rof/src/bitbucket.org/Aerendir/component-remotes
Created branch 'split'
490b1f471932a308075c568f21c36bab5f102818
Counting objects: 8, done.
Delta compression using up to 36 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (8/8), 667 bytes | 0 bytes/s, done.
Total 8 (delta 0), reused 5 (delta 0)
To _component-remotes-common
* [new branch] split -> master
> Pushing to the remote repo
> Current working directory: /home/rof/src/bitbucket.org/Aerendir/component-remotes/_component-remotes-common
spawn git push origin master
Warning: Permanently added 'bitbucket.org,104.192.143.3' (RSA) to the list of known hosts.
Enter passphrase for key '/home/rof/src/bitbucket.org/Aerendir/component-remotes/bitbucket_key': (0, string)
It seems to work untile the push: at that point, I receive no other output. Obviously the remote repo remains empty but I don't know which is the problem as I don't receive any error as output.
I'm working on this script since about 5 hours and I'm very jaded: any help is greatly appreciated to make this work! Thankyou...