I have a script that connects via SSH to test-server and retrieves the current Git branch. When I tried to use variables to print the branch and count of the number of modified files, I stack with escaping symbols.
This following works on a local folder:
mc=$(git status -s | grep -E '^[^?]+' -c);
branch=$(git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \1/');
echo $branch \($mc\)
But this won't work:
ssh -i ~/.ssh/id_rsa.cron local.stage "cd /var/www && mc=$(git status -s | grep -E '^[^?]+' -c);
branch=$(git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \1/');
echo $branch \($mc\)"
Things inside double quotes have variable expansion and command substitution performed. So, for instance:
ssh remotehost "echo $(pwd)"
will get the other host to echo what pwd produces on the local host. (I.e., the $(pwd) runs here first, then something like echo /home/user/current/dir is sent to the remote host, which dutifully echoes back the now-constant string.)
You need to prevent the command-substitution (in any suitable manner, for instance by using single quotes):
ssh remotehost 'echo $(pwd)'
which will pass the literal string echo $(pwd) to the remote host (where it will be acted-on by whatever shell you use on that host).
Aside from that, there are some minor improvements you can make to the command sequence:
The git status documentation recommends using --porcelain instead of --short (-s) in scripts.
To get the name of the current branch, use git symbolic-ref -q --short HEAD (this is much simpler than using git branch and extracting the *-ed line and modifying it).
Putting these together and converting the inner quotes to double quotes (this is OK as there are no substitutions that will occur there):
ssh -i ~/.ssh/id_rsa.cron local.stage 'cd /var/www &&
mc=$(git status --porcelain | grep -E "^[^?]+" -c);
branch=$(git symbolic-ref -q --short HEAD);
echo $branch \($mc\)'
There is still a bug here: take note of the binding of the && versus the semicolon. If /var/www does not exist, this leaves mc unset and continues on to attempt to set branch. (It's not a very consequential bug, but it's still not really right.)
Related
I would like to save all the variables that are in the directory in a separate file, cut out duplicates
To begin with, I wrote all the lines with global variables in a separate file
grep -rI "\$.*" folder/ >> output.txt
Then I tried to pull out the variables of this file
cat output.txt | sed /\$.*.[{A-Z}]/p
And output was not what I expected
So how can I take needed variables, when file after grep like this:
something.text_text.txt: - export IMAGE_NAME=${MY_REGISTY}/$MY_PR/${MY_PRNNN} something.text_text.txt:
- docker build --network host -t ${IMAGE_NAME}:${VERSION} -f $DILE_PATH --build-arg setupfile=$SET_FIL> something.text_text.txt:
- docker push ${IMAGE_NAME}:${VERSION} something.text_text.txt: - docker tag ${IMAGE_NAME}:${VERSION} ${IMAGE_NAME}:${MY_BUILD_REF_NAME} something.text_text.txt: - docker push ${IMAGE_NAME}:${MY_BUILD_REF_NAME} something.text.txt:
- /^rel_.*$/ something.text.txt: - eval $(ssh-agent -s) something.text.txt: - chmod 400 $MY_SSH_KEY something.text.txt:
- ssh-add $MY_KEY something.text.txt: - git checkout ${MY_BUIL_NAME} something.text.txt: - git reset --hard origin/${MY_F_NAME} something.text.txt: - mvn -s MY_settings.xml ${MTS} license:add-third-party something.text.txt: - cat ${LICENSE_LIST_FILE} something.text.txt: POM_XML_COMMIT_HASH_LOCAL=$(git log --oneline --follow -- pom.xml | awk '{ print $1 }' | head -n 1) || true something.text.txt: echo POMIT_HASH_LOCAL=${PCOMMIT_HASH_LOCAL} something.text.txt: POM_XML_COMMIT_HASH_REMOTE=$(git log --oneline origin/${MY_BUILD_REF_NAME} --follow -- pom.xml | awk '{ print $1 }' | h> something.text.txt: echo POM_XML_COMMIT_HASH_REMOTE=${POM_OMMIT_HASH_REMOTE} something.text.txt: if [[ ${POM_XML_COMMIT_HASH_LOCAL} = ${POMMIT_HASH_REMOTE} ]]; then something.text.txt: echo "File pom.xml is the same for local and origin ${MY_BUILD_REF_NAME} branch." something.text.txt: echo "New commits are presented in origin/${MY_BUILD_REF} branch for pom.xml file. Skipping." something.text.txt: - git add -f ${LICENSE_LIST_FILE} something.text.txt: - export MY_PUSH_URL=`echo $MY_REPOSITORY_URL | perl -pe 's#.*#(.+?(\:\d+)?)/#git#\1:#'` something.text.txt: - git remote set-url --push origin "${MY_PUSH_URL}" something.text.txt: - git push -f -o ci.skip origin ${MY_BUILD_REF_NAME} something.text_tests.txt: - docker login -u $MY_REGISTRY_USER -p $MY_REGISTRY_PASSWORD $MY_REGISTRY something.text_tests.txt: - export CONFIG_FILE=${HOME}/.docker/config.json something.text_tests.txt:
- export VERSION=$(cat current_version) something.text_tests.txt: - export MY_PROJECT_NAME_UPPER_CASE=$(echo ${MY_PROJECT_NAME} | tr a-z A-Z) something.text_tests.txt: - export ${MY_PROJECUPPER_CASE}_IMAGE=${MYISTRY}/${MY_PROJECT_PATH}/${MY_PROJECT_NAME}:${VERSION} something.text_tests.txt: - docker pull ${MY_REG}/${MY_PR}/${MY_PROJEC}:${VERS}
Try
grep -Po '\$\.*.[{_A-Z}]+' output.txt
-P makes grep using the Perl syntax
-o outputs only the matching parts
Now, improve your regex. For starters, I have already added _ to it, but it would still find ${X}{Y} (false positive) or not find ${lowercase} (false negative) and just partly find ${DIR#/} (because of extra syntax).
You can tell grep to output only matching parts using --only-matching or simply -o.
Real problem here is what really makes a valid variable identifier. This of course depends on for which shell was script written and how many different styles original author of the script used.
Let's assume something sane, identifiers starts with [a-Z] and can only contain alphanumeric characters and underscore. You can also reference same identifier using $MY_VARIABLE or ${MY_VARIABLE}.
I would go with something like that:
grep -rhIo '\$[a-zA-Z_\{\}]*' directory | sort --unique
But be aware that syntax for arrays and operations above variables will break this very quickly.
To get correct results for ${adjacent}text maybe go for
grep -hrEo '\$\{?[A-Za-z_0-9]+\}?' .
This will still not work correctly for
: <<\_
literal $text in a quoted here document
_
echo '$quoted literal text'
echo \$escaped \$dollar \$signs
etc
but for a quick and dirty attempts, maybe just ignore those corner cases, or add some sort of postprocessing to remove them.
Properly solving this requires you to have a sh parser to figure out which dollar signs are quoted etc; and with eval even that won't be complete.
As a comment on other answers here, grep -P is not portable, and requires GNU grep. If you don't have that, and really require Perl regex extensions, maybe simply go with Perl.
perl -lne 'print($&) while m/\$\{?[A-Za-z_0-9]+\}?/go' **/*
The **/* recursive wildcard is not portable either; if you require a POSIX-compatible script, maybe resort to
find . -type f -exec \
'print($&) while m/\$\{?[A-Za-z_0-9]+\}?/go' {} +
though of course Perl isn't at all POSIX either.
am trying to execute the following command:
$ ssh root#10.10.10.50 "tail -F -n 1 $(ls -t /var/log/alert_ARCDB.log | head -n1 )"
ls: cannot access /var/log/alert_ARCDB.log: No such file or directory
tail: cannot follow `-' by name
notice the error returned, when i login to ssh separately and then execute
tail -F -n 1 $(ls -t /var/log/alert_ARCDB.log | head -n1 )"
see the below:
# ls -t /var/log/alert_ARCDB.log | head -n1
/var/log/alert_ARCDB.log
why is that happening and how to fix it. am trying to do this in one line as i don't want to create a script file.
Thanks a lot
Shell parameter expansion happens before command execution.
Here's a simple example. If I type...
ls "$HOME"
...the shell replaces $HOME with the path to my home directory first, then runs something like ls /home/larsks. The ls command has no idea that the command line originally had $HOME.
If we look at your command...
$ ssh root#10.10.10.50 "tail -F -n 1 $(ls -t /var/log/alert_ARCDB.log | head -n1 )"
...we see that you're in exactly the same situation. The $(ls -t ...) expression is expanded before ssh is executed. In other words, that command is running your local system.
You can inhibit the shell expansion on your local system by using single quotes. For example, running:
echo '$HOME'
Will produce:
$HOME
So you can run:
ssh root#10.10.10.50 'tail -F -n 1 $(ls -t /var/log/alert_ARCDB.log | head -n1 )'
But there's another problem here. If /var/log/alert_ARCDB.log is a file, your command makes no sense: calling ls -t on a single file gets you nothing.
If alert-ARCDB.log is a directory, you have a different problem. The result of ls /some/directory is a list of filenames without any directory prefix. If I run something like:
ls -t /tmp
I will get output like
file1
file2
If I do this:
tail $(ls -t /tmp | head -1)
I end up with a command that looks like:
tail file1
And that will fail, because there is no file1 in my current directory.
One approach would be to pipe the commands you want to perform to ssh. One simple way to achieve that is to first create a function that will echo the commands you want executed :
remote_commands()
{
echo 'cd /var/log/alert_ARCDB.log'
echo 'tail -F -n 1 "$(ls -t | head -n1 )"'
}
The cd will allow you to use the relative path listed by ls. The single quotes make sure that everything will be sent as-is to the remote shell, with no local expansion occurring.
Then you can do
ssh root#10.10.10.50 bash < <(remote_commands)
This assumes alert_ARCDB.log is a directory (or else I am not sure why you would want to add head -n1 after that).
I'm getting the following mail every time I execute a specific cronjob. The called script runs fine when I'm calling it directly and even from cron. So the message I get is not an actual error, since the script does exactly what it is supposed to do.
Here is the cron.d entry:
* * * * * root /bin/bash -l -c "/opt/get.sh > /tmp/file"
and the get.sh script itself:
#!/bin/sh
#group and url
groups="foo"
url="https://somehost.test/get.php?groups=${groups}"
# encryption
pass='bar'
method='aes-256-xts'
pass=$(echo -n $pass | xxd -ps | sed 's/[[:xdigit:]]\{2\}/&/g')
encrypted=$(wget -qO- ${url})
decoded=$(echo -n $encrypted | awk -F '#' '{print $1}')
iv=$(echo $encrypted | awk -F '#' '{print $2}' |base64 --decode | xxd -ps | sed 's/[[:xdigit:]]\{2\}/&/g')
# base64 decode input and save to file
output=$(echo -n $decoded | base64 --decode | openssl enc -${method} -d -nosalt -nopad -K ${pass} -iv ${iv})
if [ ! -z "${output}" ]; then
echo "${output}"
else
echo "Error while getting information"
fi
When I'm not using the bash -l syntax the script hangs during the wget process. So my guess would be that it has something to do with wget and putting the output to stdout. But I have no idea how to fix it.
You actually have two questions here.
Why it prints stdin: is not a tty?
This warning message is printed by bash -l. The -l (--login) options asks bash to start the login shell, e.g. the one which is usually started when you enter your password. In this case bash expects its stdin to be a real terminal (e.g. the isatty(0) call should return 1), and it's not true if it is run by cron—hence this warning.
Another easy way to reproduce this warning, and the very common one, is to run this command via ssh:
$ ssh user#example.com 'bash -l -c "echo test"'
Password:
stdin: is not a tty
test
It happens because ssh does not allocate a terminal when called with a command as a parameter (one should use -t option for ssh to force the terminal allocation in this case).
Why it did not work without -l?
As correctly stated by #Cyrus in the comments, the list of files which bash loads on start depends on the type of the session. E.g. for login shells it will load /etc/profile, ~/.bash_profile, ~/.bash_login, and ~/.profile (see INVOCATION in manual bash(1)), while for non-login shells it will only load ~/.bashrc. It seems you defined your http_proxy variable only in one of the files loaded for login shells, but not in ~/.bashrc. You moved it to ~/.wgetrc and it's correct, but you could also define it in ~/.bashrc and it would have worked.
in your .profile, change
mesg n
to
if `tty -s`; then
mesg n
fi
I ended up putting the proxy configuration in the wgetrc. There is now no need to execute the script on a login shell anymore.
This is not a real answer to the actual problem, but it solved mine.
If you run into this problem check if you are getting all the environment variables set as you expect. Thanks to Cyrus for putting me to the right direction.
I'm trying to write a shell script that calls another script that then executes a rsync command.
The second script should run in its own terminal, so I use a gnome-terminal -e "..." command. One of the parameters of this script is a string containing the parameters that should be given to rsync. I put those into single quotes.
Up until here, everything worked fine until one of the rsync parameters was a directory path that contained a space. I tried numerous combinations of ',",\",\' but the script either doesn't run at all or only the first part of the path is taken.
Here's a slightly modified version of the code I'm using
gnome-terminal -t 'Rsync scheduled backup' -e "nice -10 /Scripts/BackupScript/Backup.sh 0 0 '/Scripts/BackupScript/Stamp' '/Scripts/BackupScript/test' '--dry-run -g -o -p -t -R -u --inplace --delete -r -l '\''/media/MyAndroid/Internal storage'\''' "
Within Backup.sh this command is run
rsync $5 "$path"
where the destination $path is calculated from text in Stamp.
How can I achieve these three levels of nested quotations?
These are some question I looked at just now (I've tried other sources earlier as well)
https://unix.stackexchange.com/questions/23347/wrapping-a-command-that-includes-single-and-double-quotes-for-another-command
how to make nested double quotes survive the bash interpreter?
Using multiple layers of quotes in bash
Nested quotes bash
I was unsuccessful in applying the solutions to my problem.
Here is an example. caller.sh uses gnome-terminal to execute foo.sh, which in turn prints all the arguments and then calls rsync with the first argument.
caller.sh:
#!/bin/bash
gnome-terminal -t "TEST" -e "./foo.sh 'long path' arg2 arg3"
foo.sh:
#!/bin/bash
echo $# arguments
for i; do # same as: for i in "$#"; do
echo "$i"
done
rsync "$1" "some other path"
Edit: If $1 contains several parameters to rsync, some of which are long paths, the above won't work, since bash either passes "$1" as one parameter, or $1 as multiple parameters, splitting it without regard to contained quotes.
There is (at least) one workaround, you can trick bash as follows:
caller2.sh:
#!/bin/bash
gnome-terminal -t "TEST" -e "./foo.sh '--option1 --option2 \"long path\"' arg2 arg3"
foo2.sh:
#!/bin/bash
rsync_command="rsync $1"
eval "$rsync_command"
This will do the equivalent of typing rsync --option1 --option2 "long path" on the command line.
WARNING: This hack introduces a security vulnerability, $1 can be crafted to execute multiple commands if the user has any influence whatsoever over the string content (e.g. '--option1 --option2 \"long path\"; echo YOU HAVE BEEN OWNED' will run rsync and then execute the echo command).
Did you try escaping the space in the path with "\ " (no quotes)?
gnome-terminal -t 'Rsync scheduled backup' -e "nice -10 /Scripts/BackupScript/Backup.sh 0 0 '/Scripts/BackupScript/Stamp' '/Scripts/BackupScript/test' '--dry-run -g -o -p -t -R -u --inplace --delete -r -l ''/media/MyAndroid/Internal\ storage''' "
Let's say I want to rename a method in source code contained in a git repository. I could do this by hand, but the method name might be in multiple places (e.g., unit test, documentation, actual method). To check where the method is used, I use 'git grep'. I get 'git grep' to show only lines that I want to change, and then I don't have a workflow to automatically change those lines.
I'm looking for an automated way (hopefully using git tools) to do this last step. I was hoping there was some sort of 'git sed' or equivalent, but I can't find any.
The interface I'm thinking would be nice: git sed 's/old-method-name/new-method-name/g'
You could use git ls-files in combination with xargs and sed:
git ls-files -z | xargs -0 sed -i -e 's/old-method-name/new-method-name/g'
Thanks to both Noufal and Greg for their posts. I combined their solutions, and found one that uses git grep (more robust than git ls-files for my repo, as it seems to list only the files that have actual src code in them - not submodule folders for example), and also has the old method name and new method name in only one place:
In the [alias] block of my ~/.gitconfig file:
sed = ! git grep -z --full-name -l '.' | xargs -0 sed -i -e
To use:
git sed 's/old-method-name/new-method-name/ig'
You could do a
for i in $(git grep --full-name -l old_method_name)
do
perl -p -i -e 's/old_method_name/new_method_name/g' $i
done
stick that in a file somewhere and then alias it as git sed in your config.
Update: The comment by tchrist below is a much better solution since it prevents perl from spawning repeatedly.
Here's a solution that combines those of of Noufal and claytontstanley and avoids touching files that won't change.
In the [alias] block of my ~/.gitconfig file:
psed = !sh -c 'git grep --null --full-name --name-only --perl-regexp -e \"$1\" | xargs -0 perl -i -p -e \"s/$1/$2/g\"' -
To use:
git psed old_method_name new_method_name
Yes, there's. In Ubuntu the package git-extras provides the command. Install it:
$ sudo apt-get install git-extras
Use it like bellow e.g. to correct a spelling issue quickly:
$ git sed 'qoute' 'quote'
Unfortunately it doesn't support file filters like what git grep does:
$ git grep -e 'class' -- '*.py'
The same functionality also exists on Mac and other operating systems. Checkout its installation page.
Unhappy with most other solutions provided (which is basically just a string-replace on git tracked files) I wrote my own script: git-sed.
It supports any expression sed supports (e.g git sed '1{/^$/d}')
Can run on a subset of paths in the repo (git sed 's/foo/bar' src tests)
Multiple expressions (git sed -e 's/foo/bar' -e '/bar/d').
etc...
Just drop it anywhere in PATH to use it or add an alias pointing to the full path.
Note that starting git 2.1 (Q3 2014), you can set "full-name" by default for git grep.
(See commit 6453f7b by Andreas Schwab)
"git grep" learned grep.fullname configuration variable to force "--full-name" to be default.
This may cause regressions on scripted users that do not expect this new behaviour.
That means the previous solutions can benefit from:
git config grep.full-name true
And use:
psed = !sh -c 'git grep --null --name-only --perl-regexp -e \"$1\" | xargs -0 perl -i -p -e \"s/$1/$2/g\"' -
See git-search-replace on github - it's designed for this exactly.
I have written a git sed which supports file filtering:
#!/bin/bash
split=$(($# + 1))
for i in $(seq 1 $#); do
if [[ "${!i}" = "--" ]]; then
split=$i
fi
done
git ls-files -z "${#:$split:$#}" | xargs -0 sed -b -i "${#:1:$(($split - 1))}"
(You probably don't want the -b parameter on non-Windows platforms; it's necessary on Windows to preserve Windows-style newlines.)
You can then add an alias in your .gitconfig:
[alias]
sed = ! <path to git-sed>
so that you can use it like git sed -e <your expression> -- <path filter>.