I'm trying to figure out how to work git filter-branch and I need help with some basic Linux scripting commands.
'git ls-files -s | sed "s-\t-&newsubdir/-" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
git update-index --index-info &&
mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
Can anyone break this down for me and explain each part?
I'm most interested in the '|' notation, the && notation, and the sed command.
| connects stdout of the preceding command to stdin of the following command.
&& executes the following command if the return code of the preceding command is 0 (that is, the command succeeded).
The s command in sed is a substitution. It searches for a regex match for the first argument and replaces it with the second. The argument separator is traditionally /, but it will use the first character that follows, - in this case. The & in the replacement is replaced with the entire match.
"|" is the unix pipe command which connects the command on the left sides output to the command on the right side's input.
"&&" is the unix "and" command which will execute the command on its right if and only if the command on it's left completes successfully
"sed" is the unix "stream editor" a powerful editing tool which parses and edits it's input
Related
How to pass argument to sed without input files. The sed is in single quote and has pipe in it already.
git filter-branch -f --index-filter \
'git ls-files -s | sed -i "s-\t\"*-&dirname/-" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
git update-index --index-info &&
mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD
The above code works fine if dirname is hard-coded. But i need to pass the dirname from command line arguments to script. I am trying like below but no luck.
'set subdir=$folder;git ls-files -s | sed -i "s-\t\"*-&$subdir/-" |
The sed says input file is missing. I can not directly pass the input file to sed. I am seeing error like below.
Rewrite 9c86de42b7f3228e3d45a278c8caf7e24c8e55cf (1/2) (0 seconds passed, remaining 0 predicted)
sed: no input files
You have two choices here. You can either evaluate $subdir to build up a fixed sed command that you then set as your filter; or you can evaluate a variable in the shell fragment that git filter-branch will invoke.
To understand the latter, realize that your --index-filter string becomes an ordinary shell variable:
--index-filter)
filter_index="$OPTARG"
;;
which is then passed to eval:
eval "$filter_index" < /dev/null ||
die "index filter failed: $filter_index"
The eval means that the expression in $filter_index, set from your --index-filter argument, has access to all the shell variables and functions in the filter-branch script. Unfortunately, none of its private variables holds the expression you'd like—but you can access its environment variables, which means you can put the value into an environment variable. In other words, you could supply subdir=<whatever> as an environment to your original expression.
In any case, as bk2204 answered, you need to remove the -i option. Besides that, some versions of sed don't accept \t as a tab character (presumably yours does, just be aware of this).
To expand the variable earlier, just do that. For instance:
... --index-filter \
'git ls-files -s | sed "s-\t\"*-&'$folder'/-" | ...
(I've removed the -i here myself). Note how this exits single quotes, expands $folder, then re-enters single quotes. If $folder might contain whitespace, be sure to use double quotes while expanding it here:
... --index-filter \
'git ls-files -s | sed "s-\t\"*-&'"$folder"'/-" | ...
The nesting of quotes here is pretty tricky: the stuff inside the single quotes is all one big string, provided as the argument that sets the variable $filter_index inside the filter-branch script. The eval runs it through a second pass of evaluation, breaking up into the pipeline (git ls-files, piped to sed, piped to git update-index) and running the various multiple commands, all of which have their stdin redirected to /dev/null.
You're using sed -i, which edits its command-line arguments (which are names of files) in place. However, you're reading from standard input and haven't provided any command-line arguments. If you want to just filter the standard input like this, omit the -i from your sed command.
I am looking to write a simple script to perform a SSH command on many hosts simultaneously, and which hosts exactly are generated from another script. The problem is that when I run the script using sometihng like sed it doesn't work properly.
It should run like sshall.sh {anything here} and it will run the {anything here} part on all the nodes in the list.
sshall.sh
#!/bin/bash
NODES=`listNodes | grep "node-[0-9*]" -o`
echo "Connecting to all nodes and running: ${#:1}"
for i in $NODES
do
:
echo "$i : Begin"
echo "----------------------------------------"
ssh -q -o "StrictHostKeyChecking no" $i "${#:1}"
echo "----------------------------------------"
echo "$i : Complete";
echo ""
done
When it is run with something like whoami it works but when I run:
[root#myhost bin]# sshall.sh sed -i '/^somebeginning/ s/$/,appendme/' /etc/myconfig.conf
Connecting to all nodes and running: sed -i /^somebeginning/ s/$/,appendme/ /etc/myconfig.conf
node-1 : Begin
----------------------------------------
sed: -e expression #1, char 18: missing command
----------------------------------------
node-1 : Complete
node-2 : Begin
----------------------------------------
sed: -e expression #1, char 18: missing command
----------------------------------------
node-2 : Complete
…
Notice that the quotes disappear on the sed command when sent to the remote client.
How do I go about fixing my bash command?
Is there a better way of achieving this?
Substitute an eval-safe quoted version of your command into a heredoc:
#!/bin/bash
# ^^^^- not /bin/sh; printf %q is an extension
# Put your command into a single string, with each argument quoted to be eval-safe
printf -v cmd_q '%q ' "$#"
while IFS= read -r hostname; do
# run bash -s remotely, with that string passed on stdin
ssh -q -o 'StrictHostKeyChecking no' "$hostname" "bash -s" <<EOF
$cmd_q
EOF
done < <(listNodes | grep -o -e "node-[0-9*]")
Why this works reliably (and other approaches don't):
printf %q knows how to quote contents to be eval'd by that same shell (so spaces, wildcards, various local quoting methods, etc. will always be supported).
Arguments given to ssh are not passed to the remote command individually!
Instead, they're concatenated into a string passed to sh -c.
However: The output of printf %q is not portable to all POSIX-derived shells! It's guaranteed to be compatible with the same shell locally in use -- ksh will always parse output from printf '%q' in ksh, bash will parse output from printf '%q' in bash, etc; thus, you can't safely pass this string on the remote argument vector, because it's /bin/sh -- not bash -- running there. (If you know your remote /bin/sh is provided by bash, then you can run ssh "$hostname" "$cmd_q" safely, but only under this condition).
bash -s reads the script to run from stdin, meaning that passing your command there -- not on the argument vector -- ensures that it'll be parsed into arguments by the same shell that escaped it to be shell-safe.
You want to pass the entire command -- with all of its arguments, spaces, and quotation marks -- to ssh so it can pass it unchanged to the remote shell for parsing.
One way to do that is to put it all inside single quotation marks. But then you'll also need to make sure the single quotation marks within your command are preserved in the arguments, so the remote shell builds the correct arguments for sed.
sshall.sh 'sed -i '"'"'/^somebeginning/ s/$/,appendme/'"'"' /etc/myconfig.conf'
It looks redundant, but '"'"' is a common Bourne trick to get a single quotation mark into a single-quoted string. The first quote ends single-quoting temporarily, the double-quote-single-quote-double-quote construct appends a single quotation mark, and then the single quotation mark resumes your single-quoted section. So to speak.
Another trick that can be helpful for troubleshooting is to add the -v flag do your ssh flags, which will spit out lots of text, but most importantly it will show you exactly what string it's passing to the remote shell for parsing and execution.
--
All of this is fairly fragile around spaces in your arguments, which you'll need to avoid, since you're relying on shell parsing on the opposite end.
Thinking outside the box: instead of dealing with all the quoting issues and the word-splitting in the wrong places, you could attempt to a) construct the script locally (maybe use a here-document?), b) scp the script to the remote end, then c) invoke it there. This easily allows more complex command sequences, with all the power of shell control constructs etc. Debugging (checking proper quoting) would be a breeze by simply looking at the generated script.
I recommend reading the command(s) from the standard input rather than from the command line arguments:
cmd.sh
#!/bin/bash -
# Load server_list with user#host "words" here.
cmd=$(</dev/stdin)
for h in ${server_list[*]}; do
ssh "$h" "$cmd"
done
Usage:
./cmd.sh <<'CMD'
sed -i '/^somebeginning/ s/$/,appendme/' /path/to/file1
# other commands
# here...
CMD
Alternatively, run ./cmd.sh, type the command(s), then press Ctrl-D.
I find the latter variant the most convenient, as you don't even need for here documents, no need for extra escaping. Just invoke your script, type the commands, and press the shortcut. What could be easier?
Explanations
The problem with your approach is that the quotes are stripped from the arguments by the shell. For example, the argument '/^somebeginning/ s/$/,appendme/' will be interpreted as /^somebeginning/ s/$/,appendme/ string (without the single quotes), which is an invalid argument for sed.
Of course, you can escape the command with the built-in printf as suggested in other answer here. But the command becomes not very readable after escaping. For example
printf %q 'sed -i /^somebeginning/ s/$/,appendme/ /home/ruslan/tmp/file1.txt'
produces
sed\ -i\ /\^somebeginning/\ s/\$/\,appendme/\ /home/ruslan/tmp/file1.txt
which is not very readable, and will look ugly, if you print it to the screen in order to show the progress.
That's why I prefer to read from the standard input and leave the command intact. My script prints the command strings to the screen, and I see them just in the form I have written them.
Note, the for .. in loop iterates $IFS-separated "words", and is generally not preferred way to traverse an array. It is generally better to invoke read -r in a while loop with adjusted $IFS. I have used the for loop for simplicity, as the question is really about invoking the ssh command.
Logging into multiple systems over SSH and using the same (or variations on the same) command is the basic use case behind ansible. The system is not without significant flaws, but for simple use cases is pretty great. If you want a more solid solution without too much faffing about with escaping and looping over hosts, take a look.
Ansible has a 'raw' module which doesn't even require any dependencies on the target hosts, and you might find that a very simple way to achieve this sort of functionality in a way that frees you from the considerations of looping over hosts, handling errors, marshalling the commands, etc and lets you focus on what you're actually trying to achieve.
We have couple of scripts where we want to replace variable evaluation method from $VAR_NAME to ${VAR_NAME}
This is required so that scripts will have uniform method for variable evaluation
I am thinking of using sed for the same, I wrote sample command which looks like follows,
echo "\$VAR_NAME" | sed 's/^$[_a-zA-Z0-9]*/${&}/g'
output for the same is
${$VAR_NAME}
Now i don't want $ inside {}, how can i remove it?
Any better suggestions for accomplishing this task?
EDIT
Following command works
echo "\$VAR_NAME" | sed -r 's/\$([_a-zA-Z]+)/${\1}/g'
EDIT1
I used following command to do replacement in script file
sed -i -r 's:\$([_a-zA-Z0-9]+):${\1}:g' <ScriptName>
Since the first part of your sed command searches for the $ and VAR_NAME, the whole $VAR_NAME part will be put inside the ${} wrapper.
You could search for the $ part with a lookbehind in your regular expression, so that you end up ending the sed call with /{&}/g as the $ will be to the left of your matched expression.
http://www.regular-expressions.info/lookaround.html
http://www.perlmonks.org/?node_id=518444
I don't think sed supports this kind of regular expression, but you can make a command that begins perl -pe instead. I believe the following perl command may do what you want.
perl -p -e 's/(?<=\$)[_a-zA-Z0-9]*/{$&}/g'
PCRE Regex to SED
This is my bash script named ooo.
#!/bin/bash
$1
This command works great.
./ooo 'echo test'
The output is:
test
This command fails.
./ooo 'cd / && ls'
There is no output.
Why is there no output?
Try this:
#!/bin/bash
eval "$1"
If your question is "how do I do what I expect to happen here?", then #ooga's answer is perfect.
If your question is "why doesn't this do what I expect?", then the answer is slightly more subtle.
What I think is happening (and the bash rules are intricate, so I'm not 100% positive here) is that
The line $1 is parsed as a one-word 'simple command' (see the bash manpage) – the one word is just $1, and the && in the expansion isn't visible (and in particular doesn't do what you expected), because the parameter hasn't been expanded yet.
That simple command is then split into words. Of course, there's only one 'word' here, so this step is trivial.
The parameter is then expanded, to give the word "cd / && ls"
Then "Word Splitting: The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word splitting." (bash manpage)
Then this word is split into four words, "cd", "/", "&&" and "ls"
Then "COMMAND EXECUTION:
After a command has been split into words, if it results in a simple command and an optional list of arguments, the following actions are taken.[...]"
So the command cd is run with the three following words as arguments. That means that, as William Pursell noted in his comment to your question, cd is given three arguments, two of which it ignores.
If you run your script with
./ooo 'echo / && ls'
you'll see this happening, and see the apparently magic && being passed to the echo command as just a normal string.
I am trying to execute a command using Vi or ex to edit a file by deleting the first five lines, replace x with y, remove extra spaces at the end of each line but retain the carraige returns, and remove the last eight lines of the file, then rename the file into a shell script and run the new script from the current script.
This will be something that is scheduled in cron. I have been looking for a simple way to do it using the command line or a Vim script or something.
Any ideas? The format of the input file does not change, just the amount of lines, so I can't specify the line numbers for the last eight lines.
You actually have about half a dozen questions here. Here's an answer for the first five which are probably the ones you'll have the most difficulty solving:
sed -e ':label' -n -e '1d' -e 's/x/y/g' -e 's/[ \t]*$//g' -e '1,9!{P;N;D};N;b label' file.txt > script.sh
Vi is an interactive editor. You probably don't want to use it for something that'll be run by cron. Also, I agree with the comments saying this is probably a bad idea. Be that as it may:
printf 'one\ntwo\nthree\nfour\nfive\necho x \n1\n2\n3\n4\n5\n6\n7\n8\n' \
| sed '1,5d;s/ *$//;s/x/y/' \
| tail -r | sed 1,8d | tail -r \
| sh
Our first sed script does most of the work. We reverse the lines with tail -r, then delete the first 8 lines, then reverse again. That trims off the last 8 lines.
Note that on Linux systems (or any with GNU coreutils), you may also have a tac command which reverse lines, but tail -r is more portable.
Also, the final | sh simply runs the output. If you REALLY want to save this as a script, you can do that by redirecting the output to a file ... but I'll leave at least that to your imagination. Can't do all your scripting for you, can we?! :-)
To edit a file by a script, you could use ed (even if it hard to learn or remember).
You could also use some scripting language (Python, Perl, AWK, Ruby) to achieve your goal.