Sometimes when I type a command in bash, I mistakenly type in the \ character at the end, as it is close to the Enter key.
Whenever I do this, I get a prompt on the next line, like this:
>_
The same output is produced when the ` character is used.
What exactly does this \ do to the command?
Are there other characters (besides \ and `) that give a similar output?
the \ character allows you to break your command into multiple lines :
$ grep "hello" /tmp/file
is equivalent to :
$ grep "hello" \
> /tmp/file
the ' and " character allows you to define multiline strings, and the ` is a way to use the output of a command as an argument to another. $(command) does the same thing.
whenever you see
>
it means that the command syntax is not complete. Some shell constructs also needs to be terminated, like while, for, if ...
The displayed > can be configured with the PS2 environnement variable.
as requested, here is an example using ` :
suppose i have a list of files into filelist.txt:
$ cat filelist.txt
a.c
a.h
Makefile
test.cfg
[...]
i want to know the number of lines in each of those files. the command would be wc -l a.c a.h Makefile [...]. to use the output of the cat filelist.txt as arguments to wc -l, i can use :
$ wc -l `
> cat filelist.txt
> `
It may be because you forgot to close the ` or ' or ".
\ is the line continuation character. When at the end of a line, the next line is considered a continuation of the current line.
` is a backtick. Backticks come in pairs, and bash allows contained newlines in pretty much any of the quotes/brackets. You'll see similar (line continuation) behavior with " and ' as well as () and {}.
Related
I have the following code snippet, I believe that the $env_vars variable isn't what I expect it to be.
Is there a program / script that I could substitute docker with, that would then print the entire command line to terminal? So I can debug what's wrong with my command line and then substitute docker back.
docker run \
-v $filePath:/script \
-v $uuid:/secrets \
$env_vars \
$containerImage \
bash /script/entry.sh
So I could have something like
someotherexe run \
-v $filePath:/script \
-v $uuid:/secrets \
$env_vars \
$containerImage \
bash /script/entry.sh
and it would print
someotherexe run -v somepath:/script -v uuid:/secrets etc...
You can replace docker with echo:
echo run -v somepath:/script -v uuid:/secrets etc...
Printing arguments with echo can be misleading, since it just mashes its arguments together with spaces in between. What if one of the arguments contains spaces, or nonprinting characters?
The common way to do this is to put set -x before the command, which makes the shell print its favorite (sometimes cryptic) representation of what the command and its arguments are as it executes it.
I also sometimes use something like this:
printargs() {
if [ $# -eq 0 ]; then
echo "printargs did not get any arguments"
else
echo "printargs got $# argument(s):"
printf " '%s'\n" "$#" | LC_ALL=C cat -vt
fi
}
This prints nonprinting and non-ASCII characters in weird-but-visible formats. For example:
carriagereturn=$'\r'
tab=$'\t'
$ echo –x "spaces and control characters$tab$carriagereturn"
–x spaces and control characters
$ printargs –x "spaces and control characters$tab$carriagereturn"
printargs got 3 argument(s):
'M-bM-^#M-^Sx'
'spaces and control characters^I^M'
What's happening here: for one thing, with the printargs version it's clear which spaces are part of arguments, and which are separators between arguments. The "–x" has a unicode en-dash instead of the plain ASCII hyphen that command-like tools recognize as indicating options; the cat -vt part converts its UTF-8 representation into a series of "meta" (M-something) characters. The carriage return and tab are ASCII control characters, specifically control-I and control-M, so it prints them using ^ as shorthand for "control-".
I implemented the following command to create a file called * (asterisk):
echo > '*'
Now I'm supposed to remove this file without using any quotations.
I know how to remove this by using quotations, but not sure how without using quotations.
I tried the following commands which I was sure that they won't work because of command line expansion:
rm ./*
rm /*
If someone can help me with this, I would greatly appreciate it.
I think you're supposed to work this out yourself :-)
The simplest solution not involving quoting is to use the pattern [*]. Bracket expressions in a shell work much like character classes in regular expressions so that will match a file whose name is the single character *. Thus, you can delete your file with
rm [*]
Note that you cannot use that pattern to create a file named * because the shell substitutes words containing patterns with the name(s) of the files which match the pattern; if no such file exists, then the pattern is not matched and no substitution is performed. So if there is no file named *, then touch [*] will create a file named [*].
You could use history expansion. If the rm command directly follows the echo command, you can use !$:
echo > '*'
rm !$
!$ is shorthand for !!:$: repeat the last word ($) of the last command (!!).
If there are commands between the echo and the rm command, you can find the history number using fc -l:
$ echo > '*'
$ cmd1
$ cmd2
$ cmd3
$ fc -l
[...]
27628 echo > '*'
27629 cmd1
27630 cmd2
27631 cmd3
$ rm !27628:$
!27628 expands to the command with that number in the history, and $ is again the last word of that command.
If you have to run this in a script, you can't really look up the command number and insert it, but you can count the number of commands between the echo and the rm and use a relative event designator:
echo > '*'
cmd1
rm !-2:$
where !-2 refers to the command two lines back. Notice that history expansion is by default disabled in non-interactive shells; use
shopt -o history
to enable it.
You could use rm -i * if the number of files is not too big. This will ask for confirmation for every single file. Confirm deletion only for the file * and reject it for all others.
Can someone please explain the following sed command?
title=$(wget -q -O - https://twitter.com/intent/user?user_id=$ID | sed -n 's/^.*<title>\(.*\) on Twitter<.title>.*$/\1/p')
printf "%s\n" "$title"
I tried (and failed terribly) to recreate it because I thought I understood what was going on in the code. So I wrote (well, more modded) it to be the following:
data-user-id=$(wget -q -O - https://twitter.com/$Username | sed -n 's/^.*"data-user-id">\([^<]*\)<.*$/\1/p')
printf "%s\n" "$data-user-id"
Obviously it errored because the syntax is wrong or something. But I'm trying to understand what is going on so I can make my own variant of it.
P.S. I can't just use the API for this due to how everything needs to be configured.
Give a try to this:
wget -q -O - https://twitter.com/"${Username}" | sed -n '/data-screen-name=.'"${Username}"'".*data-user-id=/I {s/^.*data-screen-name=.'"${Username}"'".*data-user-id="\([0-9]*\)".*$/\1/Ip;q}'
128700677
data-user-id is present in several lines, so it is needed to select a line where data-screen-name=Username
sed is using regular expression, there are 2 good tutorials to start with:
Regular Expressions
Sed - An Introduction and Tutorial by Bruce Barnett
A different sed script with a different output:
Username="StackOverflow"
wget -q -O - https://twitter.com/"${Username}" | sed -n '/data-screen-name=.'"${Username}"'".*data-user-id=/I {p;q}'
data-screen-name="StackOverflow" data-name="Stack Overflow" data-user-id="128700677"
-n instructs sed to not print anything, except when p command is used.
. means any char.
* applies to the previous char in the regex and it means zero or any number of this char.
.* means zero or any number of any char.
/data-screen-name=.'"${Username}"'".*data-user-id=/ select lines which contains data-screen-name= and any one char (.) and StackOverflow and " char and zero or any number of any char (.*) and data-user-id=.
/I means ignore case.
{p;q} are commands executed when above regex is true.
p prints the current line.
q exits the sed script.
The first sed script at the top contains an additional s/regex/replacement/ to clean up the line.
The additional elements used:
^ means the start of the line.
\( ... \) are used to define a group.
"\([0-9]*\)" is a group made of only digits, surrended with 2 " which are not part of the group. It is the first group found in the regex, so it can be referenced in the replacement part with \1.
Assuming the title of the page is "foo on Twitter", it extracts "foo" from it.
But use XMLStarlet instead, since it allows you to specify XPath to extract the data instead of having to poke around with regular expressions.
I was trying to do these few operations/commands on a single line and assign it to a variable. I have it working about 90% of the way except for one part of it.
I was unaware you could do this, but I read that you can nest $(..) inside other $(..).... So I was trying to do that to get this working, but can't seem to get it the rest of the way.
So basically, what I want to do is:
1. Take the output of a file and assign it to a variable
2. Then pre-pend some text to the start of that output
3. Then append some text to the end of the output
4. And finally remove newlines and replace them with "\n" character...
I can do this just fine in multiple steps but I would like to try and get this working this way.
So far I have tried the following:
My 1st attempt, before reading about nested $(..):
MY_VAR=$(echo -n "<pre style=\"display:inline;\">"; cat input.txt | sed ':a;N;$!ba;s/\n/\\n/g'; echo -n "</pre>")
This one worked 99% of the way except there was a newline being added between the cat command's output and the last echo command. I'm guessing this is from the cat command since sed removed all newlines except for that one, maybe...?
Other tries:
MY_VAR=$( $(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo -n "</pre>") | sed ':a;N;$!ba;s/\n/\\n/g')
MY_VAR="$( echo $(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo "</pre>") | sed ':a;N;$!ba;s/\n/\\n/g' )"
MY_VAR="$( echo "$(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo "</pre>")" | sed ':a;N;$!ba;s/\n/\\n/g' )"
*Most these others were tried with and without the extra double-quotes surrounding the different $(..) parts...
I had a few other attempts, but they didn't have any luck either... On a few of the other attempts above, it seemed to work except sed was NOT inserting the replacement part of it. The output was correct for the most part, except instead of seeing "\n" between lines it just showed each of the lines smashed together into one line without anything to separate them...
I'm thinking there is something small I am missing here if anyone has any idea..?
*P.S. Does Bash have a name for the $(..) structure? It's hard trying to Google for that since it doesn't really search symbols...
You have no need to nest command substitutions here.
your_var='<pre style="display:inline;">'"$(<input.txt)"'</pre>'
your_var=${your_var//$'\n'/'\n'}
"$(<input.txt)" expands to the contents of input.txt, but without any trailing newline. (Command substitution always strips trailing newlines; printf '%s' "$(cat ...)" has the same effect, albeit less efficiently as it requires a subshell, whereas cat ... alone does not).
${foo//bar/baz} expands to the contents of the shell variable named foo, with all instances of bar replaced with baz.
$'\n' is bash syntax for a literal newline.
'\n' is bash syntax for a two-character string, beginning with a backslash.
Thus, tying all this together, it first generates a single string with the prefix, the contents of the file, and the suffix; then replaces literal newlines inside that combined string with '\n' two-character sequences.
Granted, this is multiple lines as implemented above -- but it's also much faster and more efficient than anything involving a command substitution.
However, if you really want a single, nested command substitution, you can do that:
your_var=$(printf '%s' '<pre style="display:inline;">' \
"$(sed '$ ! s/$/\\n/g' <input.txt | tr -d '\n')" \
'</pre>')
The printf %s combines its arguments without any delimiter between them
The sed operation adds a literal \n to the end of each line except the last
The tr -d '\n' operation removes literal newlines from the file
However, even this approach could be done more efficiently without the nesting:
printf -v your_var '%s' '<pre style="display:inline;">' \
"$(sed '$ ! s/$/\\n/g' <input.txt | tr -d '\n')" \
'</pre>')
...which has the printf assign its results directly to your_var, without any outer command substitution required (and thus saving the expense of the outer subshell).
I'd like to be able to provide a long list of arbitrary/different commands (varying binary/executable and arguments) and have xargs run those commands in parallel (xargs -P).
I can use xargs -P fine when only varying arguments. It's when I want to vary the executable and arguments that I'm having difficulty.
Example: command-list.txt
% cat command-list.txt
binary_1 arg_A arg_B arg_C
binary_2 arg_D arg_E
.... <lines deleted for brevity>
binary_100 arg_AAA arg_BBB
% xargs -a command-list.txt -P 4 -L 1
** I know the above command will only echo my command-list.txt **
I am aware of GNU parallel but can only use xargs for now. I also can't just background all the commands since there could be too many for the host to handle at once.
Solution is probably staring me in the face. Thanks in advance!
If you don't have access to parallel, one solution is just to use sh with your command as the parameter.
For example:
xargs -a command-list.txt -P 4 -I COMMAND sh -c "COMMAND"
The -c for sh basically just executes the string given (instead of looking for a file). The man page explanation is:
-c string If the -c option is present, then commands are read from
string. If there are arguments after the string, they are
assigned to the positional parameters, starting with $0.
And the -I for xargs tells it to run one command at a time (like -L 1) and to search and replace the parameter (COMMAND in this case) with the current line being processed by xargs. Man page info is below:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with
names read from standard input. Also, unquoted blanks do not
terminate input items; instead the separator is the newline
character. Implies -x and -L 1.
sh seems to be very forgiving with commands containing quotations marks (") so you don't appear to need to regexp them into escaped quotations.