This question already has answers here:
How to remove carriage return from a variable in shell script
(6 answers)
Closed 1 year ago.
Doing a basic shell script I'm having trouble with getting the length of a file with curl and later dividing it, and the text:
syntax error: invalid token operator (error token is"
keeps appearing.
Simplified code would be:
head=$(curl -sI $dir | awk '/Content-Lenth/{print $2}')
res=$(($head/3))
I've been reading solutions to similar problems and they said removing the \r that the variable is probably having at its end, so I put this in the middle of the two lines, but the problem is still there:
head=${head//\r}
Any idea why this happens? Is it because of an \r character at the end? If it's the case, how to remove it?
I was able to reproduce your issue, which for me was indeed caused by a carriage return at the end of the value of $head. That arises from the HTTP protocol specification: HTTP uses carriage return / newline sequences as line terminators (like DOS and Windows), and awk will not recognize the carriage return as whitespace.
Your attempt to remove the carriage return ...
head=${head//\r}
... does not serve that purpose. The \ is a general escape character to the shell, preserving the literal meaning of whatever character follows. Since the character r has no special significance to the shell, the above code is equivalent to
# Oops, not what you meant:
head=${head//r}
I used this alternative:
# Strip all non-digits:
head=${head//[^0-9]}
, and that resolved the issue for me.
Do also note, however, that
you have misspelled "Length", and
HTTP header names are not case sensitive, but by default, awk regular expressions are. Therefore, your awk command might in some cases fail to match the wanted header even after the spelling is corrected.
Here's a version of your script that addresses all of these issues:
head=$(curl -sI "$dir" | awk 'BEGIN{IGNORECASE=1} /Content-Length/{print $2}')
head=${head//[^0-9]}
res=$(($head/3))
The last two lines could be merged into one, but I have kept them separate for clarity. That might also serve well if you want to use the value of $head for other purposes, too.
Related
I have the following Bash script which loops through the lines of a file:
INFO_FILE=playlist-info-test.txt
line_count=$(wc -l $INFO_FILE | awk '{print $1}')
for ((i=1; i<=$line_count; i++))
do
current_line=$(sed "${i}q;d" $INFO_FILE)
CURRENT_PLAYLIST_ORIG="$current_line"
input_file="$CURRENT_PLAYLIST_ORIG.mp3"
echo $input_file
done
This is a sample of the playlist-info-test.txt file:
Playlist 1
Playlist2
Playlist 3
The output of the script should be as follows:
Playlist 1.mp3
Playlist2.mp3
Playlist 3.mp3
However, I am getting the following output:
.mp3list 1
.mp3list2
.mp3list 3
I have spent a few hours on this and can't understand why the ".mp3" part is being moved to the front of the string. I initially thought it was because of the space in the lines of the input file, but removing the space doesn't make a difference. I also tried using a while loop with read line and the input file redirected into it, but that does not make any difference either.
I copied the playlist-info-test.txt contents and the script, and get the output you expected. Most likely there are non-printable characters in your playlist-info-test.txt or script which are messing up the processing. Check the binary contents of both files using for example xxd -g 1 and look for non-newline (0a) non-printing characters.
Did the file come from Windows? DOS and Windows end their lines with carriage return (hex 0d, sometimes represented as \r) followed by linefeed (hex 0a, sometimes represented as \n). Unix just uses linefeed, and so tends to treat the carriage return as part of the content of the line. In your case, it winds up at the end of the current_line variable, so input_file winds up something like "Playlist 1\r.mp3". When you print this to the terminal, the carriage return makes it go back to the beginning of the line (that's what carriage return means), so it prints as:
Playlist 1
.mp3
...with the ".mp3" printed over the "Play" part, rather than on the next line like I have it above.
Solution: either fix the file (there's a fairly standard dos2unix program that does precisely this), or change your script to strip carriage returns as it reads the file. Actually, I'd recommend a rewrite anyway, since your current use of sed to pick out lines is rather weird and inefficient. In a shell script, the standard way to read through a file line-by-line is to use a loop like while read -r current_line; do [commands here]; done <"$INFO_FILE". There's a possible problem that if any commands inside the loop read from standard input, they'll wind up inhaling part of that file; you can fix that by passing the file over unit 3 rather than standard input. With that fix and a trick to trim carriage returns, here's what it looks like:
INFO_FILE=playlist-info-test.txt
while IFS=$' \t\n\r' read -r current_line <&3; do
CURRENT_PLAYLIST_ORIG="$current_line"
input_file="$CURRENT_PLAYLIST_ORIG.mp3"
echo "$input_file"
done 3<"$INFO_FILE"
(The carriage return trim is done by read -- it always auto-trims leading and trailing whitespace, and setting IFS to $' \t\n\r' tells it to treat spaces, tabs, linefeeds, and carriage returns as whitespace. And since that assignment is a prefix to the read command, it applies only to that one command and you don't have to set IFS back to normal afterward.)
A couple of other recommendations while I'm here: double-quote all variable references (as I did with echo "$input_file" above), and avoid all-caps variable names (there are a bunch with special meanings, and if you accidentally use one of those it can have weird effects). Oh, and try passing your scripts to shellcheck.net -- it's good at spotting common mistakes.
This question already has an answer here:
Prevent "echo" from interpreting backslash escapes
(1 answer)
Closed 4 years ago.
I have a bash script that contains lines:
remote_installer_svc_args="$local_cifs_mount/eset-remote-installer.args"
svc_arg_x86="%SYSTEMROOT%\\$(basename $remote_temp_dir)\\$(basename $INSTALLER_BAT)"
svc_arg_x64="%SYSTEMROOT%\\$(basename $remote_temp_dir)\\$(basename $INSTALLER_BAT)"
echo "$svc_arg_x86" > $remote_installer_svc_args
echo "$svc_arg_x64" >> $remote_installer_svc_args
It should produce a file that looks like this (in notepad++ on windows):
instead the file looks like this:
or in vim:
What is wrong with the script? Because when I copy those lines into bash it works, only if I run the script it does produce those strange characters...
You've run into part of the mess of inconsistent behavior that plagues the echo command. Specifically, some versions of echo (in some modes) interpret escape (backslash) sequences in the string they're asked to print. Others don't. When you ask echo to print %SYSTEMROOT%\era_rd_6HbUKJTR\EraAgentInstaller.bat, it might see the \e part and think it's supposed to convert that to the ASCII escape character.
Note that there are two different characters being called "escape" here: The backslash is used by the shell as an escape character, meaning that it and the characters immediately following it have some special meaning. The ASCII escape, on the other hand, is treated as a special character by the terminal (and vim and some other things) in a somewhat similar manner. Since the ASCII escape is a nonprinting character, when notepad++ and vim have to display it, they show some sort of alternate representation ("ESC" or "^]").
Anyway, since echo is inconsistent about its treatment of the backslash character, it's best to avoid it for strings that might contain backslash. Use printf instead (see "Why is printf better than echo?" on unix.se). It's a little more complicated to use, but not too bad. The main things to realize are that the first argument to printf is a "format" string that's used to control how the rest of the arguments are printed, and that unlike echo it doesn't automatically add a newline to the end.
What you want to use is:
printf '%s\n' "$svc_arg_x86" > $remote_installer_svc_args
printf '%s\n' "$svc_arg_x64" >> $remote_installer_svc_args
Or you can simplify it to:
printf '%s\n' "$svc_arg_x86" "$svc_arg_x64" > $remote_installer_svc_args
That first argument, %s\n, says to print a plain string followed by a newline. Backslash escapes in the format string are always interpreted, but strings formatted with the %s format never have escapes interpreted. Note that in the single-command version, the format string gets applied to each of the other two arguments, so each gets a newline at the end, so each winds up on a separate line in the output file.
The problem
I have multiple property lines in a single string separated by \n like this:
LINES2="Abc1.def=$SOME_VAR\nAbc2.def=SOMETHING_ELSE\n"$LINES
The LINES variable
might contain an undefined set of characters
may be empty. If it is empty, I want to avoid the trailing \n.
I am open for any command line utility (sed, tr, awk, ... you name it).
Tryings
I tried this to no avail
sed -z 's/\\n$//g' <<< $LINES2
I also had no luck with tr, since it does not accept regex.
Idea
There might be an approach to convert the \n to something else. But since $LINES can contain arbitrary characters, this might be dangerous.
Sources
I skim read through the following questions
How can I replace a newline (\n) using sed?
sed with literal string--not input file
Here's one solution:
LINES2="Abc1.def=$SOME_VAR"$'\n'"Abc2.def=SOMETHING_ELSE${LINES:+$'\n'$LINES}"
The syntax ${name:+value} means "insert value if the variable name exists and is not empty." So in this case, it inserts a newline followed by $LINES if $LINES is not empty, which seems to be precisely what you want.
I use $'\n' because "\n" is not a newline character. A more readable solution would be to define a shell variable whose value is a single newline.
It is not necessary to quote strings in shell assignment statements, since the right-hand side of an assignment does not undergo word-splitting nor glob expansion. Not quoting would make it easier to interpolate a $'\n'.
It is not usually advisable to use UPPER-CASE for shell variables because the shell and the OS use upper-case names for their own purposes. Your local variables should normally be lower case names.
So if I were not basing the answer on the command in the question, I would have written:
lines2=Abc1.def=$someVar$'\n'Abc2.def=SOMETHING_ELSE${lines:+$'\n'$lines}
my goal is to sed the 100th line and convert it to a string, then separate the data of the sentence to word
#!/bin/bash
fid=log.txt;
sentence=`expr sed -n '100p' ${fid}`;
for word in $sentence
do
echo $word
done
but apparently this has failed.
expr: syntax error
would somebody please let me know what have I done wrong? previously for number it worked.
The expr does not seem to serve a useful purpose here, and if it did, a sed command would certainly not be a valid or useful thing to pass to it, under most circumstances. You should probably just take it out.
However, the following loop is also problematic. Unquoted variables in shell script are very frequently an error. In this case, you can't quote the thing you pass to the for loop (that would cause the loop to only run once, with the loop variable set to the quoted string) but you also cannot prevent the shell from performing wildcard expansion on the unquoted string. So if the string happened to contain *, the shell will expand that to a list of files in the current directory, for example.
Fortunately, this can all be done in an only slightly more complicated sed script.
sed '100!d;s/[ \t]\+/\n/g;q' "$fid"
That is, if the line number is not 100, delete this line and start over with the next line. Otherwise, we are at line 100; replace runs of whitespace with newlines, (print) and quit.
(The backslash escape codes \t and \n are not universally portable; and \+ for repetition is also an optional extension. I believe there are also sed variants which dislike semicolon as a command separator. Consult your sed manual page, experiment, and if everything else fails, maybe switch to Awk or Perl. Just in case, here is a version which works even on Mac OSX:
sed '100!d
s/[ ][ ]*/\
/g;q' log.txt
The stuff inside the square brackets are a space and a literal tab; in Bash, with default keybindings, type ctrl-V, tab to produce a literal tab.)
Incidentally, this also gets rid of the variable capture antipattern. There are good reasons to capture output to a variable, but if it can be avoided, you often end up with a simpler, more robust and efficient, as well as more idiomatic and elegant script. (I see no reason to put the log file name in a variable, either, in this isolated case; but in a larger script, it might make sense.)
I don't think you need expr command in this case.
expr is used to do calculations. Something like:
expr 1 + 1
Just this one is fine:
sentence=`sed -n '100p' ${fid}`;
#!/bin/bash
fid=log.txt;
sentence=$(sed -n '100p' ${fid});
for word in $sentence
do
echo $word
done
put a dollar sign and parenthesis solve the problem
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 1 year ago.
I saw a bash command sed 's%^.*/%%'
Usually the common syntax for sed is sed 's/pattern/str/g', but in this one it used s%^.* for the s part in the 's/pattern/str/g'.
My questions:
What does s%^.* mean?
What's meaning of %% in the second part of sed 's%^.*/%%'?
The % is an alternative delimiter so that you don't need to escape the forward slash contained in the matching portion.
So if you were to write the same expression with / as a delimiter, it would look like:
sed 's/^.*\///'
which is also kind of difficult to read.
Either way, the expression will look for a forward slash in a line; if there is a forward slash, remove everything up to and including that slash.
the usually used delimiter is / and the usage is sed 's/pattern/str/'.
At times you find that the delimiter is present in the pattern. In such a case you have a choice to either escape the delimiter found in the pattern or to use a different delimiter. In your case a different delimiter % has been used.
The later way is recommended as it keeps your command short and clean.