So to put it less confusing:
I run a command which prints some formatted values in the bash e.g.:
NodeID (lot of whitespace) Heap_size (again) Time
And when i try to save the output with Name:~$ script > file.txt, the output is:
ESC[93mnode_s1aESC[0m^MESC[25C1.0g
Expected output:
node_s1a 1.0g ...
node_s2aaaaa 2.0g ...
Is there a way to save raw output with the formatting into a text file ?
You can use the printf command which is like the printf function in C or Java.
printf "%-20s%s" Name:~$ script >> test.txt
I'm assuming Name:~$ and script are variables because I've never seen them before.
Related
Can someone fix this for me.
It should copy a version log file to backup after moving to a repo directory
Then it automatically appends line given as input to the log file with some formatting.
That's it.
Assume existence of log file and test directory.
#!/bin/bash
cd ~/Git/test
cp versionlog.MD .versionlog.MD.old
LOGDATE="$(date --utc +%m-%d-%Y)"
read -p "MSG > " VHMSG |
VHENTRY="- **${LOGDATE}** | ${VHMSG}"
cat ${VHENTRY} >> versionlog.MD
shell output
virufac#box:~/Git/test$ ~/.logvh.sh
MSG > testing script
EOF
EOL]
EOL
e
E
CTRL + C to get out of stuck in reading lines of input
virufac#box:~/Git/test$ cat versionlog.MD
directly outputs the markdown
# Version Log
## version 0.0.1 established 01-22-2020
*Working Towards Working Mission 1 Demo in 0.1 *
- **01-22-2020** | discovered faker.Faker and deprecated old namelessgen
EOF
EOL]
EOL
e
E
I finally got it to save the damned input lines to the file instead of just echoing the command I wanted to enter on the screen and not executing it. But... why isn't it adding the lines built from the VHENTRY variable... and why doesn't it stop reading after one line sometimes and this time not. You could see I was trying to do something to tell it to stop reading the input.
After some realizing a thing I had done in the script was by accident... I tried to fix it and saw that the | at the end of the read command was seemingly the only reason the script did any of what it did save to the file in the first place.
I would have done this in python3 if I had know this script wouldn't be the simplest thing I had ever done. Now I just have to know how you do it after all the time spent on it so that I can remember never to think a shell script will save time again.
Use printf to write a string to a file. cat tries to read from a file named in the argument list. And when the argument is - it means to read from standard input until EOF. So your script is hanging because it's waiting for you to type all the input.
Don't put quotes around the path when it starts with ~, as the quotes make it a literal instead of expanding to the home directory.
Get rid of | at the end of the read line. read doesn't write anything to stdout, so there's nothing to pipe to the following command.
There isn't really any need for the VHENTRY variable, you can do that formatting in the printf argument.
#!/bin/bash
cd ~/Git/test
cp versionlog.MD .versionlog.MD.old
LOGDATE="$(date --utc +%m-%d-%Y)"
read -p "MSG > " VHMSG
printf -- '- **%s** | %s\n' "${LOGDATE}" "$VHMSG" >> versionlog.MD
I saw the line data=$(cat) in a bash script (just declaring an empty variable) and am mystified as to what that could possibly do.
I read the man pages, but it doesn't have an example or explanation of this. Does this capture stdin or something? Any documentation on this?
EDIT: Specifically how the heck does doing data=$(cat) allow for it to run this hook script?
#!/bin/bash
# Runs all executable pre-commit-* hooks and exits after,
# if any of them was not successful.
#
# Based on
# http://osdir.com/ml/git/2009-01/msg00308.html
data=$(cat)
exitcodes=()
hookname=`basename $0`
# Run each hook, passing through STDIN and storing the exit code.
# We don't want to bail at the first failure, as the user might
# then bypass the hooks without knowing about additional issues.
for hook in $GIT_DIR/hooks/$hookname-*; do
test -x "$hook" || continue
echo "$data" | "$hook"
exitcodes+=($?)
done
https://github.com/henrik/dotfiles/blob/master/git_template/hooks/pre-commit
cat will catenate its input to its output.
In the context of the variable capture you posted, the effect is to assign the statement's (or containing script's) standard input to the variable.
The command substitution $(command) will return the command's output; the assignment will assign the substituted string to the variable; and in the absence of a file name argument, cat will read and print standard input.
The Git hook script you found this in captures the commit data from standard input so that it can be repeatedly piped to each hook script separately. You only get one copy of standard input, so if you need it multiple times, you need to capture it somehow. (I would use a temporary file, and quote all file name variables properly; but keeping the data in a variable is certainly okay, especially if you only expect fairly small amounts of input.)
Doing:
t#t:~# temp=$(cat)
hello how
are you?
t#t:~# echo $temp
hello how are you?
(A single Controld on the line by itself following "are you?" terminates the input.)
As manual says
cat - concatenate files and print on the standard output
Also
cat Copy standard input to standard output.
here, cat will concatenate your STDIN into a single string and assign it to variable temp.
Say your bash script script.sh is:
#!/bin/bash
data=$(cat)
Then, the following commands will store the string STR in the variable data:
echo STR | bash script.sh
bash script.sh < <(echo STR)
bash script.sh <<< STR
What I'm trying to accomplish is having a central configuration file, in bash, that defines some variables that are re-used in different bash files. The example below attempts to generate a file name with the current date included in the file name as well as a variable defined in another shell script. However whenever I try to concatenate this external variable it doesn't work. I can concatenate the variable in any other situation.
Example Code:
../config/vars.sh
#!/bin/bash
mysqlUser="backupuser"
mysqlPwd="fakePwd"
mysqlSocket="/var/run/mysqld/mysqld.sock"
mysqlPort="3306"
serverName="s01.catchyservername.com"
./dbBackup.sh
#!/bin/bash
source ../config/vars.sh
tempName=$(date +"%Y%m%d.sql.gz")
fileName="mysqld_${mysqlPort}_${tempName}"
echo "mysqld_${mysqlPort}"
echo ${tempName}
echo ${fileName}
output of dbBackup.sh
mysqld_3306
20140926.sql.gz
_20140926.sql.gz
As you can see when echoing "mysqld_${mysqlPort}" I get the expected output, but when echoing ${fileName} the entire first half of the string is ignored. What am I misunderstanding?
Your vars.sh file was probably created with a DOS/windows text editor:
$ ./dbBackup.sh
mysqld_3306
20140926.sql.gz
_20140926.sql.gz
$ dos2unix vars.sh
dos2unix: converting file vars.sh to Unix format ...
$
$ ./dbBackup.sh
mysqld_3306
20140926.sql.gz
mysqld_3306_20140926.sql.gz
$
As you can see above, I use the dos2unix utility to convert the line separators to Unix style.
I would like to integrate a few short scripts into one script where I can update an argument for the input file from the command line. I am going through 22 files and counting lines where $5!="1".
Here is a sample head of the input file:
Currently, I have the following 3 short scripts:
CHROM POS N_ALLELES N_CHR {FREQ}
2 45895 2 162 0.993827 0.00617284
2 45953 2 162 0.993827 0.00617284
2 264985 2 162 1 0
2 272051 2 162 0.944444 0.0555556
1) count lines (saved as wcYRI.sh): $5!="1"{sum++}END{print sum}
2) apply linecount (saved as check-annos.sh): awk -f wcYRI.sh ~/folder$1/file$1
3) apply linecount for 22 files, sum the output:
for i in {1..22};
do sh check-annos.sh $i; done
| awk '{sum+=$1}END{print sum}'
Its relatively simple, but sometimes script 1 gets a little longer for data files that look like this:
Chr Start End Ref Alt Func.refGene Gene.refGene ExonicFunc.refGene AAChange.refGene LJB2_SIFT LJB2_PolyPhen2_HDIV LJB2_PP2_HDIV_Pred LJB2_PolyPhen2_HVAR LJB2_PolyPhen2_HVAR_Pred LJB2_LRT LJB2_LRT_Pred LJB2_MutationTaster LJB2_MutationTaster_Pred LJB_MutationAssessor LJB_MutationAssessor_Pred LJB2_FATHMM LJB2_GERP++ LJB2_PhyloP LJB2_SiPhy
16 101593 101593 C T exonic POLR3K nonsynonymous SNV POLR3K:NM_016310:exon2:c.G164A:p.G55E 0.000000 0.997 D 0.913 D 0.000000 D 0.999989 D 2.205 medium 0.99 5.3 2.477000 17.524
...and I am using an awk file like this (performing an array match) as input -f to script 2 above:
NR==FNR{
arr[$1$2];next
}
$1$2 in arr && $0~/exonic/&&/nonsynonymous SNV/{nonsyn++};
$1$2 in arr && $0~/exonic/&&/synonymous SNV/ && $0!~/nonsynonymous/{syn++}
END{
print nonsyn,"nonsyn YRI","\t",syn,"YRI syn"
}
My goal is to integrate this process a bit more so I don't need to go into script 2 and change the ~/folder$1/file$1 each time-- I'd like to be able to use ~/folder$1/file$1 as an input at the command line. However when I try to use something like this in a for-loop at the command line, it doesn't accept $1 the way it does when $1 is built into a separate script being called by the for-do-done loop (as in script 3 --i.e. script 3 will take script 2, but I can't just enter the contents of script 2 explicitly into the for-loop as an argument(s)).
I am actually not so concerned about having a separate AWK file to handle the line parsing, the main thing annoying me is that I am modifying script 2 for each folder/file set, and I would like to be able to do this from the command line so that the script knows when I tell it ~/folder$1/file$1, to cycle through numbers 1-22 and I so can save one universal script for this process, since I have many folder/file combinations to look at.
Any advice is appreciated for shortening the pipeline in general, but specifically the command line argument problem is bugging me a lot!
If I understand the problem correctly, I see two ways to handle it. If the path format is consistent (i.e. the number always occurs twice, in the same positions), you could make the script accept the parts of the path as two different parameters. The script would look like this:
#!/bin/bash
folderPrefix="$1"
filePrefix="$2"
for num in {1..22}; do
awk -f wcYRI.sh "$folderPrefix$num/$filePrefix$num"
done |
awk '{sum+=$1}END{print sum}'
... and then you'd run it with ./scriptname ~/folder file. Alternately, if you need to be able to define the folder/file path format more flexibly, you could do something like this:
#!/bin/bash
for num in {1..22}; do
eval "awk -f wcYRI.sh $1"
done |
awk '{sum+=$1}END{print sum}'
... and then run it with ./scriptname '~/folder$num/file$num'. Note that the single-quotes are needed here so that the $var references don't get expanded until eval forces them to be.
BTW, the file wcYRI.sh is an awk script, not a shell script, so I'd recommend changing its file extension to prevent confusion. Actually, the preferred way to do this (for both shell and awk scripts) is to add a shebang line as the first line in the script (see my examples above; for an awk script it would be #!/usr/bin/awk -f), then make the script executable, and then run it with just ./scriptname and let the shebang take care of specifying the interpreter (sh, bash, awk -f, whatever).
I have a test script which has a lot of commands and will generate lots of output, I use set -x or set -v and set -e, so the script would stop when error occurs. However, it's still rather difficult for me to locate which line did the execution stop in order to locate the problem.
Is there a method which can output the line number of the script before each line is executed?
Or output the line number before the command exhibition generated by set -x?
Or any method which can deal with my script line location problem would be a great help.
Thanks.
You mention that you're already using -x. The variable PS4 denotes the value is the prompt printed before the command line is echoed when the -x option is set and defaults to : followed by space.
You can change PS4 to emit the LINENO (The line number in the script or shell function currently executing).
For example, if your script reads:
$ cat script
foo=10
echo ${foo}
echo $((2 + 2))
Executing it thus would print line numbers:
$ PS4='Line ${LINENO}: ' bash -x script
Line 1: foo=10
Line 2: echo 10
10
Line 3: echo 4
4
http://wiki.bash-hackers.org/scripting/debuggingtips gives the ultimate PS4 that would output everything you will possibly need for tracing:
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
In Bash, $LINENO contains the line number where the script currently executing.
If you need to know the line number where the function was called, try $BASH_LINENO. Note that this variable is an array.
For example:
#!/bin/bash
function log() {
echo "LINENO: ${LINENO}"
echo "BASH_LINENO: ${BASH_LINENO[*]}"
}
function foo() {
log "$#"
}
foo "$#"
See here for details of Bash variables.
PS4 with value $LINENO is what you need,
E.g. Following script (myScript.sh):
#!/bin/bash -xv
PS4='${LINENO}: '
echo "Hello"
echo "World"
Output would be:
./myScript.sh
+echo Hello
3 : Hello
+echo World
4 : World
Workaround for shells without LINENO
In a fairly sophisticated script I wouldn't like to see all line numbers; rather I would like to be in control of the output.
Define a function
echo_line_no () {
grep -n "$1" $0 | sed "s/echo_line_no//"
# grep the line(s) containing input $1 with line numbers
# replace the function name with nothing
} # echo_line_no
Use it with quotes like
echo_line_no "this is a simple comment with a line number"
Output is
16 "this is a simple comment with a line number"
if the number of this line in the source file is 16.
This basically answers the question How to show line number when executing bash script for users of ash or other shells without LINENO.
Anything more to add?
Sure. Why do you need this? How do you work with this? What can you do with this? Is this simple approach really sufficient or useful? Why do you want to tinker with this at all?
Want to know more? Read reflections on debugging
Simple (but powerful) solution: Place echo around the code you think that causes the problem and move the echo line by line until the messages does not appear anymore on screen - because the script has stop because of an error before.
Even more powerful solution: Install bashdb the bash debugger and debug the script line by line
If you're using $LINENO within a function, it will cache the first occurrence. Instead use ${BASH_LINENO[0]}