I use lcov --summary output as an input to some program,
So far i didn't need the --remove option so i only used:
lcov --summary "$my_trace_file"
and the output was of the following form:
Reading tracefile /local/home/......
Summary coverage rate:
lines......: 18.9% (61703 of 327102 lines)
functions..: 4.2% (12308 of 296143 functions)
branches...: no data found
Now i need to filter some test files from the out so I used:
lcov --remove "$my_trace_file" "*src/testfiles/*" alon with other patterns.
However now the output is different:
it composed a lot of lines of the form:
TN:
SF:/local/home/....
FN:21,_ZN3EBS22...
FN:31,_ZNK3EBS22
FN:36,_ZNK3EBS22
FN:41,_ZNK3EBS22
FN:46,_ZNK3EBS22
FN:51,_ZN3EBS22
FN:56,_ZN3EBS22
FN:61,_ZN3EBS22
FN:68,_sdfdsfds
FN:76,_edfsdfd
.
.
.
.
...
and
....
Removing /local/home/.....
Removing /local/home/.....
Summary coverage rate:
lines......: 19.2% (61703 of 320880 lines)
functions..: 17.1% (12308 of 72152 functions)
branches...: no data found
Apparently the first part is for stdout and the other for stderr.
however, the program must get all of it through stdout preferably at the regular form as the --summary without any additional data.
What i essentially asking is a way to tell lcov to use --remove but to print only the summary data, and to print it to stdout.
I tried to use both of the options but i got an error saying they are mutually exclusive.
(I know i can use redirection for stderr, but currently it consumed by some other code that only use stdout)
Update: I have tried to play it a little, just to get how it works and redirected the output with > my_file 2>&1 however i got a weird thing:
The redirection order is got mixed: first a some messages of stdout, then stderr and then stdout again, so what im getting is an unordered result. If i run it on the bash itself, i get the right order (that's, the lines:
lines......: 19.2% (61623 of 320880 lines)
functions..: 17.0% (12301 of 72152 functions)
branches...: no data found
are at the end of the output as opposed to the output file, there those lines are in the middle).
Im not sure if solving this will help me since basically I need to use it all with stout but i'd like to understand why this weird thing happens. Thanks
Related
I'm trying to display the output of an AWS lambda that is being captured in a temporary text file, and I want to remove that file as I display its contents. Right now I'm doing:
... && cat output.json && rm output.json
Is there a clever way to combine those last two commands into one command? My goal is to make the full combined command string as short as possible.
For cases where
it is possible to control the name of the temporary text file.
If file is not used by other code
Possible to pass "/dev/stdout" as the.name of the output
Regarding portability: see stack exchange how portable ... /dev/stdout
POSIX 7 says they are extensions.
Base Definitions,
Section 2.1.1 Requirements:
The system may provide non-standard extensions. These are features not required by POSIX.1-2008 and may include, but are not limited to:
[...]
• Additional character special files with special properties (for example, /dev/stdin, /dev/stdout, and /dev/stderr)
Using the mandatory supported /dev/tty will force output into “current” terminal, making it impossible to pipe the output of the whole command into different program (or log file), or to use the program when there is no connected terminals (cron job, or other automation tools)
No, you cannot easily remove the lines of a file while displaying them. It would be highly inefficient as it would require removing characters from the beginning of a file each time you read a line. Current filesystems are pretty good at truncating lines at the end of a file, but not at the beginning.
A simple but extremely slow method would look like this:
while [ -s output.json ]
do
head -1 output.json
sed -i 1d output.json
done
While this algorithm is plain and simple, you should know that each time you remove the first line with sed -i 1d it will copy the whole content of the file but the first line into a temporary file, resulting in approximately 0.5*n² lines written in total (where n is the number of lines in your file).
In theory you could avoid this by do something like that:
while [ -s output.json ]
do
line=$(head -1 output.json)
printf -- '%s\n' "$line"
fallocate -c -o 0 -l $((${#len}+1)) output.json
done
But this does not account for variable newline characters (namely DOS-formatted newlines) and fallocate does not always work on xfs, among other issues.
Since you are trying to consume a file alongside its creation without leaving a trace of its existence on disk, you are essentially asking for a pipe functionality. In my opinion you should look into how your output.json file is produced and hopefully you can pipe it to a script of your own.
I have a ~300 GB zipped vcf file (.vcf.gz) which contains the genomes of about 700 dogs. I am only interested in a few of these dogs and I do not have enough space to unzip the whole file at this time, although I am in the process of getting a computer to do this. Is it possible to unzip only parts of the file to begin testing my scripts?
I am trying to a specific SNP at a position on a subset of the samples. I have tried using bcftools to no avail: (If anyone can identify what went wrong with that I would also really appreciate it. I created an empty file for the output (722g.990.SNP.INDEL.chrAll.vcf.bgz) but it returns the following error)
bcftools view -f PASS --threads 8 -r chr9:55252802-55252810 -o 722g.990.SNP.INDEL.chrAll.vcf.gz -O z 722g.990.SNP.INDEL.chrAll.vcf.bgz
The output type "722g.990.SNP.INDEL.chrAll.vcf.bgz" not recognised
I am planning on trying awk, but need to unzip the file first. Is it possible to partially unzip it so I can try this?
Double check your command line for bcftools view.
The error message 'The output type "something" is not recognized' is printed by bcftools when you specify an invalid value for the -O (upper-case O) command line option like this -O something. Based on the error message you are getting it seems that you might have put the file name there.
Check that you don't have your input and output file names the wrong way around in your command. Note that the -o (lower-case o) command line option specifies the output file name, and the file name at the end of the command line is the input file name.
Also, you write that you created an empty file for the output. You don't need to do that, bcftools will create the output file.
I don't have that much experience with bcftools but generically If you want to to use awk to manipulate a gzipped file you can pipe to it so as to only unzip the file as needed, you can also pipe the result directly through gzip so it too is compressed e.g.
gzip -cd largeFile.vcf.gz | awk '{ <some awk> }' | gzip -c > newfile.txt.gz
Also zcat is an alias for gzip -cd, -c is input/output to standard out, -d is decompress.
As a side note if you are trying to perform operations on just a part of a large file you may also find the excellent tool less useful it can be used to view your large file loading only the needed parts, the -S option is particularly useful for wide formats with many columns as it stops line wrapping, as is -N for showing line numbers.
less -S largefile.vcf.gz
quit the view with q and g takes you to the top of the file.
I execute a program which print some texts. I redirect the texts to file by using > but I cannot see any texts on the file. For example, if the program prints "Hello" I can see the result on the shell:
$ ./a.out arg
Hello
But after I redirect I cannot get any message hello on shell as well as the redirected file.
$ ./a.out arg > log.txt
(print nothing)
$ cat log.txt
(print nothing)
I have no idea what's going on. Is there someone who knows what's happening here? Or is there someone who suffered similar situation?
OS: Ubuntu 14.10, x86_64 arch, and the program is really chromium-browser rather than ./a.out. I edited its JavaScript engine (v8, which is included in chromium-browser) and I tried to print some logs with lots of texts. I tried to save it by redirection but it doesn't work.
Surely I checked whether > symbol work or not. It works as expected on other programs like echo, ls, and so on.
$ echo hello > hello.txt
$ cat hello.txt
hello
How can the messages just go away? I think it should be printed on stdout (or stderr) or file. But it just goes away when I use > symbol.
It is somewhat common for programs to check isatty(stdout) and display different output based on whether stdout is connected to a terminal or not. For example, ls will display file names in a tabular format if output is to a terminal, but display them strictly one per line otherwise. It does this to make it easy to parse its output when it's part of a pipeline.
Not having looked at Chrome's source code myself, this is speculation, but it's possible Chrome is performing this sort of check and changing its output based on where stdout is redirected to.
Try to use "2>" which should redirect stderr to file
Or you can also try to use "&>" which should redirect everything (stderr and stdout)
See more at http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html
There is a nice example of HTML output from criterion at http://bos.github.com/criterion/.
Which command line option is used to generate this output?
An answer to a related question asserts that this output exits, but it does not seem to show up in the command line options when using --help.
Sorry I didn't get around to your comment-question.
The answer Jedai gives is right - just use -o. For example, here is a line from one of my Makefiles for running benchmarks using defaultMain from Criterion:
./Bench -g -u Bench.csv -o Bench.html -s $(SAMPLES)
Breaking that down, it says:
-g run GC between each sample
-u output CSV data to the given file
-o output HTML data to the given file
-s collect this many samples
Well if you just want html output, then yourBench -o yourReport.html will generate some perfectly reasonable output. If you want to use your own template, look at the templates/report.tpl example in the distribution and use the -t option.
It seems to me that you just pass the template as a command line option, and then it populates it. If the template happens to be an html template, then you've generated html.
See the source here: https://github.com/bos/criterion
I have a script which executes my unit tests using valgrind. Now the script became big, because I have maybe 10 suppression files (one per library), and it is possible that I will have to add more suppressions files.
Now instead of having a line like this :
MEMCHECK_OPTIONS="--tool=memcheck -q -v --num-callers=24 --leak-check=full --show-below-main=no --undef-value-errors=yes --leak-resolution=high --show-reachable=yes --error-limit=no --xml=yes --suppressions=$SUPPRESSION_FILES_DIR/suppression_stdlib.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_cg.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_glut.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_xlib.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_glibc.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_glib.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_qt.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_sdl.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_magick.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_sqlite.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_ld.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_selinux.supp --suppressions=$SUPPRESSION_FILES_DIR/suppression_opengl.supp"
I tried doing like this:
MEMCHECK_OPTIONS="--tool=memcheck -q -v --num-callers=24 --leak-check=full --show-below-main=no --undef-value-errors=yes --leak-resolution=high --show-reachable=yes --error-limit=no --xml=yes --suppressions=$SUPPRESSION_FILES_DIR/*.supp"
but valgrind needs a filename (doesn't accept the asterix).
Since I am doing this in a bash script, can someone tell me what is the easiest way to form that line?
I thought about listing all files in the suppression directory, then iterating over that list, and adding --suppressions= prefix.
EDIT
I forgot to ask. This is what I have so far :
ALL_SUPPRESION_FILES=`ls $SUPPRESSION_FILES_DIR/*.supp`
but I can not find how to transfer that into an array. Can someone help?
Just do it this way:
# form the list of suppression files to pass to the valgrind
VALGRIND_SUPPRESSION_FILES_LIST=""
for SUPPRESSION_FILE in $SUPPRESSION_FILES_DIR/*.supp; do
VALGRIND_SUPPRESSION_FILES_LIST+=" --suppressions=$SUPPRESSION_FILE"
done
There's no need for ls.
Here's a way to do it without a loop:
array=($SUPPRESSION_FILES_DIR/*.supp)
VALGRIND_SUPPRESSION_FILES_LIST=${array[#]/#/--suppressions=}
Neither of these work properly if filenames contain spaces, but additional steps can take care of that.
For those who still facing this problem - have a look at Valgrind Suppression File Howto.
When valgrind runs its default tool, Memcheck, it automatically tries to read a file called $PREFIX/lib/valgrind/default.supp ($PREFIX will normally be /usr). However you can make it use additional suppression files of your choice by adding --suppressions= to your command-line invocation. You can repeat this up to 100 times, which should be sufficient for most situations ;)
Rather than having to type this each time, it's more sensible to write it to an rc file. Each time it runs, valgrind looks for options in files called ~/.valgrindrc and ./.valgrindrc. [...]
Create the files if they don't already exist. So I now have a ~/.valgrindrc containing:
--memcheck:leak-check=full
--show-reachable=yes
--suppressions=/file/path/file1.supp
--suppressions=/file/path/file2.suppth/file2.supp
To check that valgrind is actually using the suppression files, run it with the -v option. The list of suppression files read is near the beginning of the output.
Well, I managed to solve the issue this way :
# form the list of suppression files to pass to the valgrind
ALL_SUPPRESION_FILES=`ls $SUPPRESSION_FILES_DIR/*.supp`
VALGRIND_SUPPRESSION_FILES_LIST=""
for SUPPRESSION_FILE in ${ALL_SUPPRESION_FILES[#]}; do
VALGRIND_SUPPRESSION_FILES_LIST="$VALGRIND_SUPPRESSION_FILES_LIST --suppressions=$SUPPRESSION_FILE"
done
I used tokenizing strings and concanating strings to form the list.