I execute a program which print some texts. I redirect the texts to file by using > but I cannot see any texts on the file. For example, if the program prints "Hello" I can see the result on the shell:
$ ./a.out arg
Hello
But after I redirect I cannot get any message hello on shell as well as the redirected file.
$ ./a.out arg > log.txt
(print nothing)
$ cat log.txt
(print nothing)
I have no idea what's going on. Is there someone who knows what's happening here? Or is there someone who suffered similar situation?
OS: Ubuntu 14.10, x86_64 arch, and the program is really chromium-browser rather than ./a.out. I edited its JavaScript engine (v8, which is included in chromium-browser) and I tried to print some logs with lots of texts. I tried to save it by redirection but it doesn't work.
Surely I checked whether > symbol work or not. It works as expected on other programs like echo, ls, and so on.
$ echo hello > hello.txt
$ cat hello.txt
hello
How can the messages just go away? I think it should be printed on stdout (or stderr) or file. But it just goes away when I use > symbol.
It is somewhat common for programs to check isatty(stdout) and display different output based on whether stdout is connected to a terminal or not. For example, ls will display file names in a tabular format if output is to a terminal, but display them strictly one per line otherwise. It does this to make it easy to parse its output when it's part of a pipeline.
Not having looked at Chrome's source code myself, this is speculation, but it's possible Chrome is performing this sort of check and changing its output based on where stdout is redirected to.
Try to use "2>" which should redirect stderr to file
Or you can also try to use "&>" which should redirect everything (stderr and stdout)
See more at http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html
Related
I use lcov --summary output as an input to some program,
So far i didn't need the --remove option so i only used:
lcov --summary "$my_trace_file"
and the output was of the following form:
Reading tracefile /local/home/......
Summary coverage rate:
lines......: 18.9% (61703 of 327102 lines)
functions..: 4.2% (12308 of 296143 functions)
branches...: no data found
Now i need to filter some test files from the out so I used:
lcov --remove "$my_trace_file" "*src/testfiles/*" alon with other patterns.
However now the output is different:
it composed a lot of lines of the form:
TN:
SF:/local/home/....
FN:21,_ZN3EBS22...
FN:31,_ZNK3EBS22
FN:36,_ZNK3EBS22
FN:41,_ZNK3EBS22
FN:46,_ZNK3EBS22
FN:51,_ZN3EBS22
FN:56,_ZN3EBS22
FN:61,_ZN3EBS22
FN:68,_sdfdsfds
FN:76,_edfsdfd
.
.
.
.
...
and
....
Removing /local/home/.....
Removing /local/home/.....
Summary coverage rate:
lines......: 19.2% (61703 of 320880 lines)
functions..: 17.1% (12308 of 72152 functions)
branches...: no data found
Apparently the first part is for stdout and the other for stderr.
however, the program must get all of it through stdout preferably at the regular form as the --summary without any additional data.
What i essentially asking is a way to tell lcov to use --remove but to print only the summary data, and to print it to stdout.
I tried to use both of the options but i got an error saying they are mutually exclusive.
(I know i can use redirection for stderr, but currently it consumed by some other code that only use stdout)
Update: I have tried to play it a little, just to get how it works and redirected the output with > my_file 2>&1 however i got a weird thing:
The redirection order is got mixed: first a some messages of stdout, then stderr and then stdout again, so what im getting is an unordered result. If i run it on the bash itself, i get the right order (that's, the lines:
lines......: 19.2% (61623 of 320880 lines)
functions..: 17.0% (12301 of 72152 functions)
branches...: no data found
are at the end of the output as opposed to the output file, there those lines are in the middle).
Im not sure if solving this will help me since basically I need to use it all with stout but i'd like to understand why this weird thing happens. Thanks
I have a executable binary on Ubuntu 12.04 64bit, which i uploaded to http://te.ngok.in/InitSDX
I am wondering what kind of output/input it is using. I need to log all of its output to a file, but simply running
./InitSDX seed > log.txt
does not populate any text. But when I do
./InitSDX > log.txt
It does show show error, and logged into the file.
Another method, ./InitSDX seed | less show nothing, but ./IniSDX | less, show the error.
Note, simply running ./InitSDX seed outputs some text on screen.
Can anyone please point something here? What/which output stream should I use and how? Thank you.
UPDATE:
I've tried InitSDX seed > log.txt > 2>&1 still put no text to the file.
Most probably ./InitSDK outputs to stderr. You can redirect it to stdout with:
./InitSDK seed > log.txt 2>&1
You can redirect only stderr to file with:
./InitSDK seed 2> error_log.txt
I'm trying to compile a haskell file that has a HUGE number of errors in it. I want to start debugging the first one but unfortunately there are so many that they go off the screen.
I want to pipe the error messages to a file so that I can read from the top, but normal methods don't seem to work.
I've tried:
ghc File.hs > errors.log
ghc File.hs >> errors.log
ghc File.hs | more
None of them work. Using > and >> only writes the first couple of lines to the file and then the rest to standard output. Using more, less, cat etc doesn't make any difference at all.
Is there a flag for GHC that will let me output to a file?
(I should probably let you know that I'm working on a Windows machine with Cygwin.)
Most programs write output to the standard output (file descriptor 1) and error messages to the standard error (file descriptor 2).
You can ask your shell to redirect the standard error to another location like this:
ghc File.hs > output.log 2> errors.log
or if you want them in the same file:
ghc File.hs > output.log 2>&1
See your shell's manpage section of redirections for full details. Note that the shells are picky about the order of the redirections.
You can also view the output directly, using the same redirect as sarnold's solution, but without the intermediate output file:
ghc File.hs 2>&1 | less
(same goes for more instead of less, etc.)
If I use a command like this one:
./program >> a.txt &
, and the program is a long running one then I can only see the output once the program ended. That means I have no way of knowing if the computation is going well until it actually stops computing. I want to be able to read the redirected output on file while the program is running.
This is similar to opening a file, appending to it, then closing it back after every writing. If the file is only closed at the end of the program then no data can be read on it until the program ends. The only redirection I know is similar to closing the file at the end of the program.
You can test it with this little python script. The language doesn't matter. Any program that writes to standard output has the same problem.
l = range(0,100000)
for i in l:
if i%1000==0:
print i
for j in l:
s = i + j
One can run this with:
./python program.py >> a.txt &
Then cat a.txt .. you will only get results once the script is done computing.
From the stdout manual page:
The stream stderr is unbuffered.
The stream stdout is line-buffered
when it points to a terminal.
Partial lines will not appear until
fflush(3) or exit(3) is called, or
a new‐line is printed.
Bottom line: Unless the output is a terminal, your program will have its standard output in fully buffered mode by default. This essentially means that it will output data in large-ish blocks, rather than line-by-line, let alone character-by-character.
Ways to work around this:
Fix your program: If you need real-time output, you need to fix your program. In C you can use fflush(stdout) after each output statement, or setvbuf() to change the buffering mode of the standard output. For Python there is sys.stdout.flush() of even some of the suggestions here.
Use a utility that can record from a PTY, rather than outright stdout redirections. GNU Screen can do this for you:
screen -d -m -L python test.py
would be a start. This will log the output of your program to a file called screenlog.0 (or similar) in your current directory with a default delay of 10 seconds, and you can use screen to connect to the session where your command is running to provide input or terminate it. The delay and the name of the logfile can be changed in a configuration file or manually once you connect to the background session.
EDIT:
On most Linux system there is a third workaround: You can use the LD_PRELOAD variable and a preloaded library to override select functions of the C library and use them to set the stdout buffering mode when those functions are called by your program. This method may work, but it has a number of disadvantages:
It won't work at all on static executables
It's fragile and rather ugly.
It won't work at all with SUID executables - the dynamic loader will refuse to read the LD_PRELOAD variable when loading such executables for security reasons.
It's fragile and rather ugly.
It requires that you find and override a library function that is called by your program after it initially sets the stdout buffering mode and preferably before any output. getenv() is a good choice for many programs, but not all. You may have to override common I/O functions such as printf() or fwrite() - if push comes to shove you may just have to override all functions that control the buffering mode and introduce a special condition for stdout.
It's fragile and rather ugly.
It's hard to ensure that there are no unwelcome side-effects. To do this right you'd have to ensure that only stdout is affected and that your overrides will not crash the rest of the program if e.g. stdout is closed.
Did I mention that it's fragile and rather ugly?
That said, the process it relatively simple. You put in a C file, e.g. linebufferedstdout.c the replacement functions:
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>
char *getenv(const char *s) {
static char *(*getenv_real)(const char *s) = NULL;
if (getenv_real == NULL) {
getenv_real = dlsym(RTLD_NEXT, "getenv");
setlinebuf(stdout);
}
return getenv_real(s);
}
Then you compile that file as a shared object:
gcc -O2 -o linebufferedstdout.so -fpic -shared linebufferedstdout.c -ldl -lc
Then you set the LD_PRELOAD variable to load it along with your program:
$ LD_PRELOAD=./linebufferedstdout.so python test.py | tee -a test.out
0
1000
2000
3000
4000
If you are lucky, your problem will be solved with no unfortunate side-effects.
You can set the LD_PRELOAD library in the shell, if necessary, or even specify that library system-wide (definitely NOT recommended) in /etc/ld.so.preload.
If you're trying to modify the behavior of an existing program try stdbuf (part of coreutils starting with version 7.5 apparently).
This buffers stdout up to a line:
stdbuf -oL command > output
This disables stdout buffering altogether:
stdbuf -o0 command > output
Have you considered piping to tee?
./program | tee a.txt
However, even tee won't work if "program" doesn't write anything to stdout until it is done. So, the effectiveness depends a lot on how your program behaves.
If the program writes to a file, you can read it while it is being written using tail -f a.txt.
Your problem is that most programs check to see if the output is a terminal or not. If the output is a terminal then output is buffered one line at a time (so each line is output as it is generated) but if the output is not a terminal then the output is buffered in larger chunks (4096 bytes at a time is typical) This behaviour is normal behaviour in the C library (when using printf for example) and also in the C++ library (when using cout for example), so any program written in C or C++ will do this.
Most other scripting languages (like perl, python, etc.) are written in C or C++ and so they have exactly the same buffering behaviour.
The answer above (using LD_PRELOAD) can be made to work on perl or python scripts, since the interpreters are themselves written in C.
The unbuffer command from the expect package does exactly what you are looking for.
$ sudo apt-get install expect
$ unbuffer python program.py | cat -
<watch output immediately show up here>
When I run the command haizea -c simulated.conf > result.txt, the program (haizea) still prints its output to the screen. But when I try haizea -c simulated.conf 1>& result.txt, the output is now on the file result.txt. I'm quite confused about this situation. What is the difference between > and 1>&, then?
What you're seeing on the terminal is the standard error of your process. Both of these are directed to the same terminal device by default (assuming no redirection put into effect).
The redirection >&xyz redirects both standard output and error to the file xyz.
I've never used it but I would think, by extension, that N>&xyz would redirect file handle N and standard error to your file. So 1>&xyz is equivalent to >&xyz which is also equivalent to >xyz 2>&1.
The number before the > stands for the descriptor.
Standard Input - 0
Standard Output - 1
Standard Error - 2
The & will direct both standard output and standard error.
http://linuxdevcenter.com/pub/a/linux/lpt/13_01.html#doc2ac15b1c13
> redirects standard output alone.
>& or &> or 1>& redirect both standard output and standard error.
Your program is printing on standard error which is not getting redirected in case 1.