I have a bash script, which when it's finished running, it going to write some files into a directory.
It prints a lot of warnings, which slows down the process, I'm looking for an effective way to prevent printing the warnings in the shell.
I added 2> /dev/null to end of my command at mybash.sh:
#!/bin/bash
command -f file 2> /dev/null
the original command:
java -mx28000M -jar ChromHMM.jar BinarizeBam CHROMSIZES/hg19.txt /Volumes/Control/ /primary_bam_files/FCX/Control/Marks_Run1.txt /Volumes/Control/output1/
When I run my mybash.sh, it will write 20 different output files in a directory.
However, when I use 2>/dev/null, it does not print any warning in the shell which is great but also does not write anything in the output directory which in principle should those 20 files.
Can anyone help me to solve this problem?
Related
What I know and what I've tried: I have a script in R (called GAM.R) that I want to run in the background that outputs .rdata, .pdf, and .jpg files. Running this from the command line is relatively simple:
$ Rscript GAM.R
However, this code takes a very long time to run so I would love to send it to the background and let it work even after I have logged out and turned the computer off. I understand this is pretty easy, as well, and my code would look like this:
$ nohup Rscript GAM.R >/dev/null 2>&1 &
I used this to see if it was working:
$ fg
nohup Rscript GAM.R > /dev/null 2>&1
The problem: I don't know how to check if the code is working (is there a way I can see its progress?) and I don't know where the outputs are going. I can see the progress and output with the first code so I must not be too far off. It doesn't seem that the second code's outputs are going where the first code's outputs went.
Your command line is diverting all output to /dev/null aka, The Bit Bucket.
Consider diverting it to a temporary file:
$ nohup Rscript GAM.R >/tmp/GAM.R.output 2>&1 &
Then you can tail /tmp/GAM.R.output to see the results, it will show the last 10 lines of the file by default. You can use tail -f to show the end of the file, plus new output in real time.
Note, the /tmp/ filesystem is not guaranteed to exist between reboots. You can put the file somewhere else (like ~/GAM.R.output if you need to be sure.
Note, however, that if you turn your computer off, then all processing gets aborted. For this to work your machine must continue to run and not go to sleep, or shutdown.
What you are doing is that with the > you are redirecting the output of your script to /dev/null and by doing 2>&1 you are redirecting the error output to the same place. Finally nohup executes your process and detach it from the current terminal.
So to sum up what you are doing is creating a process and redirecting its output and error output to a file called null that is stored under /dev.
To answer your question I suggest you redirect your outputs to a folder that you can access as normal user and not super user. Then to make sure that everything is ok you can print this file.
For example you can do :
nohup Rscript GAM.R >/home/username/documents/output_file 2>&1 &
and then to see the file from a terminal you can do:
cat /home/username/documents/output_file
Lastly I don't think that your program will keep on running if your turn off your pc and I don't think there is a way to do that.
If you want to run your program in the background and access the output of the program you can easily do that by writing
exec 3< <(Rscript GAM.R)
And then when you wish to check the output of the program run
cat <&3 # or you can use 'cat /dev/fd/3'
Excellent! Thanks everyone for your helpful answers, particularly #Greg Tarsa. Ultimately I needed to use:
$ nohup Rscript GAM.R >/usr/emily/gams/2017_03_14 2>&1 &
The above is used to run the script and save the screen output to emily/gams (called "2017_03_14", a file to be made by the command, not a folder as I had origionally thought). This also outputs my .rdata, .pdf, and .jpg output filesf from the script to usr/emily.
Then I can see progress and running programs using:
$ tail -f 2017_03_14 #Shows the last 10 lines of the program's progress
$ ps #shows your running projects
$ ps -fu emily #see running projects regardless of session, where username==emily
In the spirit of completeness, I can also note here that to cancel a process, you can use:
$ kill -HUP processid #https://kb.iu.edu/d/adqw
I'm using this code to write all terminal's output to the file
exec > >(tee -a "${LOG_FILE}" )
exec 2> >(tee -a "${LOG_FILE}" >&2)
How to tell it to stop? If, for example, I don't want some output to get into log..
Thank you!
It's not completely clear what your goal is here, but here is something to try.
Do you know about the script utility? You can run script myTerminalOutputFile and any commands and output after that will be captured to myTerminalOutputFile. The downside is that it captures everything. You'll see funny screen control chars like ^[[5m;27j. So do a quick test to see if that works for you.
To finish capturing output just type exit and you are returned to you parent shell cmd-line.
Warning: check man script for details about the inherent funkyness of your particular OS's version of script. Some are better than others. script -k myTerminalOutputFile may work better in this case.
IHTH
You may pipe through "sed" and then tee to log file
Is there any way to tell Linux system put all output(stdout,stderr) to a file?
With out using redirection, pipe or modification the how scrips get called.
Just tell the Linux use a file for output.
for example:
script test1.sh:
#!/bin/bash
echo "Testing 123 "
If i run it like "./test1.sh" (with out redirection or pipe)
i'd like to see "Testing 123" in a file (/tmp/linux_output)
Problem: in the system a binary makes a call to a script and this script call many other scrips. it is not possible to modify each call so If i can modify Linux put "output" to a file i can review the logs.
#!/bin/bash
exec >file 2>&1
echo "Testing 123 "
You can read more about exec here
If you are running the program from a terminal, you can use the command script.
It will open up a sub-shell. Do what you need to do.
It will copy all output to the terminal into a file. When you are done, exit the shell. ^D, or exit.
This does not use redirection or pipes.
You could set your terminal's scrollback buffer to a large number of lines and then see all the output from your commands in the buffer - depending on your terminal window and the options in its menus, there may be an option in there to capture terminal I/O to a file.
Your requirement if taken literally is an impractical one, because it is based in a slight misunderstanding. Fundamentally, to get the output to go in a file, you will have to change something to direct it there - which would violate your literal constraint.
But the practical problem is solvable, because unless explicitly counteracted in the child, the output directions configured in a parent process will be inherited. So you only have to setup the redirection once, using either a shell, or a custom launcher program or intermediary. After that it will be inherited.
So, for example:
cat > test.sh
#/bin/sh
echo "hello on stdout"
rm nosuchfile
./test2.sh
And a child script for it to call
cat > test2.sh
#/bin/sh
echo "hello on stdout from script 2"
rm thisfileisnteither
./nonexistantscript.sh
Run the first script redirecting both stdout and stderr (bash version - and you can do this in many ways such as by writing a C program that redirects its outputs then exec()'s your real program)
./test.sh &> logfile
Now examine the file and see results from stdout and stderr of both parent and child.
cat logfile
hello on stdout
rm: nosuchfile: No such file or directory
hello on stdout from script 2
rm: thisfileisnteither: No such file or directory
./test2.sh: line 4: ./nonexistantscript.sh: No such file or directory
Of course if you really dislike this, you can always always modify the kernel - but again, that is changing something (and a very ungainly solution too).
I have made a script that uses a program called Diascope, its a video slideshow program.
I am trying to run this command in encodeVideo.sh
echo "Running diascope -clean /mnt/videos/video$1.txt..."
diascope -clean /mnt/videos/video$1.txt > /var/www/html/video/encodeVideo.log
echo "Removing old deploy dir, and making new one..."
And I am running this script from rc.local so that it runs every time I boot the instance.
All I need is to get the output of the "diascope" command, in rc.local I run encodeVideo:
/bin/bash /var/www/html/video/encodeVideo.sh > /var/www/html/video/newlog
and in newlog I can see this
Running diascope -clean /mnt/videos/video7.txt...
Removing old deploy dir, and making new one...
and /var/www/html/video/encodeVideo.log is completely empty! Diascope uses gawk, and btw I can see the output of the processing when I manually run the encodeVideo.sh script. Why can I not capture it in this log?
Is it possible that it's not standard output, so ">" doesn't actually capture it?
Any ideas would be helpful, thanks!
try redirecting stderr and stdout to filename
diascope -clean /mnt/videos/video$1.txt 1> /var/www/html/video/encodeVideo.log 2>&1
Is it possible that it's not standard output, so ">" doesn't actually capture it?
Absolutely: it could be standard error, in which case you'd have to use 2> instead of >.
I'm starting a node process with the following upstart script, logging stdout & stderr into separate files:
script
sudo -u node /usr/local/bin/node /var/node/services/someServer.js 1> /var/log/node/someServer.log 2> /var/log/node/someServer.error.log
end script
The problem is that both log files have binary data in the head. I can't use less or more to quickly check those logs, which is terribly annoying. Any ideas how I can prevent that process logging binary data?
Try opening using less with the -f and -R options. -f will force open binary files and -R will better handle control characters, if they exist. Does cat display the contents ok?