This question already has answers here:
How can I use a file in a command and redirect output to the same file without truncating it?
(14 answers)
Closed 3 years ago.
I'm testing basic bash command on Ubuntu, when I put on terminal
wc -l < file1.txt > file2.txt
all lines from file1 are read by wc and the number is saved to file2.txt.
However if I use the similar command wc -l < file1.txt > file1.txt but I try to save it in the same > file1 the output is always 0.
What is the reason of this behavior ?
When you write the command wc -l < file1.txt > file1.txt, your shell will first read the command as a whole and redirect the standard input, output from/to file1.txt.
Then when a file is open for writing using >, if a file exists it will be overwritten and an empty file is created. the process (your wc) will then start its execution and it will write in it until the end of the execution.
Your wc -l will however have to read the file and count the number of lines in it, as the file is empty (overwritten) it will just dump in it:
0 file1.txt
To prove this look at what happens in appending mode using >>
$ cat file1.txt
a
b
b
$ wc -l file1.txt
3 file1.txt
$ wc -l file1.txt >> file1.txt
$ cat file1.txt
a
b
b
3 file1.txt
Your file content is intact and wc does properly read the number of lines before appending the result to the file.
Note:
In general, it is never recommended to write and read in the same file except if you know exactly what you are doing. Some commands have an inline option to modify the file during the run of the command and it is highly recommended to use them, as the file will not be corrupted even if the command crash in the middle of the execution.
Its because redirections have higher priority than other commands.
Here the file1 will be emptied before any operations can be performed on it
Try >> on file1.txt instead of >. >> will append to file unlike > overwriting whole file. Try -
wc -l < file1.txt >> file1.txt
Related
I want to copy lines 10 to 15 of a file into another file in Unix.
I am having files file1.txt and file2.txt.
I want to copy lines 10 to 15 from file1.txt to file2.txt.
Open a terminal with a shell then
sed -n '10,15p' file1.txt > file2.txt
Simple & easy.
If you want to append to the end instead of wiping file2.txt, use >> for redirection.
sed -n '10,15p' file1.txt >> file2.txt
^^
AWK is also a powerful command line text manipulator:
awk 'NR>=10 && NR<=15' file1.txt > file2.txt
In complement to the previous answer, you can use one of the following 3 solutions.
sed
Print only the lines in the range and redirect it to the output file
sed -n '10,15p' file1.txt > file2.txt
head/tail combination
Use head and tail to cut the file and to get only the range you need before redirecting the output to a file
head -n 15 file1.txt | tail -n 6 > file2.txt
awk
Print only the lines in the range and redirect it to the output file
awk 'NR>=10 && NR<=15' file1.txt > file2.txt
I am running a script that has been working fine. However, yesterday, I got a couple errors. These errors are after several loops of the script:
sed: cant read file3.txt: No such file or directory
grep: file3.txt: No such file or directory
grep: file3.txt: No such file or directory
sed: cant read file3.txt: No such file or directory
grep: file3.txt: No such file or directory
Keep in mind, these errors do not happen consistently. It's occurring once in a while somewhere near this part of the script. File3.txt is the file not being found:
cat file1.txt | while read LINE; do grep -m 1 $LINE file2.txt >> file3.txt; done
sed -i 's/string//g' file3.txt
grep 'string' file3.txt | cut -d '|' -f1-2 > file4.txt
grep -v 'string' file3.txt | cut -d '|' -f1-2 >> file5.txt
sed -i 's/string//' file3.txt
grep -Fvf file3.txt file1.txt > file6.txt
Now, I'm thinking that since file3.txt is being appended, or later operated on by SED, sometimes the next command starts too soon and it can't find the file? Should I put a wait command in between?
I have looked up many pages with this error, but was unable to find anything:
cat file_name | grep "something" results "cat: grep: No such file or directory" in shell scripting
Pipe multiple commands to a single command with no EOF signal wait
grep command works in command line, but not in bash script: get no such file or directory erro
https://serverfault.com/questions/169539/sed-cant-find-a-file-that-obviously-exists
"No such file or directory" but it exists
If you think that putting a wait or sleep command will help, please let me know. Or, if you think there's a better solution, that would be great too. I'm running on Cygwin terminal. Any insight is greatly appreciated.
Instead of redirecting to file3.txt inside the while loop, redirect the whole loop. Then the file will be created even if the loop never runs because the input file is empty.
while read LINE; do
grep -m 1 $LINE file2.txt
done < file1.txt > file3.txt
If file1.txt is ever empty then file3.txt won't be created.
Also do grep -m 1 $LINE file2.txt will cause problems if there are crucial characters (space is the easiest of them).
Let's assume that the $LINE variable contains more than one word separated by spaces: hello world.
Now the command looks like this: grep -m 1 hello world file2.txt - grep interpretation will look something like this: let's find all hello in file named world and file named file2.txt in current folder.
Using "$LINE" instead of $LINE will lead you to a whole different scenario.
Look at the difference between the following two:
grep -m 1 $LINE file2.txt
grep -m 1 "$LINE" file2.txt
I am looking to compare file1.txt contents with the last n file contents of file2.txt Can someone help in identifying this logic using anything in shell script.
Example if file1.txt has 10 lines, the last 10 lines of file2.txt should be compared for difference.
With bash's process substitution <() and command substitution $().
diff file1.txt <(tail -n $(wc -l < file1.txt) file2.txt)
I'm trying to understand a shell code which includes a line like this:
grep -n data file1.txt > file2.txt
Where data is the text i want to search for.
What does this command mean?
You can have a detailled answer here: http://explainshell.com/explain?cmd=%20grep%20-n%20data%20file1.txt%20%3E%20file2.txt
To sum it up:
grep will look for the string data in file1.txt and will output both the matching lines and their line number (because of the -n flag).
You could read the manual (man grep) to have a better understanding of what grep does.
The output will be redirected into file2.txt; that's what > is used for
I have two files: file1 and file2. How do I append the contents of file2 to file1 so that contents of file1 persist the process?
Use bash builtin redirection (tldp):
cat file2 >> file1
cat file2 >> file1
The >> operator appends the output to the named file or creates the named file if it does not exist.
cat file1 file2 > file3
This concatenates two or more files to one. You can have as many source files as you need. For example,
cat *.txt >> newfile.txt
Update 20130902
In the comments eumiro suggests "don't try cat file1 file2 > file1." The reason this might not result in the expected outcome is that the file receiving the redirect is prepared before the command to the left of the > is executed. In this case, first file1 is truncated to zero length and opened for output, then the cat command attempts to concatenate the now zero-length file plus the contents of file2 into file1. The result is that the original contents of file1 are lost and in its place is a copy of file2 which probably isn't what was expected.
Update 20160919
In the comments tpartee suggests linking to backing information/sources. For an authoritative reference, I direct the kind reader to the sh man page at linuxcommand.org which states:
Before a command is executed, its input and output may be redirected
using a special notation interpreted by the shell.
While that does tell the reader what they need to know it is easy to miss if you aren't looking for it and parsing the statement word by word. The most important word here being 'before'. The redirection is completed (or fails) before the command is executed.
In the example case of cat file1 file2 > file1 the shell performs the redirection first so that the I/O handles are in place in the environment in which the command will be executed before it is executed.
A friendlier version in which the redirection precedence is covered at length can be found at Ian Allen's web site in the form of Linux courseware. His I/O Redirection Notes page has much to say on the topic, including the observation that redirection works even without a command. Passing this to the shell:
$ >out
...creates an empty file named out. The shell first sets up the I/O redirection, then looks for a command, finds none, and completes the operation.
Note: if you need to use sudo, do this:
sudo bash -c 'cat file2 >> file1'
The usual method of simply prepending sudo to the command will fail, since the privilege escalation doesn't carry over into the output redirection.
Try this command:
cat file2 >> file1
Just for reference, using ddrescue provides an interruptible way of achieving the task if, for example, you have large files and the need to pause and then carry on at some later point:
ddrescue -o $(wc --bytes file1 | awk '{ print $1 }') file2 file1 logfile
The logfile is the important bit. You can interrupt the process with Ctrl-C and resume it by specifying the exact same command again and ddrescue will read logfile and resume from where it left off. The -o A flag tells ddrescue to start from byte A in the output file (file1). So wc --bytes file1 | awk '{ print $1 }' just extracts the size of file1 in bytes (you can just paste in the output from ls if you like).
As pointed out by ngks in the comments, the downside is that ddrescue will probably not be installed by default, so you will have to install it manually. The other complication is that there are two versions of ddrescue which might be in your repositories: see this askubuntu question for more info. The version you want is the GNU ddrescue, and on Debian-based systems is the package named gddrescue:
sudo apt install gddrescue
For other distros check your package management system for the GNU version of ddrescue.
Another solution:
tee < file1 -a file2
tee has the benefit that you can append to as many files as you like, for example:
tee < file1 -a file2 file3 file3
will append the contents of file1 to file2, file3 and file4.
From the man page:
-a, --append
append to the given FILEs, do not overwrite
Zsh specific: You can also do this without cat, though honestly cat is more readable:
>> file1 < file2
The >> appends STDIN to file1 and the < dumps file2 to STDIN.
cat can be the easy solution but that become very slow when we concat large files, find -print is to rescue you, though you have to use cat once.
amey#xps ~/work/python/tmp $ ls -lhtr
total 969M
-rw-r--r-- 1 amey amey 485M May 24 23:54 bigFile2.txt
-rw-r--r-- 1 amey amey 485M May 24 23:55 bigFile1.txt
amey#xps ~/work/python/tmp $ time cat bigFile1.txt bigFile2.txt >> out.txt
real 0m3.084s
user 0m0.012s
sys 0m2.308s
amey#xps ~/work/python/tmp $ time find . -maxdepth 1 -type f -name 'bigFile*' -print0 | xargs -0 cat -- > outFile1
real 0m2.516s
user 0m0.028s
sys 0m2.204s