How do I prepend the data from file1.txt to file2.txt?
The following command will take the two files and merge them into one
cat file1.txt file2.txt > file3.txt; mv file3.txt file2.txt
You can do this in a pipeline using sponge from moreutils:
cat file1.txt file2.txt | sponge file2.txt
Another way using GNU sed:
sed -i -e '1rfile1.txt' -e '1{h;d}' -e '2{x;G}' file2.txt
That is:
On line 1, append the content of the file file1.txt
On line 1, copy pattern space to hold space, and delete pattern space
On line 2, exchange the content of the hold and pattern spaces, and append the hold space to pattern space
The reason it's a bit tricky is that the r command appends content,
and line 0 is not addressable, so we have to do it on line 1,
moving the content of the original line out of the way and then bringing it back after the content of the file is appended.
If it's available on your system, then sponge from moreutils is designed for this. Here is an example:
cat file1.txt file2.txt | sponge file2.txt
If you don't have sponge, then the following script does the same job using a temporary file. It makes sure that the temporary file is not accessible by other users, and cleans it up at the end.
If your system, or the script crashes, you may need to clean up the temporary file manually. Tested on Bash 4.4.23, and Debian 10 (Buster) Gnu/Linux.
#!/bin/bash
#
# ----------------------------------------------------------------------------------------------------------------------
# usage [ from, to ]
# [ from, to ]
# ----------------------------------------------------------------------------------------------------------------------
# Purpose:
# Prepend the contents of file [from], to file [to], leaving the result in file [to].
# ----------------------------------------------------------------------------------------------------------------------
# check
[[ $# -ne 2 ]] && echo "[exit]: two filenames are required" >&2 && exit 1
# init
from="$1"
to="$2"
tmp_fn=$( mktemp -t TEMP_FILE_prepend.XXXXXXXX )
chmod 600 "$tmp_fn"
# prepend
cat "$from" "$to" > "$tmp_fn"
mv "$tmp_fn" "$to"
# cleanup
rm -f "$tmp_fn"
# [End]
The way of writing file is like 1). append at the end of the file or 2). rewrite that file.
If you want to put the content in file1.txt ahead of file2.txt, I'm afraid you need to rewrite the combined fine.
Related
I have a file which looks like this (file1.txt)
258.2222
I have to write this file1.txt value to another file. if there in no value in file1.txt then
it should print as "Passed".
this is what I tried
for final in $(cat file1.txt);do
if [ "$final" ];then
echo $final > file2.txt
else
echo "Passed" > file2.txt
fi
done
this only works with 1 scenario. if there is no value in file1.txt then it is not writing as "Passed"
expected output:
if there is a value in file1.txt:
258.2222
if there is no value (empty) in file1.txt:
Passed
Can someone help me to figure out this? Thanks in advance!
Note: I am not allowed to use general purpose scripting language (JavaScript, Python etc).
If file1.txt contains something (its file size is non-zero), you want that something to go into file2.txt:
if [ -s file1.txt ]; then
cp file1.txt file2.txt
fi
If file1.txt is empty (or missing), then you want a line saying Passed written to file2.txt:
if [ -s file1.txt ]; then
cp file1.txt file2.txt
else
echo Passed >file2.txt
fi
If you want to avoid creating file2.txt if file1.txt is missing:
if [ -s file1.txt ]; then
cp file1.txt file2.txt
elif [ -e file1.txt ]; then
echo Passed >file2.txt
fi
The echo is now only executed if the -s test fails (the file does not exist, or it is empty) and -e test succeeds (the file exists).
At no point do you actually have to read the data from the file in a loop.
The line for final in $(cat file1.txt); do is always of questionable value. It will munge whitespace, and is generally a bad idea. Sometimes it has a place, but for the most part there are better ways to iterate over the contents of a file. In your case, this is causing an issue because when the file is empty the loop is executed zero times.
It's not clear to me if you are trying to parse the value in some way, or filter it from other content. If you just want to copy the full contents of the file or write "Passed" if the file is empty, you can do:
if test -s file1.txt; then
cat file1.txt
else
echo Passed
fi > file2.txt
or
if ! grep '[^ ]' file1.txt; then echo Passed; fi > file2.txt
The test command in the first case will succeed (return 0) if file1.txt exists and has some content. Note that this may simply be whitespace; if the file having non-zero size is not sufficient for your needs, you may want to use the second solution where grep only succeeds if it matches at least one line that has a non-space character (which may be a tab!). (But note that this grep will filter out lines that are only space!). If the test or the grep returns non-zero (eg, the file is empty or there are no lines that contain non-whitespace), then "Passed" is written. Either way, the output is redirected to file2.txt.
that is mean file1.txt is empty
Then I would harness GNU AWK for this task following way
awk 'END{print NR?$0:"Passed"}' file1.txt > file2.txt
Explanation: I use so called ternary operator condition?valueiftrue:valueiffalse for condition I use just NR (number row) which inside END is total number of rows processed, if file1.txt has one or more rows I print $0 which inside END is last line of file, otherwise i.e. where there is 0 rows in file I print text Passed.
(tested in gawk 4.2.1)
Using any awk:
awk '{print} END{if (!NR) print "Passed"}' file1.txt > file2.txt
I want to generate a text file with the list of files present in the folder
ls | xargs echo > text.txt
I want to prepend the IP address to each file so that I can run parallel wget as per this post : Parallel wget in Bash
So my text.txt file content will have these lines :
123.123.123.123/file1
123.123.123.123/file2
123.123.123.123/file3
How can I append a string as the ls feeds xargs? (and also add line break at the end.)
Thank you
Simply printf and globbing to get the filenames:
printf '123.123.123.123/%s\n' * >file.txt
Or longer approach, leverage a for construct with help from globbing:
for f in *; do echo "123.123.123.123/$f"; done >file.txt
Assuming no filename with newline exists.
So far I've been able to find out how to add a line at the beginning of a file but that's not exactly what I want. I'll show it with an example:
File content
some text at the beginning
Result
<added text> some text at the beginning
It's similar but I don't want to create any new line with it...
I would like to do this with sed if possible.
sed can operate on an address:
$ sed -i '1s/^/<added text> /' file
What is this magical 1s you see on every answer here? Line addressing!.
Want to add <added text> on the first 10 lines?
$ sed -i '1,10s/^/<added text> /' file
Or you can use Command Grouping:
$ { echo -n '<added text> '; cat file; } >file.new
$ mv file{.new,}
If you want to add a line at the beginning of a file, you need to add \n at the end of the string in the best solution above.
The best solution will add the string, but with the string, it will not add a line at the end of a file.
sed -i '1s/^/your text\n/' file
If the file is only one line, you can use:
sed 's/^/insert this /' oldfile > newfile
If it's more than one line. one of:
sed '1s/^/insert this /' oldfile > newfile
sed '1,1s/^/insert this /' oldfile > newfile
I've included the latter so that you know how to do ranges of lines. Both of these "replace" the start line marker on their affected lines with the text you want to insert. You can also (assuming your sed is modern enough) use:
sed -i 'whatever command you choose' filename
to do in-place editing.
Use subshell:
echo "$(echo -n 'hello'; cat filename)" > filename
Unfortunately, command substitution will remove newlines at the end of file. So as to keep them one can use:
echo -n "hello" | cat - filename > /tmp/filename.tmp
mv /tmp/filename.tmp filename
Neither grouping nor command substitution is needed.
To insert just a newline:
sed '1i\\'
You can use cat -
printf '%s' "some text at the beginning" | cat - filename
To add a line to the top of the file:
sed -i '1iText to add\'
my two cents:
sed -i '1i /path/of/file.sh' filename
This will work even is the string containing forward slash "/"
Hi with carriage return:
sed -i '1s/^/your text\n/' file
Note that on OS X, sed -i <pattern> file, fails. However, if you provide a backup extension, sed -i old <pattern> file, then file is modified in place while file.old is created. You can then delete file.old in your script.
There is a very easy way:
echo "your header" > headerFile.txt
cat yourFile >> headerFile.txt
PROBLEM: tag a file, at the top of the file, with the base name of the parent directory.
I.e., for
/mnt/Vancouver/Programming/file1
tag the top of file1 with Programming.
SOLUTION 1 -- non-empty files:
bn=${PWD##*/} ## bn: basename
sed -i '1s/^/'"$bn"'\n/' <file>
1s places the text at line 1 of the file.
SOLUTION 2 -- empty or non-empty files:
The sed command, above, fails on empty files. Here is a solution, based on https://superuser.com/questions/246837/how-do-i-add-text-to-the-beginning-of-a-file-in-bash/246841#246841
printf "${PWD##*/}\n" | cat - <file> > temp && mv -f temp <file>
Note that the - in the cat command is required (reads standard input: see man cat for more information). Here, I believe, it's needed to take the output of the printf statement (to STDIN), and cat that and the file to temp ... See also the explanation at the bottom of http://www.linfo.org/cat.html.
I also added -f to the mv command, to avoid being asked for confirmations when overwriting files.
To recurse over a directory:
for file in *; do printf "${PWD##*/}\n" | cat - $file > temp && mv -f temp $file; done
Note also that this will break over paths with spaces; there are solutions, elsewhere (e.g. file globbing, or find . -type f ... -type solutions) for those.
ADDENDUM: Re: my last comment, this script will allow you to recurse over directories with spaces in the paths:
#!/bin/bash
## https://stackoverflow.com/questions/4638874/how-to-loop-through-a-directory-recursively-to-delete-files-with-certain-extensi
## To allow spaces in filenames,
## at the top of the script include: IFS=$'\n'; set -f
## at the end of the script include: unset IFS; set +f
IFS=$'\n'; set -f
# ----------------------------------------------------------------------------
# SET PATHS:
IN="/mnt/Vancouver/Programming/data/claws-test/corpus test/"
# https://superuser.com/questions/716001/how-can-i-get-files-with-numeric-names-using-ls-command
# FILES=$(find $IN -type f -regex ".*/[0-9]*") ## recursive; numeric filenames only
FILES=$(find $IN -type f -regex ".*/[0-9 ]*") ## recursive; numeric filenames only (may include spaces)
# echo '$FILES:' ## single-quoted, (literally) prints: $FILES:
# echo "$FILES" ## double-quoted, prints path/, filename (one per line)
# ----------------------------------------------------------------------------
# MAIN LOOP:
for f in $FILES
do
# Tag top of file with basename of current dir:
printf "[top] Tag: ${PWD##*/}\n\n" | cat - $f > temp && mv -f temp $f
# Tag bottom of file with basename of current dir:
printf "\n[bottom] Tag: ${PWD##*/}\n" >> $f
done
unset IFS; set +f
Just for fun, here is a solution using ed which does not have the problem of not working on an empty file. You can put it into a shell script just like any other answer to this question.
ed Test <<EOF
a
.
0i
<added text>
.
1,+1 j
$ g/^$/d
wq
EOF
The above script adds the text to insert to the first line, and then joins the first and second line. To avoid ed exiting on error with an invalid join, it first creates a blank line at the end of the file and remove it later if it still exists.
Limitations: This script does not work if <added text> is exactly equal to a single period.
echo -n "text to insert " ;tac filename.txt| tac > newfilename.txt
The first tac pipes the file backwards (last line first) so the "text to insert" appears last. The 2nd tac wraps it once again so the inserted line is at the beginning and the original file is in its original order.
The simplest solution I found is:
echo -n "<text to add>" | cat - myFile.txt | tee myFile.txt
Notes:
Remove | tee myFile.txt if you don't want to change the file contents.
Remove the -n parameter if you want to append a full line.
Add &> /dev/null to the end if you don't want to see the output (the generated file).
This can be used to append a shebang to the file. Example:
# make it executable (use u+x to allow only current user)
chmod +x cropImage.ts
# append the shebang
echo '#''!'/usr/bin/env ts-node | cat - cropImage.ts | tee cropImage.ts &> /dev/null
# execute it
./cropImage.ts myImage.png
Another solution with aliases. Add to your init rc/ env file:
addtail () { find . -type f ! -path "./.git/*" -exec sh -c "echo $# >> {}" \; }
addhead () { find . -type f ! -path "./.git/*" -exec sh -c "sed -i '1s/^/$#\n/' {}" \; }
Usage:
addtail "string to add at the beginning of file"
addtail "string to add at the end of file"
With the echo approach, if you are on macOS/BSD like me, lose the -n switch that other people suggest. And I like to define a variable for the text.
So it would be like this:
Header="my complex header that may have difficult chars \"like these quotes\" and line breaks \n\n "
{ echo "$Header"; cat "old.txt"; } > "new.txt"
mv new.txt old.txt
TL;dr -
Consider using ex. Since you want the front of a given line, then the syntax is basically the same as what you might find for sed but the option of "in place editing" is built-in.
I cannot imagine an environment where you have sed but not ex/vi, unless it is a MS Windows box with some special "sed.exe", maybe.
sed & grep sort of evolved from ex / vi, so it might be better to say sed syntax is the same as ex.
You can change the line number to something besides #1 or search for a line and change that one.
source=myFile.txt
Front="This goes IN FRONT "
man true > $source
ex -s ${source} <<EOF
1s/^/$Front/
wq
EOF
$ head -n 3 $source
This goes IN FRONT TRUE(1) User Commands TRUE(1)
NAME
Long version, I recommend ex (or ed if you are one of the cool kids).
I like ex because it is portable, extremely powerful, allows me to write in-place, and/or make backups all without needing GNU (or even BSD) extensions.
Additionally, if you know the ex way, then you know how to do it in vi - and probably vim if that is your jam.
Notice that EOF is not quoted when we use "i"nsert and using echo:
str="+++ TOP +++" && ex -s <<EOF
r!man true
1i
`echo "$str"`
.
"0r!echo "${str}"
wq! true.txt
EOF
0r!echo "${str}" might also be used as shorthand for :0read! or :0r! that you have likely used in vi mode (it is literally the same thing) but the : is optional here and some implementations do not support "r"ead address of zero.
"r"eading directly to the special line #0 (or from line 1) would automatically push everything "down", and then you just :wq to save your changes.
$ head -n 3 true.txt | nl -ba
1 +++ TOP +++
2 TRUE(1) User Commands TRUE(1)
3
Also, most classic sed implementations do not have extensions (like \U&) that ex should have by default.
cat concatenates multiple files. <() sends output of a command as a file. Combining these two, we can insert lines at the beginning and end of a file by,
cat <(echo "line before the file") file.txt <(echo "line after the file")
I am using cat *.txt to merge multiple txt files into one, but I need each file to be on a separate line.
What is the best way to merge files with each file appearing on a new line?
just use awk
awk 'FNR==1{print ""}1' *.txt
If you have a paste that supports it,
paste --delimiter=\\n --serial *.txt
does a really great job
You can iterate through each file with a for loop:
for filename in *.txt; do
# each time through the loop, ${filename} will hold the name
# of the next *.txt file. You can then arbitrarily process
# each file
cat "${filename}"
echo
# You can add redirection after the done (which ends the
# for loop). Any output within the for loop will be sent to
# the redirection specified here
done > output_file
for file in *.txt
do
cat "$file"
echo
done > newfile
I'm assuming you want a line break between files.
for file in *.txt
do
cat "$file" >> result
echo >> result
done
This question already has answers here:
Why doesnt "tail" work to truncate log files?
(6 answers)
Closed 1 year ago.
I was trying to remove all the lines of a file except the last line but the following command did not work, although file.txt is not empty.
$cat file.txt |tail -1 > file.txt
$cat file.txt
Why is it so?
Redirecting from a file through a pipeline back to the same file is unsafe; if file.txt is overwritten by the shell when setting up the last stage of the pipeline before tail starts reading off the first stage, you end up with empty output.
Do the following instead:
tail -1 file.txt >file.txt.new && mv file.txt.new file.txt
...well, actually, don't do that in production code; particularly if you're in a security-sensitive environment and running as root, the following is more appropriate:
tempfile="$(mktemp file.txt.XXXXXX)"
chown --reference=file.txt -- "$tempfile"
chmod --reference=file.txt -- "$tempfile"
tail -1 file.txt >"$tempfile" && mv -- "$tempfile" file.txt
Another approach (avoiding temporary files, unless <<< implicitly creates them on your platform) is the following:
lastline="$(tail -1 file.txt)"; cat >file.txt <<<"$lastline"
(The above implementation is bash-specific, but works in cases where echo does not -- such as when the last line contains "--version", for instance).
Finally, one can use sponge from moreutils:
tail -1 file.txt | sponge file.txt
You can use sed to delete all lines but the last from a file:
sed -i '$!d' file
-i tells sed to replace the file in place; otherwise, the result would write to STDOUT.
$ is the address that matches the last line of the file.
d is the delete command. In this case, it is negated by !, so all lines not matching the address will be deleted.
Before 'cat' gets executed, Bash has already opened 'file.txt' for writing, clearing out its contents.
In general, don't write to files you're reading from in the same statement. This can be worked around by writing to a different file, as above:$cat file.txt | tail -1 >anotherfile.txt
$mv anotherfile.txt file.txtor by using a utility like sponge from moreutils:$cat file.txt | tail -1 | sponge file.txt
This works because sponge waits until its input stream has ended before opening its output file.
When you submit your command string to bash, it does the following:
Creates an I/O pipe.
Starts "/usr/bin/tail -1", reading from the pipe, and writing to file.txt.
Starts "/usr/bin/cat file.txt", writing to the pipe.
By the time 'cat' starts reading, 'file.txt' has already been truncated by 'tail'.
That's all part of the design of Unix and the shell environment, and goes back all the way to the original Bourne shell. 'Tis a feature, not a bug.
tmp=$(tail -1 file.txt); echo $tmp > file.txt;
This works nicely in a Linux shell:
replace_with_filter() {
local filename="$1"; shift
local dd_output byte_count filter_status dd_status
dd_output=$("$#" <"$filename" | dd conv=notrunc of="$filename" 2>&1; echo "${PIPESTATUS[#]}")
{ read; read; read -r byte_count _; read filter_status dd_status; } <<<"$dd_output"
(( filter_status > 0 )) && return "$filter_status"
(( dd_status > 0 )) && return "$dd_status"
dd bs=1 seek="$byte_count" if=/dev/null of="$filename"
}
replace_with_filter file.txt tail -1
dd's "notrunc" option is used to write the filtered contents back, in place, while dd is needed again (with a byte count) to actually truncate the file. If the new file size is greater or equal to the old file size, the second dd invocation is not necessary.
The advantages of this over a file copy method are: 1) no additional disk space necessary, 2) faster performance on large files, and 3) pure shell (other than dd).
As Lewis Baumstark says, it doesn't like it that you're writing to the same filename.
This is because the shell opens up "file.txt" and truncates it to do the redirection before "cat file.txt" is run. So, you have to
tail -1 file.txt > file2.txt; mv file2.txt file.txt
echo "$(tail -1 file.txt)" > file.txt
Just for this case it's possible to use cat < file.txt | (rm file.txt; tail -1 > file.txt)
That will open "file.txt" just before connection "cat" with subshell in "(...)". "rm file.txt" will remove reference from disk before subshell will open it for write for "tail", but contents will be still available through opened descriptor which is passed to "cat" until it will close stdin. So you'd better be sure that this command will finish or contents of "file.txt" will be lost
It seems to not like the fact you're writing it back to the same filename. If you do the following it works:
$cat file.txt | tail -1 > anotherfile.txt
tail -1 > file.txt will overwrite your file, causing cat to read an empty file because the re-write will happen before any of the commands in your pipeline are executed.