I have a large text file of ~45 GB, and need to append a line both at the beginning and at the end of the file. What is the quickest way to accomplish this? I don't have access to any big data framework such as Hadoop, etc.
In addition, if I have to also do a string replace in this large file, is there a similar efficient way?
I tried with echo command and it really is taking ages.
You can use this sed to add a line in the beginning:
sed -i '1s/^/your heading line\n/' file
And to the end:
echo "my new line" >> file
Test
$ cat a
hello
bye
$ sed -i '1s/^/new line sample\n/' a
$ cat a
new line sample
hello
bye
$ echo "my new line" >> a
$ cat a
new line sample
hello
bye
my new line
Using sed, you could say:
sed -e '1ifoo' -e '$abar'
and this would insert foo at the top of the file and bar at the end of the file.
$ seq 5 | sed -e '1ifoo' -e '$abar'
foo
1
2
3
4
5
bar
Related
I have a big txt file where I want to add a fasta symbol before every line as a new line. I tried with sed, I can add it before the line but not as a new line.
I have file like this
AAAAAAAAAACA
AAAAAAAAAACTTAT
AAAAAAAAACATGTGACTA
AAAAAAAAACTTATTCTTTTT
AAAAAAAACATGTGACT
And I want something like this
>
AAAAAAAAAACA
>
AAAAAAAAAACTTAT
>
AAAAAAAAACATGTGACTA
>
AAAAAAAAACTTATTCTTTTT
>
AAAAAAAACATGTGACT
Thanks,
you can use the sed command like this:
sed 's/^/>\n/g' file.txt> file2.txt
SED
# If you want to edit the file in-place
sed -i -e 's/^/prefix/' file
# If you want to create a new file
sed -e 's/^/prefix/' file > file.new
How?
simple format is : sed 's/x/y/g' - with this, x will be replaced with y
Use -e to add the script to the command (better to use it always) (More described here (click here))
Use -i for input in the same file
AWK
awk '{print ">\n"$0}' file >> newFile
How?
Simple format : awk '{print $0}' file will print whole lines as it is.
Then just add the prefix you need "xyz\n"\
WHILE LOOP
while read line ;
do
echo -e ">\n$line" ;
done < file
With this you can play around your each lines
If you want to do for particular lines alone use the file input as done < <(cat file | grep "TT")
If you want to add some condition, if line contains... then.. echo "prefix"$line is also possible.
Note: it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS)
Note:
If prefix contains /, you can use any other character not in prefix, or
escape the /, so the sed command becomes
's#^#/opt/workdir#'
# or
's/^/\/opt\/workdir/'
$ awk '$0=">\n"$0' <(echo -e "foo\nbar")
>
foo
>
bar
# change inplace
$ cat file
foo
bar
baz
$ awk -i inplace '$0=">\n"$0' file
$ cat file
>
foo
>
bar
>
baz
I'm sure this question has been answered somewhere, but while searching for it, I can't find my exact scenario. I have two files that I am concatenating into a single file, but I am also adding user input between the two files. The problem is that a newline is being inserted after the first file, the rest of it works as desired.
touch newFile
cat file1 > newFile
echo -n $userInput >> newFile
cat file2 >> newFile
How do I prevent or remove the newline when file1 is added to newFile? If I cat file1 there seems to be a newline added by cat but everything I see about cat says it doesn't do that. If I vim file1 there's not a blank line at the end of the file that would indicate the newline is a part of the file, so either cat is actually adding a newline, or the redirect > is doing it, or echo adds a newline at the beginning of its output, none of which would be desirable in this situation. One solution I saw was to use
cat file1 | tr -d '\n'
but that discards all the newlines in the file, also not desirable. So, to repeat my question:
How do I cat file1 into the new file and add user input without adding the newline between them?
(cat is not a requirement, but I am not familiar with printf, so if that's the solution then please elaborate on its use).
With these inputs:
userInput="Test Test Test"
echo "Line 1
Line 2
Line 3" >file1
echo "Line 4
Line 5
Line 6" >file2
I would do:
printf "%s%s%s" "$(cat file1)" "$userInput" "$(cat file2)" >newfile
The creation of >newfile is equivalent to touch and adding content in your first step. A bit easier to see intent with this.
I get:
$ cat newfile
Line 1
Line 2
Line 3Test Test TestLine 4
Line 5
Line 6
Like all other Unix tools, Vim considers \n a line terminator and not a line separator.
This means that a linefeed after the last piece of text will be considered part of the last line, and will not show an additional blank line.
If there is no trailing linefeed, Vim will instead show [noeol] in the status bar when the file is loaded:
foo
~
~
~
~
~
"file" [noeol] 1L, 3C 1,1 All
^---- Here
So no, the linefeed is definitely part of your file and not being added by bash in any way.
If you want to strip all trailing linefeeds, you can do this as a side effect of command expansion:
printf '%s' "$(<file1)" >> newfile
touch newFile
echo -n "$(cat file1)" > newFile
echo -n $userInput >> newFile
cat file2 >> newFile
That did the trick.
I can grap a specific line from a file using sed. Is there an easy way to take this line or paragraph and insert onto a specific line in another file?
sed -n 1,10p >> foo appends the result to foo, which places it at the bottom. Is there a standard unix tool to insert onto a specific line?
Perhaps you are looking for sed's r command?
sed '123r file.txt' main.txt
inserts the contents of file.txt at line 123 of main.txt, printing everything to standard output.
(If your sed has the -i option, you can make it modify main.txt directly; otherwise, it will not modify its input files.)
If you want to replace the nth line in file foo you can do it with
cp foo foo.tmp
head -n $((n-1)) foo.tmp > foo
echo "newline" >> foo
tail -n +$((n+1)) foo.tmp >> foo
So you take the first n-1 lines with head -n NR, append your new line and then append the rest starting from line n+1 with tail -n +NR.
This might work for you (GNU sed):
sed '123s|.*|sed '\''1,10!d'\'' insert.txt|e' main.txt
How do I copy the first few lines of a giant file and add a line of text at the end of it, using some Linux commands?
The head command can get the first n lines. Variations are:
head -7 file
head -n 7 file
head -7l file
which will get the first 7 lines of the file called "file". The command to use depends on your version of head. Linux will work with the first one.
To append lines to the end of the same file, use:
echo 'first line to add' >> file
echo 'second line to add' >> file
echo 'third line to add' >> file
or:
echo 'first line to add
second line to add
third line to add' >> file
to do it in one hit.
So, tying these two ideas together, if you wanted to get the first 10 lines of the input.txt file to output.txt and append a line with five "=" characters, you could use something like:
( head -10 input.txt ; echo '=====' ) > output.txt
In this case, we do both operations in a sub-shell so as to consolidate the output streams into one, which is then used to create or overwrite the output file.
I am assuming what you are trying to achieve is to insert a line after the first few lines of of a textfile.
head -n10 file.txt >> newfile.txt
echo "your line >> newfile.txt
tail -n +10 file.txt >> newfile.txt
If you don't want to rest of the lines from the file, just skip the tail part.
First few lines: man head.
Append lines: use the >> operator (?) in Bash:
echo 'This goes at the end of the file' >> file
sed -n '1,10p' filename > newfile
echo 'This goes at the end of the file' >> newfile
I want to add a row of headers to an existing CSV file, editing in place. How can I do this?
echo 'one, two, three' > testfile.csv
and I want to end up with
column1, column2, column3
one, two, three
Changing the initial CSV output is out of my hands.
Any standard command will do. The important thing is the file is edited in place, and the line is inserted at the beginning of the file.
To answer your original question, here's how you do it with sed:
sed -i '1icolumn1, column2, column3' testfile.csv
The "1i" command tells sed to go to line 1 and insert the text there.
The -i option causes the file to be edited "in place" and can also take an optional argument to create a backup file, for example
sed -i~ '1icolumn1, column2, column3' testfile.csv
would keep the original file in "testfile.csv~".
This adds custom text at the beginning of your file:
echo 'your_custom_escaped_content' > temp_file.csv
cat testfile.csv >> temp_file.csv
mv temp_file.csv testfile.csv
This doesn't use sed, but using >> will append to a file. For example:
echo 'one, two, three' >> testfile.csv
Edit: To prepend to a file, try something like this:
echo "text"|cat - yourfile > /tmp/out && mv /tmp/out yourfile
I found this through a quick Google search.
As far as I understand, you want to prepend column1, column2, column3 to your existing one, two, three.
I would use ed in place of sed, since sed write on the standard output and not in the file.
The command:
printf '0a\ncolumn1, column2, column3\n.\nw\n' | ed testfile.csv
should do the work.
perl -i is worth taking a look as well.
sed is line based, so I'm not sure why you want to do this with sed. The paradigm is more processing one line at a time( you could also programatically find the # of fields in the CSV and generate your header line with awk) Why not just
echo "c1, c2, ... " >> file
cat testfile.csv >> file
?
Use perl -i, with a command that replaces the beginning of line 1 with what you want to insert (the .bk will have the effect that your original file is backed up):
perl -i.bk -pe 's/^/column1, column2, column3\n/ if($.==1)' testfile.csv
Add a given line at the beginning of a file in two commands:
cat <(echo "blablabla") input_file.txt > tmp_file.txt
mv tmp_file.txt input_file.txt
how to add line inside a file
sed -i -e "48r../../../../folder1/lines_to_add.txt" a.txt
a.txt - the file you want to change 48r is line number of a.txt some
lines are inside lines_to_add.txt file
../../../../scripts3_2d/lines_to_add.txt
-i update a.txt, try without -i before run the code, be careful with new lines,
"keep a newline at the end of lines_to_add.txt"