I have a very long file. The contents of the file are like:
myserver1
kernel_version
os
myserver2
kernel_version
os
myserver3
kernel_version
os
...
There are more than 10.000 entries and 3 entries for each host. Hostname, kernel_version and OS version.
I would like to have an output like:
myserver1, kernel_version, os
myserver2, kernel_version, os
myserver3, kernel_version, os
...
instead. So what is the best awk/sed command to provide this output?
With sed:
$ sed '/^$/d;N;N;s/\n/, /g' infile
myserver1, kernel_version, os
myserver2, kernel_version, os
myserver3, kernel_version, os
This works as follows:
/^$/d # Delete line if empty (skips rest of commands)
N # Append second line to pattern space
N # Append third line to pattern space
s/\n/, /g # Replace newlines by comma and a blank
If you want the criterion for the line to be skipped not be "empty line" but its line number (4, 8, 12...), you can replace the first command (this is a GNU extension):
sed '4~4d;N;N;s/\n/, /g' infile
You can also use paste:
paste -d ',,\0' - - - - <file
You can use :
awk 'BEGIN{RS="";OFS=", "} {print $1,$2,$3}' data.txt
defining record separator as empty line with output field separator (OFS) as ", "
You can also use :
awk 'BEGIN{RS="";OFS=", "} {$1=$1; print $0}' data.txt
$1=$1 forces the record to be reconstituted, see this
While AWK/SED could help you perform this task, a better way would be to use Python, assuming that the *NIX system you are working on has it installed to process this data.
You could use the following in python to process this quite easily:
import csv
output_file = csv.writer(open("/path/to/output/file","w"))
column_num = 3 # number of columns in your end-state data
with open("</path/to/your/input/file>","r") as input:
row = []
iteration_counter = 0
for line in input:
iteration_counter += 1
stripped = line.strip() # to remove the newlines (\n)
if iteration_counter <= column_num:
row.append(stripped)
else:
iteration_counter = 0 # reset the counter to 0
output_writer.writerow(row) # output the list as a csv row
row = [] # clear the row list to nothing
iteration_counter += 1
row.append(stripped)
Related
I have written an awk command
awk 'NR==5 {sub(substr($1,14,1),(substr($1,14,1) + 1)); print "test.py"}' > test.py
This is trying to change the 14th character on the 5th line of a python file. For some reason this doesn't stop executing and I have to break it. It also deletes the contents of the file.
Sample input:
import tools
tools.setup(
name='test',
tagvisc='0.0.8',
packages=tools.ges(),
line xyz
)
`
Output:
import tools
tools.setup(
name='test',
tagvisc='0.0.9',
packages=tools.ges(),
line xyz
)
If I understand the nuances of what you need to do now, you will need to split the first field of the 5th record into an array using "." as the fieldsep and then remove the "\"," from the end of the 3rd element of the array (optional) before incrementing the number and putting the field back together. You can do so with:
awk '{split($1,a,"."); sub(/["],/,"",a[3]); $1=a[1]"."a[2]"."(a[3]+1)"\","}1'
(NR==5 omitted for example)
Example Use/Output
$ echo 'tagvisc="3.4.30"', |
awk '{split($1,a,"."); sub(/["],/,"",a[3]); $1=a[1]"."a[2]"."(a[3]+1)"\","}1'
tagvisc="3.4.31",
I'll leave redirecting to a temp file and then back to the original to you. Let me know if this isn't what you need.
Adding NR == 5 you would have
awk 'NR==5 {split($1,a,"."); sub(/["],/,"",a[3]); $1=a[1]"."a[2]"."(a[3]+1)"\","}1' test.py > tmp; mv -f tmp test.py
Get away from the fixed line number (NR==5) and fixed character position (14) and instead look at dynamically finding what you want to change/increment, eg:
$ cat test.py
import tools
tools.setup(
name='test',
tagvisc='0.0.10',
packages=tools.ges(),
line xyz
)
One awk idea to increment the 10 (3rd line, 3rd numeric string in line):
awk '
/tagvisc=/ { split($0,arr,".") # split line on periods
sub("." arr[3]+0 "\047","." arr[3]+1 "\047") # replace .<oldvalue>\047 with .<newvalue>\047; \047 == single quote
}
1
' test.py
NOTES:
arr[3] = 10',; with arr[3]+0 awk will take the leftmost all-numeric content, strip off everything else, then add 0, leaving us with arr[3] = 10; same logic applies for arr[3]+1 (arr[3]+1 = 11); basically a trick for discarding any suffix that is not numeric
if there are multiple lines in the file with the string tagvisc='x.y.z' then this will change z in all of the lines; we can get around this by adding some more logic to only change the first occurrence, but I'll leave that out for now assuming it's not an issue
This generates:
import tools
tools.setup(
name='test',
tagvisc='0.0.11',
packages=tools.ges(),
line xyz
)
If the objective is to overwrite the original file with the new values you have a couple options:
# use temporary file:
awk '...' test.py > tmp ; mv tmp test.py
# if using GNU awk, and once accuracy of script has been verified:
awk -i inplace '...' test.py
Using awk to make changes to nth character in [mth] line in a file:
$ awk 'BEGIN{FS=OFS=""}NR==5{$18=9}1' file # > tmp && mv tmp file
Outputs:
import tools
tools.setup(
name='test',
tagvisc='0.0.9', <----- this is not output but points to what changed
packages=tools.ges(),
line xyz
)
Explained:
$ awk '
BEGIN {
FS=OFS="" # set the field separators to empty and you can reference
} # each char in record by a number
NR==5 { # 5th record
$18=9 # and 18th char is replaced with a 9
}1' file # > tmp && mv tmp file # output to a tmp file and replace
Notice: Some awks (probably all but GNU awk) will fail if you try to replace a multibyte char by a single byte one (for example utf8 ä (0xc3 0xa4) with an a (0x61) will result in 0x61 0xa4). Naturally an ä before the position you'd like to replace will set your calculations off by 1.
Oh yeah, you can replace one char with multiple chars but not vice versa.
something like this...
$ awk 'function join(a,k,s,sep) {for(k in a) {s=s sep a[k]; sep="."} return s}
BEGIN {FS=OFS="\""}
/^tagvisc=/{v[split($2,v,".")]++; $2=join(v)}1' file > newfile
Using GNU awk for the 3rd arg to match() and "inplace" editing:
$ awk -i inplace '
match($0,/^([[:space:]]*tagvisc=\047)([^\047]+)(.*)/,a) {
split(a[2],ver,".")
$0 = a[1] ver[1] "." ver[2] "." ver[3]+1 a[3]
}
{ print }
' test.py
$ cat test.py
import tools
tools.setup(
name='test',
tagvisc='0.0.9',
packages=tools.ges(),
line xyz
)
I have a FASTA file test.fasta which has the following information:
>QWE2J2_DEFR00000200123 DEFR00000560077.11 DEFR00000100333.7 3:444563-33443(-
)
acccaaagggagggagagagggctattatcatggaaaactaatttttcccagagaatttcctttcaaacctcccagtatc
tatgatcactcccaacgggaggtttaagtgcaacaccaggctgtgtctttctatcacggatttccacccggacacgtgga
acccggcctggtctgtctccaccatcctgactgggctcctgagcttcatggtggagaagggccccaccctgggcagtata
gagacgtcggacttcacgaaaagacaactggcagtgcagagaaaaggggggggggggggggataaagtcttttgtgaatt
atttcctgaagtcgtggaggagattaaacaaaaacagaaagcacaagacgaactcagtagcagaccccagactctcccct
tgccagacgtggttccagaaaaaaaaaaaaacctcgtccagaacgggattcagctgctcaacgggcatgcgccgggggcc
gtcccaaacctcgcagggctccagcaggccaaccggcaccacggactcctgggtggcgccctggcgaacttgtttgtgat
agttgggtttgcagcctttgcttacacggtcaagtaggggggggggggggcgcaggagtg
I need to convert it to CSV in the following format:
>QWE2J2_DEFR00000200123,DEFR00000560077.11,DEFR00000100333.7,3:444563-33443(-),acccaaagggagggagagagggctattatcatggaaaactaatttttcccagagaatttcctttcaaacctcccagtatctatgatcactcccaacgggaggtttaagtgcaacaccaggctgtgtctttctatcacggatttccacccggacacgtggaacccggcctggtctgtctccaccatcctgactgggctcctgagcttcatggtggagaagggccccaccctgggcagtatagagacgtcggacttcacgaaaagacaactggcagtgcagagaaaaggggggggggggggggataaagtcttttgtgaattatttcctgaagtcgtggaggagattaaacaaaaacagaaagcacaagacgaactcagtagcagaccccagactctccccttgccagacgtggttccagaaaaaaaaaaaaacctcgtccagaacgggattcagctgctcaacgggcatgcgccgggggccgtcccaaacctcgcagggctccagcaggccaaccggcaccacggactcctgggtggcgccctggcgaacttgtttgtgatagttgggtttgcagcctttgcttacacggtcaagtaggggggggggggggcgcaggagtg
I have tried in Linux terminal:
input_file=test.fasta; vim -c '0,$s/>\(.*\)\n/>\1,/' -c '0,$s/\(.*\)\n\([^>]\)/\1\2/' -c 'w! my-tmp.fasta.csv' -c 'q!' $input_file; mv my-tmp.fasta.csv $input_file.csv
However, it gives me wrong output:
>QWE2J2_DEFR00000200123 DEFR00000560077.11 DEFR00000100333.7 3:444563-33443(-,)acccaaagggagggagagagggctattatcatggaaaactaatttttcccagagaatttcctttcaaacctcccagtatctatgatcactcccaacgggaggtttaagtgcaacaccaggctgtgtctttctatcacggatttccacccggacacgtggaacccggcctggtctgtctccaccatcctgactgggctcctgagcttcatggtggagaagggccccaccctgggcagtatagagacgtcggacttcacgaaaagacaactggcagtgcagagaaaaggggggggggggggggataaagtcttttgtgaattatttcctgaagtcgtggaggagattaaacaaaaacagaaagcacaagacgaactcagtagcagaccccagactctccccttgccagacgtggttccagaaaaaaaaaaaaacctcgtccagaacgggattcagctgctcaacgggcatgcgccgggggccgtcccaaacctcgcagggctccagcaggccaaccggcaccacggactcctgggtggcgccctggcgaacttgtttgtgatagttgggtttgcagcctttgcttacacggtcaagtaggggggggggggggcgcaggagtg
How can I create this CSV file?
Using awk with RS set to > is just simple:
awk -vRS='>' 'NR>1{
gsub(/ /, ",")
sub(/\)\n/, "),")
gsub("\n", "")
print RS $0
}' file
GNU sed with -z looks simple too:
sed -z '
s/ /,/g
s/)\n/),/g
s/\n//g
s/>/\n>/g
s/^\n//
' file
The following sed script should also work:
sed -n '
# if line does not start with >
/^>/!{
# append the line to hold space
H
# if its not the end of file, start over
$!b
}
# switch pattern space with hold space
x
# add a comma after )
s/)/),/
# remove all the newlines
s/\n//g
# print it all, if hold space not empty
/^$/!p
# switch pattern space with hold space
x
# replace spaces with comma
s/ /,/g
# hold the line
h
' file
Scripts written and tested on repl:
>QWE2J2_DEFR00000200123,DEFR00000560077.11,DEFR00000100333.7,3:444563-33443(-),acccaaagggagggagagagggctattatcatggaaaactaatttttcccagagaatttcctttcaaacctcccagtatcacccggcctggtctgtctccaccatcctgactgggctcctgagcttcatggtggagaagggccccaccctgggcagtataatttcctgaagtcgtggaggagattaaacaaaaacagaaagcacaagacgaactcagtagcagaccccagactctcccctgtcccaaacctcgcagggctccagcaggccaaccggcaccacggactcctgggtggcgccctggcgaacttgtttgtgat
Prefer sed instead of vim.
This is the content of file.txt:
hello bro
my nam§
is Jhon Does
The file could also contain non-printable characters (for example \x00, or \x02), and, as you can see, the lenght of the lines are not the same.
Then I want to read it each each 5 characters without having into a count line breaks. I thought in something like this using awk:
awk -v RS='' '{
s=s $0;
}END{
n=length(s);
for(x=1; x<n; x=x+5){
# Here I will put some calcs and stuff
i++;
print "line " i ": #" substr(s,x,5) "#"
}
}' file.txt
The output is the following:
line 1: #hello#
line 2: # bro
#
line 3: #my na#
line 4: #m§
is#
line 5: # Jhon#
line 6: # Does#
It works perfectly, but the input file will be very large, so the performance is important.
In short, I'm looking for something like this:
awk -v RS='.{5}' '{ # Here I will put some calcs and stuff }'
But it doesn't works.
Another alternative that works ok:
xxd -ps mifile.txt | tr -d '\n' | fold -w 10 | awk '{print "23" $0 "230a"}' | xxd -ps -r
Do you have any idea or alternative? Thank you.
I'm not sure I understand what you want but this outputs the same as the script in your question that you say works perfectly so hopefully this is it:
$ awk -v RS='.{5}' 'RT!=""{ print "line", NR ": #" RT "#" }' file
line 1: #hello#
line 2: # bro
#
line 3: #my na#
line 4: #m§
is#
line 5: # Jhon#
line 6: # Does#
The above uses GNU awk for multi-char RS and RT.
If you are okay with Python, You may try this
f = open('filename', 'r+')
w = f.read(5)
while(w != ''):
print w;
w = f.read(5);
f.close()
You can use perl and binmode assuming you are using normal characters.
use strict;
use warnings;
open my $fh, '<', 'test';
#open the file.
binmode $fh;
# Set to binary mode
$/ = \5;
#Read a record as 5 bytes
while(<$fh>){
#Read records
print "$_#"
#Do whatever calculations you want here
}
For extended character sets you can use UTF8 and read every 5 characters instead of bytes.
use strict;
use warnings;
open my $fh, '<:utf8', 'test';
#open file in utf8.
binmode(STDOUT, ":utf8");
# Set stdout to utf8 as well
while ((read($fh, my $data, 5)) != 0){
#Read 5 characters into variable data
print "$data#";
#Do whatever you want with data here
}
So you asked How to read a file each n characters instead of each line using awk.
Solution:
If you have a modern gawk implementation use FPAT
Normally, when using FS, gawk defines the fields as the parts of the
record that occur in between each field separator. In other words, FS
defines what a field is not, instead of what a field is. However,
there are times when you really want to define the fields by what they
are, and not by what they are not.
Code:
gawk 'BEGIN{FS="\n";RS="";FPAT=".{,5}"}
{for (i=1;i<=NF;i++){
printf("$%d = <%s>\n", i, $i)}
}' file
Check the demo
I have a large data file in text format and I want to convert it to csv by specifying each column length.
number of columns = 5
column length
[4 2 5 1 1]
sample observations:
aasdfh9013512
ajshdj 2445df
Expected Output
aasd,fh,90135,1,2
ajsh,dj, 2445,d,f
GNU awk (gawk) supports this directly with FIELDWIDTHS, e.g.:
gawk '$1=$1' FIELDWIDTHS='4 2 5 1 1' OFS=, infile
Output:
aasd,fh,90135,1,2
ajsh,dj, 2445,d,f
I would use sed and catch the groups with the given length:
$ sed -r 's/^(.{4})(.{2})(.{5})(.{1})(.{1})$/\1,\2,\3,\4,\5/' file
aasd,fh,90135,1,2
ajsh,dj, 2445,d,f
Here's a solution that works with regular awk (does not require gawk).
awk -v OFS=',' '{print substr($0,1,4), substr($0,5,2), substr($0,7,5), substr($0,12,1), substr($0,13,1)}'
It uses awk's substr function to define each field's start position and length. OFS defines what the output field separator is (in this case, a comma).
(Side note: This only works if the source data does not have any commas. If the data has commas, then you have to escape them to be proper CSV, which is beyond the scope of this question.)
Demo:
echo 'aasdfh9013512
ajshdj 2445df' |
awk -v OFS=',' '{print substr($0,1,4), substr($0,5,2), substr($0,7,5), substr($0,12,1), substr($0,13,1)}'
Output:
aasd,fh,90135,1,2
ajsh,dj, 2445,d,f
Adding a Generic way of handling this(alternative to FIELDSWIDTH option) in awk(where we need not to harcode sub string positions, this will work as per position nuber provided by user wherever comma needs to be inserted) could be as follows, written and tested in GNU awk. To use this, we have to define values(like OP showed in samples), position numbers where we need to insert commas, awk variable name is colLength give position numbers with space between them.
awk -v colLengh="4 2 5 1 1" '
BEGIN{
num=split(colLengh,arr,OFS)
}
{
j=sum=0
while(++j<=num){
if(length($0)>sum){
sub("^.{"arr[j]+sum"}","&,")
}
sum+=arr[j]+1
}
}
1
' Input_file
Explanation: Simple explanation would be, creating awk variable named colLengh where we need to define position numbers wherever we need to insert commas. Then in BEGIN section creating array arr which has value of indexes where we need to insert commas in it.
In main program section first of all nullifying variables j and sum here. Then running a while loop from j=1 to till value of j becomes equal to num. In each run substituting from starting of current line(if length of current line is greater than sum else it doesn't make sense to perform substitution to I have put addiotnal check here) everything with everything + , as per need. Eg: sub function will become .{4} for first time loop runs then it becomes, .{7} because its 7th position we need to insert comma and so on. So sub will substitute those many characters from starting to till generated numbers with matched value + ,. At last in this program mentioning 1 will print edited/non-edited lines.
If any one is still looking for a solution, I have developed a small script in python. its easy to use provided you have python 3.5
https://github.com/just10minutes/FixedWidthToDelimited/blob/master/FixedWidthToDelimiter.py
"""
This script will convert Fixed width File into Delimiter File, tried on Python 3.5 only
Sample run: (Order of argument doesnt matter)
python ConvertFixedToDelimiter.py -i SrcFile.txt -o TrgFile.txt -c Config.txt -d "|"
Inputs are as follows
1. Input FIle - Mandatory(Argument -i) - File which has fixed Width data in it
2. Config File - Optional (Argument -c, if not provided will look for Config.txt file on same path, if not present script will not run)
Should have format as
FieldName,fieldLength
eg:
FirstName,10
SecondName,8
Address,30
etc:
3. Output File - Optional (Argument -o, if not provided will be used as InputFIleName plus Delimited.txt)
4. Delimiter - Optional (Argument -d, if not provided default value is "|" (pipe))
"""
from collections import OrderedDict
import argparse
from argparse import ArgumentParser
import os.path
import sys
def slices(s, args):
position = 0
for length in args:
length = int(length)
yield s[position:position + length]
position += length
def extant_file(x):
"""
'Type' for argparse - checks that file exists but does not open.
"""
if not os.path.exists(x):
# Argparse uses the ArgumentTypeError to give a rejection message like:
# error: argument input: x does not exist
raise argparse.ArgumentTypeError("{0} does not exist".format(x))
return x
parser = ArgumentParser(description="Please provide your Inputs as -i InputFile -o OutPutFile -c ConfigFile")
parser.add_argument("-i", dest="InputFile", required=True, help="Provide your Input file name here, if file is on different path than where this script resides then provide full path of the file", metavar="FILE", type=extant_file)
parser.add_argument("-o", dest="OutputFile", required=False, help="Provide your Output file name here, if file is on different path than where this script resides then provide full path of the file", metavar="FILE")
parser.add_argument("-c", dest="ConfigFile", required=False, help="Provide your Config file name here,File should have value as fieldName,fieldLength. if file is on different path than where this script resides then provide full path of the file", metavar="FILE",type=extant_file)
parser.add_argument("-d", dest="Delimiter", required=False, help="Provide the delimiter string you want",metavar="STRING", default="|")
args = parser.parse_args()
#Input file madatory
InputFile = args.InputFile
#Delimiter by default "|"
DELIMITER = args.Delimiter
#Output file checks
if args.OutputFile is None:
OutputFile = str(InputFile) + "Delimited.txt"
print ("Setting Ouput file as "+ OutputFile)
else:
OutputFile = args.OutputFile
#Config file check
if args.ConfigFile is None:
if not os.path.exists("Config.txt"):
print ("There is no Config File provided exiting the script")
sys.exit()
else:
ConfigFile = "Config.txt"
print ("Taking Config.txt file on this path as Default Config File")
else:
ConfigFile = args.ConfigFile
fieldNames = []
fieldLength = []
myvars = OrderedDict()
with open(ConfigFile) as myfile:
for line in myfile:
name, var = line.partition(",")[::2]
myvars[name.strip()] = int(var)
for key,value in myvars.items():
fieldNames.append(key)
fieldLength.append(value)
with open(OutputFile, 'w') as f1:
fieldNames = DELIMITER.join(map(str, fieldNames))
f1.write(fieldNames + "\n")
with open(InputFile, 'r') as f:
for line in f:
rec = (list(slices(line, fieldLength)))
myLine = DELIMITER.join(map(str, rec))
f1.write(myLine + "\n")
Portable awk
Generate an awk script with the appropriate substr commands
cat cols
4
2
5
1
1
<cols awk '{ print "substr($0,"p","$1")"; cs+=$1; p=cs+1 }' p=1
Output:
substr($0,1,4)
substr($0,5,2)
substr($0,7,5)
substr($0,12,1)
substr($0,13,1)
Combine lines and make it a valid awk-script:
<cols awk '{ print "substr($0,"p","$1")"; cs+=$1; p=cs+1 }' p=1 |
paste -sd, | sed 's/^/{ print /; s/$/ }/'
Output:
{ print substr($0,1,4),substr($0,5,2),substr($0,7,5),substr($0,12,1),substr($0,13,1) }
Redirect the above to a file, e.g. /tmp/t.awk and run it on the input-file:
<infile awk -f /tmp/t.awk
Output:
aasd fh 90135 1 2
ajsh dj 2445 d f
Or with comma as the output separator:
<infile awk -f /tmp/t.awk OFS=,
Output:
aasd,fh,90135,1,2
ajsh,dj, 2445,d,f
I have to replace a single nth character of each (alternate) row of a large file with correponding column of another file. For eg I am changing every 5th character.
file1:
>chr1:101842566-101842576
CCTCAACTCA
>chr1:101937281-101937291
GAATTGGATA
>chr1:101964276-101964286
AAAAAATAGG
>chr1:101972950-101972960
ggctctcatg
>chr1:101999969-101999979
CATCATGACG
file2:
G
A
T
A
C
output:
>chr1:101842566-101842576
CCTCGACTCA
>chr1:101937281-101937291
GAATAGGATA
>chr1:101964276-101964286
AAAATATAGG
>chr1:101972950-101972960
ggctAtcatg
>chr1:101999969-101999979
CATCCTGACG
The number of characters in each (alternate) row can be large. And number of rows are large too. How this can be done efficiently?
Here is one way with awk:
awk 'NR==FNR{a[NR]=$1;next}!/^>/{$1=substr($1,1,n-1) a[++i] substr($1,n+1)}1' n=5 f2 f1
Explanation:
We iterate over second file and store it in an array indexed at line number.
Once the second file is loaded in memory, we move to the second file.
We look for lines not starting with >.
When found we substitute the value from our array. We do this by using substr function.
The variable n defined allows you to modify the nth character
For the lines that donot have > we print them as is using 1 which is default for
printing.
This solution assumes the format of the file is as shown above. That is, the first file will always start with > followed by the line you want to make changes on. Substitution from the second file will be made in the order it is seen.
Demo:
Every 5th character:
$ awk 'NR==FNR{a[NR]=$1;next}!/^>/{$1=substr($1,1,n-1) a[++i] substr($1,n+1)}1' n=5 f2 f1
>chr1:101842566-101842576
CCTCGACTCA
>chr1:101937281-101937291
GAATAGGATA
>chr1:101964276-101964286
AAAATATAGG
>chr1:101972950-101972960
ggctAtcatg
>chr1:101999969-101999979
CATCCTGACG
Every 3rd character:
$ awk 'NR==FNR{a[NR]=$1;next}!/^>/{$1=substr($1,1,n-1) a[++i] substr($1,n+1)}1' n=3 f2 f1
>chr1:101842566-101842576
CCGCAACTCA
>chr1:101937281-101937291
GAATTGGATA
>chr1:101964276-101964286
AATAAATAGG
>chr1:101972950-101972960
ggAtctcatg
>chr1:101999969-101999979
CACCATGACG
This is how I would use perl. First read all of file2 into an array, and then iterate over that array reading two and two lines from file1, printing the first line unmodified and then change the 5th character on the second line:
#!/usr/bin/perl
use strict;
use warnings;
use diagnostics;
#use Data::Printer;
# Read all of file2
my $lines;
open(FILE, $ARGV[1]);
{
local $/;
$lines = <FILE>;
}
close(FILE);
my #new_chars = split(/\n/, $lines);
# Read and process file1
open(FILE, $ARGV[0]);
foreach my $new_char (#new_chars) {
# >chr1:101842566-101842576
my $line = <FILE>;
print $line;
# CCTCAACTCA
$line = <FILE>;
$line =~ s/^(....)./$1$new_char/; # Replace 5th character
print $line;
}
close(FILE);
You could replace a column in the file inplace using mmap in Python:
#!/usr/bin/env python3
"""Replace inplace a column of a large file.
Usage:
$ ./replace-inplace file1 file2 5
"""
import sys
from mmap import ACCESS_WRITE, mmap
def main():
ncolumn = int(sys.argv[3]) - 1 # 1st column is 1
with open(sys.argv[1], 'r+b') as file1:
with mmap(file1.fileno(), 0, access=ACCESS_WRITE) as mm:
with open(sys.argv[2], 'rb') as file2:
while True:
mm.readline() # ignore every other line
pos = mm.tell() # remember current position
if not mm.readline(): # EOF
break
replacement = file2.readline().strip()[0]
mm[pos + ncolumn] = replacement # replace the column
main()
It assumes that you are replacing a byte with a byte i.e., no content is moved in the file.
This might work for you (GNU sed, paste and cat):
cat file1 | paste -d\\n\\t\\n - file2 - | sed -r 's/^(.)\t(.{4})./\2\1/' >file3
Embed the data from file2 into file1 and then re-arrange it.