changing a character of columns of a large file - string

Question

I have to replace a single nth character of each (alternate) row of a large file with correponding column of another file. For eg I am changing every 5th character.
file1:
>chr1:101842566-101842576
CCTCAACTCA
>chr1:101937281-101937291
GAATTGGATA
>chr1:101964276-101964286
AAAAAATAGG
>chr1:101972950-101972960
ggctctcatg
>chr1:101999969-101999979
CATCATGACG
file2:
G
A
T
A
C
output:
>chr1:101842566-101842576
CCTCGACTCA
>chr1:101937281-101937291
GAATAGGATA
>chr1:101964276-101964286
AAAATATAGG
>chr1:101972950-101972960
ggctAtcatg
>chr1:101999969-101999979
CATCCTGACG
The number of characters in each (alternate) row can be large. And number of rows are large too. How this can be done efficiently?

Answer 1

Here is one way with awk:
awk 'NR==FNR{a[NR]=$1;next}!/^>/{$1=substr($1,1,n-1) a[++i] substr($1,n+1)}1' n=5 f2 f1
Explanation:
We iterate over second file and store it in an array indexed at line number.
Once the second file is loaded in memory, we move to the second file.
We look for lines not starting with >.
When found we substitute the value from our array. We do this by using substr function.
The variable n defined allows you to modify the nth character
For the lines that donot have > we print them as is using 1 which is default for
printing.
This solution assumes the format of the file is as shown above. That is, the first file will always start with > followed by the line you want to make changes on. Substitution from the second file will be made in the order it is seen.
Demo:
Every 5th character:
$ awk 'NR==FNR{a[NR]=$1;next}!/^>/{$1=substr($1,1,n-1) a[++i] substr($1,n+1)}1' n=5 f2 f1
>chr1:101842566-101842576
CCTCGACTCA
>chr1:101937281-101937291
GAATAGGATA
>chr1:101964276-101964286
AAAATATAGG
>chr1:101972950-101972960
ggctAtcatg
>chr1:101999969-101999979
CATCCTGACG
Every 3rd character:
$ awk 'NR==FNR{a[NR]=$1;next}!/^>/{$1=substr($1,1,n-1) a[++i] substr($1,n+1)}1' n=3 f2 f1
>chr1:101842566-101842576
CCGCAACTCA
>chr1:101937281-101937291
GAATTGGATA
>chr1:101964276-101964286
AATAAATAGG
>chr1:101972950-101972960
ggAtctcatg
>chr1:101999969-101999979
CACCATGACG

Answer 2

This is how I would use perl. First read all of file2 into an array, and then iterate over that array reading two and two lines from file1, printing the first line unmodified and then change the 5th character on the second line:
#!/usr/bin/perl
use strict;
use warnings;
use diagnostics;
#use Data::Printer;
# Read all of file2
my $lines;
open(FILE, $ARGV[1]);
{
local $/;
$lines = <FILE>;
}
close(FILE);
my #new_chars = split(/\n/, $lines);
# Read and process file1
open(FILE, $ARGV[0]);
foreach my $new_char (#new_chars) {
# >chr1:101842566-101842576
my $line = <FILE>;
print $line;
# CCTCAACTCA
$line = <FILE>;
$line =~ s/^(....)./$1$new_char/; # Replace 5th character
print $line;
}
close(FILE);

Answer 3

You could replace a column in the file inplace using mmap in Python:
#!/usr/bin/env python3
"""Replace inplace a column of a large file.
Usage:
$ ./replace-inplace file1 file2 5
"""
import sys
from mmap import ACCESS_WRITE, mmap
def main():
ncolumn = int(sys.argv[3]) - 1 # 1st column is 1
with open(sys.argv[1], 'r+b') as file1:
with mmap(file1.fileno(), 0, access=ACCESS_WRITE) as mm:
with open(sys.argv[2], 'rb') as file2:
while True:
mm.readline() # ignore every other line
pos = mm.tell() # remember current position
if not mm.readline(): # EOF
break
replacement = file2.readline().strip()[0]
mm[pos + ncolumn] = replacement # replace the column
main()
It assumes that you are replacing a byte with a byte i.e., no content is moved in the file.

Answer 4

This might work for you (GNU sed, paste and cat):
cat file1 | paste -d\\n\\t\\n - file2 - | sed -r 's/^(.)\t(.{4})./\2\1/' >file3
Embed the data from file2 into file1 and then re-arrange it.