Best way to split large string in linux with single character shift - linux

Question

I have a large file that contains a single example string
ABCDEFGHI (example length 10 characters).
Actual file length could be millions of characters.
I would like to split the string into multiple lines with a predetermined length but while splitting the character is shifted 1 at a time. This means after splits the
no. of lines = string length - split size + 1
Example if I split it by 3 character at a time then desired output
ABC
BCD
CDE
DEF
...
If I split by 4 characters then
ABCD
BCDE
CDEF
DEFG
What is the best way of doing this split using shell commands or scripting?
Thanks for any hints

Answer 1

You can try something like this:
gawk -v FS="" '{
r=3 # Set the length
s=1 # Set the start point
while(s<=NF-r+1) {
for (i=s;i<r+s;i++) {
printf $i
}
s++
print ""
}
}'
Test:
$ echo "ABCDEFGHI" | gawk -v FS="" '{r=4; s=1; while(s<=NF-r+1) { for (i=s;i<r+s;i++) printf $i ; s++; print ""}}'
ABCD
BCDE
CDEF
DEFG
EFGH
FGHI
$ echo "ABCDEFGHI" | gawk -v FS="" '{r=3; s=1; while(s<=NF-r+1) { for (i=s;i<r+s;i++) printf $i ; s++; print ""}}'
ABC
BCD
CDE
DEF
EFG
FGH
GHI

Answer 2

Here is a way with sed (in bash):
GNU sed:
sed -r ':a;s/([^\n])([^\n]{'$(( n-1 ))'})([^\n])/\1\2\n\2\3/;ta' filename
or POSIX sed (I think):
sed ':a;s/\([^\n]\)\([^\n]\{'$(( n-1 ))'\}\)\([^\n]\)/\1\2\n\2\3/;ta' filename
Output:
with n=3:
ABC
BCD
CDE
DEF
EFG
FGH
GHI
with n=4:
ABCD
BCDE
CDEF
DEFG
EFGH
FGHI

Answer 3

Another awk-based option, involving substr
echo 'abcdefgh' |
awk -v limit=3 'BEGIN{FS=""};
{value=$0; for (i=1; i<= NF-limit +1; ++i) print substr(value, i, limit)}'
abc
bcd
cde
def
efg
fgh
ghi

Answer 4

While I generally dislike bringing in heavyweight scripting languages like this, python makes this pretty much trivial
$ cat test.py
#!/usr/bin/env python
from os import sys
n = int(sys.argv[1])
s = sys.argv[2]
while len(s) > 0:
print s[:n]
s = s[1:]
$ python test.py 3 abcdef
abc
bcd
cde
def
ef
f
$ python test.py 4 abcdef
abcd
bcde
cdef
def
ef
f
$
If you want to stop once you run out of characters, you can change the while condition to len(s) >= n.

Answer 5

using python you could write something like this:
import itertools
filename = "myfile"
length = 4
with open(filename, 'r') as f:
out = ''
# get your input character by character
for c in itertools.chain.from_iterable(f):
# append it to your output buffer
out += c
# if your buffer is more than N characters, remove the first char
if len(out) > length:
out = out[1:]
# if your buffer is exactly N characters, print it out (or do something else)
if len(out) is length:
print out
# if the last iteration was less than N characters, print it out (or do something else)
if len(out) < length:
print out
where file is a string containing the full path of your string. You can use also raw_input() instead of open()/read(). There sure is a neat solution using awk, but I would need to RTFM to tell you how to do it.
Whatever your solution is, this algorithm is a good way to do it, as you always keep only up to N+1 characters for the buffer, plus one character for the new read. So the complexity of this algorithm is linear (O(n)) to the input character stream.