input
pattern01 pattern11
pattern02 NonNumeric pattern12
ouput
pattern01 pattern11
pattern02 pattern12
pattern0x has got a length from 7 to 15 characters which can be numerical or '.'
patthern1x has got length between 2 and 4 numerical characters
NonNumric is a unique strickly non numerical character
ex:
input
12.1 58
135454& 548
124.485* 5587
12.58.336./ 54
output
12.1 58
135454 548
124.485 5587
12.58.336. 54
thank you very much!
Try sed using below:
sed -r 's/[^0-9. ]//g' test.txt
Related
I have a tab delim file with a 2 columns like following
A 123
A 23
A 45
A 67
B 88
B 72
B 50
B 23
C 12
C 14
I want to transpose with the above data based on the first column like following
A 123 23 45 67
B 88 72 50 23
C 12 14
I tried the datamash transpose < input-file.txt but it didnt yield the output as expected.
One awk version:
awk '{printf ($1!=f?"\n%s":" "$2),$0;f=$1}' file
A 123 23 45 67
B 88 72 50 23
C 12 14
With this version, you get on blank line, but should be fast and handle large data since no loop or array variable are used.
$1!=f?"\n%s":" "$2),$0 If first field is not equal f, print new line and all fields
if $1 = f, only print field 2.
f=$1 set f to first field
datamash --group=1 --field-separator=' ' collapse 2 <file | tr ',' ' '
Output:
A 123 23 45 67
B 88 72 50 23
C 12 14
Input must be sorted, as in the question.
This might work for you (GNU sed):
sed -E ':a;N;s/^((\S+)\s+.*)\n\2/\1/;ta;P;D' file
Append the next line and if the first field of the first line is the same as the first field of the second line, remove the newline and the first field of the second line. Print the first line in the pattern space and then delete it and the following newline and repeat.
I had a text file f.txt which encoded as UTF-8 as shown below:
chengs-MBP:test cheng$ cat f.txt
Wіnd
like
chengs-MBP:test cheng$ FILE -I f.txt
f.txt: text/plain; charset=utf-8
However, this two words in this file Wind and like are diiferent, as the like can be find by grep command while Wind cannot, it confused me:
chengs-MBP:test cheng$ cat f.txt | grep like
like
chengs-MBP:test cheng$ cat f.txt | grep Wind
chengs-MBP:test cheng$
And I want to transform this file to us-ascii by iconv command, but I failed:
chengs-MBP:test cheng$ iconv -f UTF-8 -t US-ASCII f.txt > new.txt
conv: f.txt:1:0: cannot convert
My goal is to transform this file to a format which all of the words inside this file could be find by grep or sed... that's all.
UTF-8 is an encoding for the Unicode character set. Some Unicode characters are in look-alike subsets, sometimes collectively called "confusables". So,
f.txt | grep "Wind"
will look for LATIN SMALL LETTER I, while
f.txt | grep "Wіnd"
will look for CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I. You could write it as
f.txt | grep "W\xD1\x96nd"
Since і is not a member of the ASCII character set, it can't be encoded in ASCII.
If you want to take this further, I suggest that you don't abandon UTF-8 as your text file encoding but you might want to transliterate confusable letters to Basic Latin or use a search library that deals with transliteration as a feature. grep just gives you exactly what you ask for.
In my examples I will use 'i' in word 'Wind' is CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I in hex d196.
For finding out symbol hex representation you can use xxd or hexdump:
$ xxd -g 1 f.txt
00000000: 57 d1 96 6e 64 0a 6c 69 6b 65 0a W..nd.like.
$ hexdump -C f.txt
00000000 57 d1 96 6e 64 0a 6c 69 6b 65 0a |W..nd.like.|
As you can see, on the right side, in ASCII section UTF symbols replaced with dots.
Also you can use Unicode Utitilies.
$ uniname f.txt
character byte UTF-32 encoded as glyph name
0 0 000057 57 W LATIN CAPITAL LETTER W
1 1 000456 D1 96 і CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I
2 3 00006E 6E n LATIN SMALL LETTER N
3 4 000064 64 d LATIN SMALL LETTER D
4 5 00000A 0A LINE FEED (LF)
5 6 00006C 6C l LATIN SMALL LETTER L
6 7 000069 69 i LATIN SMALL LETTER I
7 8 00006B 6B k LATIN SMALL LETTER K
8 9 000065 65 e LATIN SMALL LETTER E
9 10 00000A 0A LINE FEED (LF)
After you find out what symbols in your file are not ASCII, you can replace them in file to ASCII equivalent.
$ sed -i 's/\xd1\x96/i/g' f.txt
Can someone please help! I'm trying to delete the last portion (following "_cO") in the second column of the following list in the bash shell. E.G where it says "_seq1" in this particular list. I do not want to change any other info in the remaining columns.
Thanks!
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
Here you go, a simple substitution using sed:
sed -e s/_seq1//
Using sed:
sed -i.bak 's/^\(.*_c0\)[^ ]*\( .*\)$/\1\2/' file
OR using awk:
awk '{sub(/_c0[^ ]*/, "_c0", $2)} 1' file
I've a big file from which i want to remove some content, the file is binary, and i don't have line numbers, but hex address, so how can i remove the region between:
0x13e70a00 and 0x1eaec03ff
With sed (both inclusive)
Will something like this, work?
sed -n 's/\x13e70a00/,s/\x1eaec03ff/ p' orig-data-file > new-file
from what you wrote it looks like you are trying to delete all the bytes between the two hex patterns. for that you will need
this deletes all the bytes between the patterns inclusive of the patterns.
sed 's/\x13\xe7\x0a\x00.*\x1e\xae\xc0\x3f//g' in >out
This deletes all bytes between patterns leaving the patterns intact. (there is a way to this with numbered parts of regexes but this is a bit clearer to beging with)
sed 's/\x13\xe7\x0a\x00.*\x1e\xae\xc0\x3f/\x13\xe7\x0a\x00\x1e\xae\xc0\x3f/g' in >out
They search s/ for a <pattern1> followed by any text .* followed by <pattern2> and replace it with either nothing //g or just the two edges /<pattern1><pattern2>/g throughout the file /g
If you want to delete (or replace) from byte 300 to byte 310:
sed 's/\(.\{300\}\).\{10\}/\1rep-str/' in>out
this matches the first 300 characters (.\{300\} )and remembers them (the \(\) ). It matches the next 10 characters too. It replaces this whole combined match with the first 300 characters (\1) followed by your replacement string rep-str this replacement string can be empty to just delete the text between bytes 300 and 310.
However, this is quite brittle if there are any newline characters. if you can live without replacement:
dd if=file bs=1 skip=310|dd of=file bs=1 seek=300 conv=notrunc
this does an in place replacement by copying from the 310th byte onwards till into the file starting from 300 position thus deleting 10 bytes
an even more general alternative is
dd if=in bs=1 count=300>out
printf "replacement text">>out
dd if=in bs=1 skip=310>>out
though the simplest thing to do will be to use a hex editor like Bless
You should be able to use a clever combination of converting bash numbers from hex to decimal, bash math to add 1 to the decimal offsets, and cut --complement -b to remove the correct segment from the file.
EDIT: Like this:
$ snip_out 0x0f 0x10 <<< "0123456789abcdeffedcba9876543210" | od -t x1
0000000 30 31 32 33 34 35 36 37 38 39 61 62 63 64 65 65
0000020 64 63 62 61 39 38 37 36 35 34 33 32 31 30
0000036
Where snip_out is a two-parameter shell-script that operates on stdin and stdout:
#!/bin/bash
START_RANGE_DEC=$(printf "%d" $1)
END_RANGE_DEC=$(printf "%d" $2)
# Most hex ranges begin with 0; cut begins with 1.
CUT_START_DEC=$(( $START_RANGE_DEC + 1 ))
CUT_END_DEC=$(( $END_RANGE_DEC + 1 ))
# cut likes to append a newline after output. Use head to remove it.
exec cut --complement -b $CUT_START_DEC-$CUT_END_DEC | head -c -1
thegladiator:~/cp$ cat new.txt
Hello World This is a Trest Progyy
thegladiator:~/cp$ hexdump new.txt
0000000 6548 6c6c 206f 6f57 6c72 2064 6854 7369
0000010 6920 2073 2061 7254 7365 2074 7250 676f
0000020 7979 000a
0000023
How is that text data represented in hex like that? What is the meaning of this?
it's just what it says, a dump of the data in hexidecimal format:
H 48
e 65
l 6c
l 6c
o 6f
It is odd though that all of the bytes are swapped (65 48 : e H)
If you're on a *nix system, you can use 'od -x', or 'man od' will tell you all the ways to get data from od :)
The text in the file new.txt is stored using ASCII encoding. Each letter is represented by a number, decimal: 32-127 hexidecimal: 20-7F. So the first three letters (H,e,l), are represented by the decimal numbers: 72,101,108 and the hexidecimal numbers: 48,65,6C
Hexdump by default takes each 16 bit word of the input file new.txt and outputs this word as a Hexidecimal number. Because it is operating on 16 bits, not 8 bits, you see the output in an unexpected order.
If you instead use xxd new.txt, you will see the output in the expected order.