What is the difference between these commands? Why does the first match, but grepping the individual file hexdumps returns nothing?
$ hexdump -C DRAFT* FAIL* RCV* | grep "49 d0 38 ec 06 a1 c3"
0001f430 49 d0 38 ec 06 a1 c3 7a 10 39 9c 07 bd cd 66 10 |I.8....z.9....f.|
$ hexdump -C RCV* | grep "49 d0 38 ec 06 a1 c3"
$ hexdump -C DRAFT* | grep "49 d0 38 ec 06 a1 c3"
$ hexdump -C FAIL* | grep "49 d0 38 ec 06 a1 c3"
$ ls
DRAFT DRAFT.HDR DRAFTUSED FAILED FAILED.HDR FAILEDUSED RCVD RCVD.HDR RCVDUSED SENT SENT.HDR SENTUSED SMS.html
The files are from my Samsung Sync SGH-A707. I'm trying to parse these sms files so I can make a plain-text backup of my text messages. The string I'm grepping for is a control message I encoded according to http://www.dreamfabric.com/sms/hello.html
Hexdump will concatenate the files, then convert to hex. If the character sequence is split across a line boundary, then it will not match your grep. I suspect this happens in some cases, but not others.
You could try using the -n option with a large number (larger than the files you are using) to put the whole output on a single line.
Related
$ cat -e test.csv | grep 150463452112
65,150463452112,609848340831,2.87,126,138757585104,0,0,^M$
65,150463452112,609848340832,3.37,126,138757585105,1,0,^M$
$ grep 150463452112 test.csv | grep '0,^M$'
$
I enter the '^M' with Ctrl+V Ctrl+M and need to match the line with the ending of `0,^M$'. However, the grep returns empty lines.
Question> What is the correct syntax to search the ending?
Thank you
,0,0, seen in hexdump is as follows:
2c 30 2c 30 2c 0d 0a
|,0,0,..|
The underlying problem is that your file doesn't actually contain any two-character ^M sequence (and even if it did, ^ is special to regex and doesn't match itself). Rather, it contains a carriage return before its final linefeed (being a DOS-style rather than UNIX-style text file). What you want to match is not a ^M sequence but a literal carriage return.
One way to do this is to pass grep a shell literal using bash and ksh $'' C-style string literal syntax:
grep $'0,\r$'
...which you can test as follows:
## test function: generate two lines with CRLFs, one "hello world", the other "foo,0,"
$ generate_sample_data() { printf '%s\r\n' 'hello world' 'foo,0,'; }
## demonstrate that we have 0d 0a line endings on the output from this function
$ generate_sample_data | hexdump -C
00000000 68 65 6c 6c 6f 20 77 6f 72 6c 64 0d 0a 66 6f 6f |hello world..foo|
00000010 2c 30 2c 0d 0a |,0,..|
00000015
## demonstrate that the given grep matches only the line ending in "0," before the CRLF
$ generate_sample_data | egrep $'0,\r$'
foo,0,
As pointed here, escape characters for color highlighting might be interfering with the ^M character.
You probably have grep aliased to grep --color=auto or something similar. Use \grep or grep --color=never.
$ grep 150463452112 test.csv | \grep '0,^M$'
65,150463452112,609848340831,2.87,126,138757585104,0,0,
65,150463452112,609848340832,3.37,126,138757585105,1,0,
With ^M entered with Ctrl+V Ctrl+M.
I have a document whose lines are separated by "\t\n". Records are separated either by "\t", OR by "\n".
Normally, this should be a straigtforward awk query:
BEGIN {
RS='\t\n';
}
{
print;
print "Next entry:";
}
However, on a Mac, regular expressions do not seem to be supported (maybe I'm not doing something right?) So I tried, RS="\t\n"; however, this is interpreted as RS='\t | \n'. Similar problems running awk from the command line:
awk 1 RS='\t\n' ORS='abc' input > output
replaces the \t's, but leaves the \n's be.
Next try: using tr. This obviously fails for sequence of more than one character-- since \t and \n are both used individually in the rows.
Next:
sed -e '/\t\n/s//NextEntry:/g' input > output
However, doesn't work. Entering any ASCII character sequence instead of \t\n works.
Read the manual. It says that \t is not supported in sed strings. Fair enough
sed -e '/\x9\xa/s//abc/' input > output
Still doesn't work. Idea: use tr to replace \t and \n by characters unused in the input file, use sed to change them to what I want, and then tr to change the remaining characters back to what they should be.
tr: Illegal byte sequence
Turns out, that f6 character makes tr just totally fail.
Went through the suggestions in Sed not recognizing \t instead it is treating it as 't' why? . That might work for replacing output strings (except the "Pasting tab into command prompt via CTRL+V" suggestion-- the shell just rejected that paste.), but did not seem to help in my case.
Maybe it's because it's a Mac? Maybe it's because that's the text I'm looking for, not replacing with? Maybe it's the combination with \n?
Any other suggestions?
UPDATE:
I found thread How can I replace a newline (\n) using sed? . Apparently, I am unable even to replace a \n by the string "abc" using the suggestions in that thread.
EDIT: Hex head of source file:
5a 20 4e 4f 09 0a 41 53 20 4f 46 20 30 31 2d 30
34 2d 30 35 20 45 4d 50 4c 4f 59 45 45 0a 47 52
4f 55 50 09 48 49 52 45 20 44 41 54 45 09 53 41
4c 41 52 59 09 4a 4f 42 20 54 49 54 4c 45 09 0a
4a 4f 42 20 4c 45 56 45 4c 0a 53 45 52 49 45 53
09 41 50 50 54 20 54 59 50 45 09 0a 50 41 59 20
53 54 41 54 55 53 0a f6
Unfortunately, BSD awk, as also used on macOS, doesn't support multi-character record separators (RS) altogether (in line with POSIX) - only a single, literal character is supported.
BSD sed, as also used on macOS, supports only \n in regexes - any other escapes, including hex ones (e.g., \x09) are not supported.
See this answer of mine for a comprehensive comparison of GNU and BSD sed.
Assuming that your sed command works in principle, you can use an ANSI C-quoted string
($'\t') to splice a literal tab char. into your sed script (assumes bash (the macOS default shell), ksh, or zsh),:
sed -e ':a' -e '$!{N;ba' -e '}' -e '/'$'\t''\n/s//NextEntry:/g'
Note that, in order to replace newlines, you must instruct sed to read the entire file into memory first, which is what -e ':a' -e '$!{N;ba' -e '}' does (the BSD Sed-compatible form of the common GNU sed idiom :a;$!{N;ba}).
I have hex code of a binary in text (string) format. How do I convert it to a binary file using linux commands like cat and echo ?
I know command following command with create a binary test.bin. But what if this hexcode is in another .txt file ? How do I "cat" the content of text file to "echo" and generate a binary file ?
# echo -e "\x00\x001" > test.bin
use xxd -r. it reverts a hexdump to its binary representation.
source and source
Edit: The -p parameter is also very useful. It accepts "plain" hexadecimal values, but ignores whitespace and line changes.
So, if you have a plain text dump like this:
echo "0000 4865 6c6c 6f20 776f 726c 6421 0000" > text_dump
You can convert it to binary with:
xxd -r -p text_dump > binary_dump
And then get useful output with something like:
xxd binary_dump
If you have long text or text in file you can also use the binmake tool that allows you to describe in text format some binary data and generate a binary file (or output to stdout). It allows to change the endianess and number formats and accepts comments.
Its default format is hexadecimal but not limited to this.
First get and compile binmake:
$ git clone https://github.com/dadadel/binmake
$ cd binmake
$ make
You can pipe it using stdin and stdout:
$ echo '32 decimal 32 61 %x20 %x61' | ./binmake | hexdump -C
00000000 32 20 3d 20 61 |2 = a|
00000005
Or use files. So create your text file file.txt:
# an exemple of file description of binary data to generate
# set endianess to big-endian
big-endian
# default number is hexadecimal
00112233
# man can explicit a number type: %b means binary number
%b0100110111100000
# change endianess to little-endian
little-endian
# if no explicit, use default
44556677
# bytes are not concerned by endianess
88 99 aa bb
# change default to decimal
decimal
# following number is now decimal
0123
# strings are delimited by " or '
"this is some raw string"
# explicit hexa number starts with %x
%xff
Generate your binary file file.bin:
$ ./binmake file.txt file.bin
$ hexdump file.bin -C
00000000 00 11 22 33 4d e0 77 66 55 44 88 99 aa bb 7b 74 |.."3M.wfUD....{t|
00000010 68 69 73 20 69 73 20 73 6f 6d 65 20 72 61 77 20 |his is some raw |
00000020 73 74 72 69 6e 67 ff |string.|
00000027
In addition of xxd, you should also look at the packages/commands od and hexdump. All are similar, however each provide slightly different options that will allow you to tailor the output to your desired needs. For example hexdump -C is the traditional hexdump with associated ASCII translation along side.
I have a 100M row file that has some encoding problems -- was "originally" EBCDIC, saved as US-ASCII, now UTF-8. I don't know much more about its heritage, sorry -- I've just been asked to analyze the content.
The "cents" character from EBCDIC is "hidden" in this file in random places, causing all sorts of errors. Here is more on this bugger: cents character in hex
Converting this file using iconv -f foo -t UTF-8 -c is not working -- the cents character prevails.
When I use hex editor, I can find the appearance of 0xC2 0xA2 (c2a2). But in a BIG file, this isn't ideal. Sed doesn't work at hex level, so... Not sure about tr -- I only really use it for carriage return / new line.
What linux utility / command can I use to find and delete this character reasonably quickly on very big files?
2 parts:
1 -- utility / command to find / count the number of these occurrences (octal \242)
2 -- command to replace (this works tr '\242' ' ' < source > output )
How the text appears on my ubuntu terminal:
1019EQ?IT DEPT GENERATED
With xxd, how it looks at hex level (ascii to the side looks the same as above):
0000000: 3130 3139 4551 a249 5420 4445 5054 2047 454e 4552 4154 4544 0d0a
With xxd, how it looks with "show ebcdic" -- here, just showing the ebcdic from side:
......s.....&....+........
So hex "a2" is the culprit. I'm now trying xxd -E foo | grep a2 to count the instances up.
Adding output from od -ctxl, rather than xxd, for those interested:
0000000 1 0 1 9 E Q 242 I T D E P T G
31 30 31 39 45 51 a2 49 54 20 44 45 50 54 20 47
0000020 E N E R A T E D \r \n
45 4e 45 52 41 54 45 44 0d 0a
When you say the file was converted what do you mean? Do you mean the binary file was simply dumped from an IBM 360 to another ASCII based computer, or was the file itself converted over to ASCII when it was transferred over?
The question is whether the file is actually in a well encoded state or not. The other question is how do you want the file encoded?
On my Mac (which uses UTF-8 by default, just like Linux systems), I have no problem using sed to get rid of the ¢ character:
Here's my file:
$ cat test.txt
This is a test --¢-- TEST TEST
$ od -ctx1 test.txt
0000000 T h i s i s a t e s t -
54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 2d
0000020 - ¢ ** - - T E S T T E S T \n
2d c2 a2 2d 2d 20 54 45 53 54 20 54 45 53 54 0a
0000040
You can see that cat has no problems printing out that ¢ character. And, you can see in the od dump the c2a2 encoding of the ¢ character.
$ sed 's/¢/$/g' test.txt > new_test.txt
$ cat new_test.txt
This is a test --$-- TEST TEST
$ od -ctx1 new_test.txt
0000000 T h i s i s a t e s t -
54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 2d
0000020 - $ - - T E S T T E S T \n
2d 24 2d 2d 20 54 45 53 54 20 54 45 53 54 0a
0000037
Here's my sed has no problems changing that ¢ into a $ sign. The dump now shows that this test file is equivalent to a strictly ASCII encoded file. That two hexadecimal digit encoded ¢ is now a nice clean single hexadecimal digit encoded $.
It looks like sed can handle your issue.
If you want to use this file on a Windows system, you can convert the file to the standard Windows Code Page 1252:
$ iconv -f utf8 -t cp1252 test.txt > new_test.txt
$ cat new_test.txt
This is a test --?-- TEST TEST
$ od -ctx1 new_test.txt
0000000 T h i s i s a t e s t -
54 68 69 73 20 69 73 20 61 20 74 65 73 74 20 2d
0000020 - 242 - - T E S T T E S T \n
2d a2 2d 2d 20 54 45 53 54 20 54 45 53 54 0a
0000037
Here's the file now in Codepage 1252 just like the way Windows likes it! Note that the ¢ is now a nice hex 242 character.
So, what is exactly the issue? Do you need to file in pure ASCII defined 127 characters? Do you need the file encoded, so Windows machines can work on it? Are you having problems entering the ¢ character?
Let me know. I'm not from the government, and yet I'm here to help you.
This question already has answers here:
Capturing multiple line output into a Bash variable
(7 answers)
Closed 7 years ago.
I'm writing a shell script which will store the output of a command in a variable, process the output, and later echo the results. Here's what I've got:
stuff=$(diff -u pens tape)
# process the output
echo $stuff
The problem is, the output I get from running the script is this:
--- pens 2009-09-27 10:29:06.000000000 -0400 +++ tape 2009-09-18 16:45:08.000000000 -0400 ## -1,4 +1,2 ## -highlighter -marker -pencil -POSIX +masking +duct
Whereas I was expecting this:
--- pens 2009-09-27 10:29:06.000000000 -0400
+++ tape 2009-09-18 16:45:08.000000000 -0400
## -1,4 +1,2 ##
-highlighter
-marker
-pencil
-POSIX
+masking
+duct
It looks like the newline characters are being removed somehow. How do I get them to say in?
If you want to preserve the newlines, enclose the variable in double quotes:
echo "$stuff"
When you write it without the double quotes, the shell expands $stuff into a space-separated list of words (where 'words' are sequences of non-space characters, and the space characters are blanks and tabs and newlines; upon experimentation, it seems that form feeds, carriage returns and back-spaces are not counted as space).
Demonstrating interpretation of control characters as white space. ASCII 8 is backspace, 9 is tab, 10 is new line (LF), 11 is vertical tab, 12 is form feed, 13 is carriage return. The first command generates a sequence of characters separated by the various control characters. The second command echoes with the result with the original characters preserved - see the hex dump. The third command echoes the result with the shell splitting the words; you can see that the tab and newline were replaced by blank (0x20).
$ x=$(./ascii 64 65 8 66 67 9 68 69 10 70 71 11 72 73 12 74 75 13 76 77)
$ echo "$x" | odx
0x0000: 40 41 08 42 43 09 44 45 0A 46 47 0B 48 49 0C 4A #A.BC.DE.FG.HI.J
0x0010: 4B 0D 4C 4D 0A K.LM.
0x0015:
$ echo $x | odx
0x0000: 40 41 08 42 43 20 44 45 20 46 47 0B 48 49 0C 4A #A.BC DE FG.HI.J
0x0010: 4B 0D 4C 4D 0A K.LM.
0x0015:
$