I have this rule in my Makefile, to replace ||| (three pipe characters; hex 7c 7c 7c) with CRLFNUL (carriage return + line feed + null; hex 0d 0a 00):
rom.hex: rom.txt
hexdump -C rom.txt | cut -c10-60 > rom.hex
sed -i -e 's/ / /g' rom.hex
sed -i -e 's/7c 7c 7c/0d 0a 00/g' rom.hex
This works some of the time - but, if the output of hexdump splits a 7c 7c 7c sequence across two lines it isn't matched by sed.
The replacement has to be the same length as the match, so as not to shift the subsequent bytes.
You could make the replacement first, before transforming into hex:
rom.hex: rom.txt
sed -e 's/|||/\r\n\x00/g' $< | hexdump -v | cut -c'10-60' >$#
Note that the backslash escapes are a GNU sed extension, so this is not a completely portable solution. If you need a portable sed command, you'll need to put it in a separate file, because you can't include a NUL in a command-line argument. The literal newline must be quoted, too:
s/|||/^M\
^#/g
For clarity, the control characters above are
73 2f 7c 7c 7c 2f 0d 5c 0a 00 2f 67 |s/|||/.\../g|
Then the rule would be
rom.hex: rom.txt
sed -f "transform.sed" $< | hexdump -v | cut -c'10-60' >$#
- Toby Speight's helpful answer elegantly bypasses the OP's problem by using GNU sed to replace data at the source, without needing to operate on a hex. representation (his portable alternative doesn't work with BSD sed, but that's only because of the NUL character in the replacement string).
- The value of this answer is in solving the OP's problem exactly as stated, notably using tr -s '\n' ' ', and in providing a relatively simple portable solution at the bottom - it is of interest from a byte-represenation / text processing perspective.
- See my other answer for a simpler solution that uses hexdump's formatting options to produce the desired output format directly.
Note:
The solutions below transform the byte-value representation of the input into a single line, so as to enable robust use of sed to replace values.
If you do want the fixed-width multi-line output that hexdump produces by default, pipe the output to ... | fmt -w48
The following command normalizes all whitespace in the output from hexdump -C:
hexdump -vC rom.txt | cut -c10-60 | tr -s '\n' ' ' > rom.hex
Note the addition of -v, which prevents loss of information.
Without -v, duplicates in adjacent repeating lines would be represented as *.
The result is:
a single line bookended by a leading and trailing space,
If you want to strip these, see the portable solution at the bottom.
with byte values all separated by a single space each; e.g.:
23 21 2f 62 69 6e 2f 62 61 73 68 0a 0a 23 20 23 20 76 3d 24 5f 0a 23 20 23 20 65 63 68 6f 20 22 ....
Note that tr's -s ("squeeze") option, after having performed the translation (\n to in this case, i.e.), folds runs of multiple occurrences of the target character ( (space) in this case) into single-character runs.
Thus:
The intermediate sed command (sed -i -e 's/ /...) to normalize the line-internal spaces is no longer needed.
The final sed command (sed -i -e 's/7c 7c 7c/ ...) can safely use space-separated values as the search string, without worrying about where the line breaks happened to be in hexdump -C's output.
There is room for simplification:
A single pipeline can be used - no need to write to the file in an intermediate form and update it in place later.
As a side effect, because -i is no longer needed, the sed command becomes portable (POSIX-compliant); while this form will work on both Linux and BSD/OSX platforms, it is still not strictly POSIX-compliant as a whole, because hexdump is a nonstandard utility; see the bottom for a strictly POSIX-compliant solution.
Special make variables $<, the (first) prerequisite (rom.hex), and $#, the target (rom.txt) can be used.
There is no need for the -C option of hexdump, if only the byte values are needed; this allows simplification of the cut command, which, incidentally, strips the leading space from the output (and also makes tr's -s option unnecessary):
rom.hex: rom.txt
hexdump -v $< | cut -sd' ' -f2- | tr '\n' ' ' | sed 's/7c 7c 7c/0d 0a 00/g' > $#
cut -sd' ' -f2-:
-s means that lines not containing the delimiter (separator) specified with -d are skipped, which skips a trailing empty line (empty except for the byte-offset column) that hexdump may output.
-d' ' splits the input into fields using a single space as the delimiter.
-f2- outputs the 2nd field through the end of the line (-), effectively stripping the 1st field (the input-address offset column in hexdump's output).
To make the command fully portable, POSIX utility od can be used in lieu of the nonstandard hexdump utility.
Furthermore, an extra sed command is used to strip the leading and trailing space from the output.
rom.hex: rom.txt
od -t x1 -A n -v $< | tr -s '\n' ' ' | sed 's/^ //; s/ $//' | sed 's/7c 7c 7c/0d 0a 00/g' > $#
od -t x1 -A n -v outputs hex. (x) bytes (1) across multiple lines of fixed width, similar to hexdump, except that -A n blanks out the input-address offset column; -v ensures that all bytes are represented; without it, adjacent duplicate lines would be represented as *.
tr -s '\n' ' ', as above, normalizes the whitespace to produce a single, long line with byte values separated by a single space, bookended by a single leading and trailing space.
sed 's/^ //; s/ $//' removes the leading and trailing space.
The rest of the command is as before.
- See my other answer for how to solve the problem as stated or if you need a POSIX-compliant solution.
- This answer is of interest from a byte-representation formatting perspective.
Note:
The solutions below transform the byte-value representation of the input into a single line, so as to enable robust use of sed to replace values.
If you do want the fixed-width multi-line output that hexdump produces by default, pipe the output to ... | fmt -w48
The problem can be bypassed by passing formatting options to hexdump:
hexdump -ve '1/1 "%02x "'
produces the desired output format as a single line directly (there will be a single trailing space).
-v prevents abbreviation of repeating bytes as *
-e '1/1 "%02x "':
1/1 specifies that the following format string be applied to 1 unit of byte size 1, i.e., each byte.
"%02x " is the format string to apply to each byte: a 2-digit hex number followed by a space.
To put it all together, using special make variables $<, the (first) prerequisite (rom.hex), and $#, the target (rom.txt):
rom.hex: rom.txt
hexdump -ve '1/1 "%02x "' $< | sed 's/7c 7c 7c/0d 0a 00/g' > $#
Alternative solution, using the (also nonstandard) xxd utility; like hexdump, however, it is available on both Linux and BSD/OSX:
rom.hex: rom.txt
xxd -p $< | tr -d '\n' | sed 's/../& /g; s/ $//' | sed 's/7c 7c 7c/0d 0a 00/g' > $#
xxd -p prints a stream of byte values without separators, broken into lines of fixed length.
tr -d '\n' removes the newlines from the output.
sed 's/../& /g; s/ $//' inserts a space after every 2 characters, then deletes the trailing space at the end of the line.
Finally, as Toby Speight points out in a [since cleaned-up] comment, you can use the GNU version of od with the nonstandard -w option:
rom.hex: rom.txt
od -t x1 -A n -w1 -v $< | tr -d '\n' | sed 's/7c 7c 7c/0d 0a 00/g' > $#
od -t x1 -A n -w1 -v outputs hex. (x) bytes (1) 1 byte at a time (-w1); -A n omits the input-address offset column; -v ensures that all bytes are represented; without it, adjacent duplicate lines would be represented as *.
tr -d '\n' simply removes all newlines, and since each line starts with a space, the result is a single long line with a leading space.
Related
I have a command
pdftotext -f 3 -l 3 -x 205 -y 40 -W 180 -H 75 -layout input.pdf -
When run it produces output as below
[[_थी] 2206255388
नाव मीराबाई sad
पतीचे नाव dame
| घर क्रमांक Photo's |
|वय 51 लिंग महिला Available |
I need to make each lines enclosed with double quotes and then joined to a single line separated by comma using a shell script command?
As an example, you could modify the output of your command like that:
cat <<EOF | sed 's/\(.*\)/\"\1\"/g' | tr '\n' ',' | sed 's/.$//'
> foobar
> bar
> foo
> EOF
"foobar","bar","foo"
The 1st 'sed' will add the double quotes, the 'tr' will replace the CR by a comma, last sed will remove the last comma.
So, your command will be:
pdftotext -f 3 -l 3 -x 205 -y 40 -W 180 -H 75 -layout input.pdf - | sed 's/\(.*\)/\"\1\"/g' | tr '\n' ',' | sed 's/.$//'
I have a variable "text" which contains the value "IN▒ENJERING"
Hex-values : 49 4E B4 45 4E 4A 45 52 49 4E 47
I want to remove the special character B4.
Now if I remove a regular character (e.g. "I") using the command
text=$(printf "$text" | sed "s/\x49/ /g")
the command works fine
Result : text=N▒ENJER NG
If I want to remove the special character, it seems not to work
text=$(printf "$text" | sed "s/\xB4/ /g")
Result : IN▒ENJERING
Any idea what is wrong ?
This might work for you (GNU sed):
sed 's/\o342\o226\o222/ /g' file
To find the octal representation use:
<<<"IN▒ENJERING" sed -n l
IN\342\226\222ENJERING$
Then to replace each octal character prepend \o and use as a pattern match.
$ cat -e test.csv | grep 150463452112
65,150463452112,609848340831,2.87,126,138757585104,0,0,^M$
65,150463452112,609848340832,3.37,126,138757585105,1,0,^M$
$ grep 150463452112 test.csv | grep '0,^M$'
$
I enter the '^M' with Ctrl+V Ctrl+M and need to match the line with the ending of `0,^M$'. However, the grep returns empty lines.
Question> What is the correct syntax to search the ending?
Thank you
,0,0, seen in hexdump is as follows:
2c 30 2c 30 2c 0d 0a
|,0,0,..|
The underlying problem is that your file doesn't actually contain any two-character ^M sequence (and even if it did, ^ is special to regex and doesn't match itself). Rather, it contains a carriage return before its final linefeed (being a DOS-style rather than UNIX-style text file). What you want to match is not a ^M sequence but a literal carriage return.
One way to do this is to pass grep a shell literal using bash and ksh $'' C-style string literal syntax:
grep $'0,\r$'
...which you can test as follows:
## test function: generate two lines with CRLFs, one "hello world", the other "foo,0,"
$ generate_sample_data() { printf '%s\r\n' 'hello world' 'foo,0,'; }
## demonstrate that we have 0d 0a line endings on the output from this function
$ generate_sample_data | hexdump -C
00000000 68 65 6c 6c 6f 20 77 6f 72 6c 64 0d 0a 66 6f 6f |hello world..foo|
00000010 2c 30 2c 0d 0a |,0,..|
00000015
## demonstrate that the given grep matches only the line ending in "0," before the CRLF
$ generate_sample_data | egrep $'0,\r$'
foo,0,
As pointed here, escape characters for color highlighting might be interfering with the ^M character.
You probably have grep aliased to grep --color=auto or something similar. Use \grep or grep --color=never.
$ grep 150463452112 test.csv | \grep '0,^M$'
65,150463452112,609848340831,2.87,126,138757585104,0,0,
65,150463452112,609848340832,3.37,126,138757585105,1,0,
With ^M entered with Ctrl+V Ctrl+M.
Using the command du, I would like to get the total size of a directory
Output of command du myfolder:
5454 kkkkk
666 aaaaa
3456788 total
I'm able to extract the last line, but not to remmove the string total:
du -c myfolder | grep total | cut -d ' ' -f 1
Results in:
3456788 total
Desired result
3456788
I would like to have all the command in one line.
That's probably because it's tab delimited (which is the default delimiter of cut):
~$ du -c foo | grep total | cut -f1
4
~$ du -c foo | grep total | cut -d' ' -f1
4
to insert a tab, use Ctrl+v, then TAB
Alternatively, you could use awk to print the first field of the line ending with total:
~$ du -c foo | awk '/total$/{print $1}'
4
First of, you probably want to use tail -n1 instead of grep total ... Consider what happens if you have a directory named local? :-)
Now, let's look at the output of du with hexdump:
$ du -c tmp | tail -n1 | hexdump -C
00000000 31 34 30 33 34 34 4b 09 74 6f 74 61 6c 0a |140344K.total.|
That''s the character 0x09 after the K, man ascii tells us:
011 9 09 HT '\t' (horizontal tab) 111 73 49 I
It's a tab, not a space :-)
The tab character is already the default delimiter (this is specified in the POSIX spec, so you can safely rely on it), so you don't need -d at all.
So, putting that together, we end up with:
$ du -c tmp | tail -n1 | cut -f1
140344K
Why don't you use -s to summarize it? This way you don't have to grep "total", etc.
$ du .
24 ./aa/bb
...
# many lines
...
2332 .
$ du -hs .
2.3M .
Then, to get just the value, pipe to awk. This way you don't have to worry about the delimiter being a space or a tab:
du -s myfolder | awk '{print $1}'
From man du:
-h, --human-readable
print sizes in human readable format (e.g., 1K 234M 2G)
-s, --summarize
display only a total for each argument
I would suggest using awk for this:
value=$(du -c myfolder | awk '/total/{print $1}')
This simply extracts the first field of the line that matches the pattern "total".
If it is always the last line that you're interested in, an alternative would be to use this:
value=$(du -c myfolder | awk 'END{print $1}')
The values of the fields in the last line are accessible in the END block, so you can get the first field of the last line this way.
I've a big file from which i want to remove some content, the file is binary, and i don't have line numbers, but hex address, so how can i remove the region between:
0x13e70a00 and 0x1eaec03ff
With sed (both inclusive)
Will something like this, work?
sed -n 's/\x13e70a00/,s/\x1eaec03ff/ p' orig-data-file > new-file
from what you wrote it looks like you are trying to delete all the bytes between the two hex patterns. for that you will need
this deletes all the bytes between the patterns inclusive of the patterns.
sed 's/\x13\xe7\x0a\x00.*\x1e\xae\xc0\x3f//g' in >out
This deletes all bytes between patterns leaving the patterns intact. (there is a way to this with numbered parts of regexes but this is a bit clearer to beging with)
sed 's/\x13\xe7\x0a\x00.*\x1e\xae\xc0\x3f/\x13\xe7\x0a\x00\x1e\xae\xc0\x3f/g' in >out
They search s/ for a <pattern1> followed by any text .* followed by <pattern2> and replace it with either nothing //g or just the two edges /<pattern1><pattern2>/g throughout the file /g
If you want to delete (or replace) from byte 300 to byte 310:
sed 's/\(.\{300\}\).\{10\}/\1rep-str/' in>out
this matches the first 300 characters (.\{300\} )and remembers them (the \(\) ). It matches the next 10 characters too. It replaces this whole combined match with the first 300 characters (\1) followed by your replacement string rep-str this replacement string can be empty to just delete the text between bytes 300 and 310.
However, this is quite brittle if there are any newline characters. if you can live without replacement:
dd if=file bs=1 skip=310|dd of=file bs=1 seek=300 conv=notrunc
this does an in place replacement by copying from the 310th byte onwards till into the file starting from 300 position thus deleting 10 bytes
an even more general alternative is
dd if=in bs=1 count=300>out
printf "replacement text">>out
dd if=in bs=1 skip=310>>out
though the simplest thing to do will be to use a hex editor like Bless
You should be able to use a clever combination of converting bash numbers from hex to decimal, bash math to add 1 to the decimal offsets, and cut --complement -b to remove the correct segment from the file.
EDIT: Like this:
$ snip_out 0x0f 0x10 <<< "0123456789abcdeffedcba9876543210" | od -t x1
0000000 30 31 32 33 34 35 36 37 38 39 61 62 63 64 65 65
0000020 64 63 62 61 39 38 37 36 35 34 33 32 31 30
0000036
Where snip_out is a two-parameter shell-script that operates on stdin and stdout:
#!/bin/bash
START_RANGE_DEC=$(printf "%d" $1)
END_RANGE_DEC=$(printf "%d" $2)
# Most hex ranges begin with 0; cut begins with 1.
CUT_START_DEC=$(( $START_RANGE_DEC + 1 ))
CUT_END_DEC=$(( $END_RANGE_DEC + 1 ))
# cut likes to append a newline after output. Use head to remove it.
exec cut --complement -b $CUT_START_DEC-$CUT_END_DEC | head -c -1