I am trying to make a binary operation then add 0b to it then save it into an output.dat file. for example 0b1101. But it seems like binary operation overrides itself to the 0b.
#!/bin/bash
binary="0b"
while IFS=" ," read i1 i2 i3 #assigns each line into three seperate entity.
do
#checks if it's in binary, decimal or hexadecimal
if [[ $i1 == *"0b"* ]]; then #binary
i1=${i1//$binary/}
i3=${i3//$binary/}
if [ "$i2" = "+" ]; then
echo "0b" >$HOME/Desktop/Homework_1/output.dat
echo "ibase=2;obase=2; $i1+$i3" | bc -l
>$HOME/Desktop/Homework_1/output.dat
There are two errors here:
First, you're redirecting both outputs with >. You should be aware that this will clear the content of the target file before writing. To append, use >> as the redirection operator:
echo "ibase=2;obase=2; $i1+$i3" | bc -l >> $HOME/Desktop/Homework_1/output.dat
# ^^
Second, there's another issue with your bc calculation: You cannot specify obase=2 after ibase=2. You should change it into this:
echo "obase=2;ibase=2; $i1+$i3" | bc -l
# specify obase first
You can read more about this issue here #>> bc: Why does ibase=16; obase=10; FF returns FF and not 255?
Good day,
This is kinda lenghty, Im hoping for the kind help of anybody who can support me on this simple problem (to others) but taking me almost forever to figure out.
I have this file (EOL.txt) which consists of the following sample lists:
35 - 5976
36 - 5976C0
53 - 5976C2
64 - 5976D0
69 - 43593
72 - 43593C0
Im using the following commands to extract the leftmost figure since this correspond to a routine number of another script:
grep 5976C2 EOL.txt | head -n1 | cut -d- -f1
After I acquired that number, I will input that along with the other data on another script (N.csh-syntax as follows) that will execute another one (Test.csh):
$./N.csh 53 XXXX.XX "01 02 03"
N.csh --> printf "$1\n$2\n$3\nYYYY\n1\nN\n" | /export/home/Script/Test.csh
What I want to do now is to incorporate the grep command to N.csh so that I wont have to do that separately. It should look like this:
$./N.csh 5976C2 XXXX.XX "01 02 03"
I tried the following commands but its not working.
grep $1 EOL.txt | head -n1 | cut -d- -f1 >> A ; set B=`cat A` ; printf %s "$B\n$2\n$3\n82869\n1\nN\n"
Im new to this stuff, any help will be highly appreciated.
Thanks a lot in advance.
Mike
You can use the following in the file N.csh:
set mynumber = `grep $1 EOL.txt | head -n1 | cut -d- -f1`
printf "$mynumber\n$2\n$3\n...
and then invoke N.csh like
./N.csh 5976C2 XXXX.XX "01 02 03"
Note that after set mynumber =, in the first line, there is a "backtick" - a reversed single quote. The shell executes the commands delimited by two backticks, takes the output, and puts it back in place of the original contents, so the first line turns into set mynumber = 53.
Is it possible to create an alias which contains a variable in its name (i.e. without space)? I would like to set conversion shortcuts like hexa to binary:
For instance, when I enter:
0xff
I would like to display the following:
11111111
The idea would be something like (this example is obviously not working):
alias 0x$1='echo "obase=2; ibase=16; $1" | bc'
bash aliases does not take variables as input(s). You can use a function instead:
$ hex_to_bin () { echo "obase=2; ibase=16; $1" | bc ;}
$ hex_to_bin 6A
1101010
No, in bash, aliases work on individual words. So, unless you want to do:
alias 0x00='echo 00000000'
alias 0x01='echo 00000001'
then you're not going to get it working easily. It can be done (for a limited range) using that method by creating a script to source on the fly but it will result in a lot of aliases:
( for i in {0..255} ; do printf "alias 0x%02x='echo %d'\n" $i $i ; done ) >xyzzy_$$.sh
source xyzzy_$$.sh
rm xyzzy_$$.sh
Instead, I'd just opt for an function x where you could do:
pax> x ff
11111111
There are no more keystrokes than you would need for 0xff and you don't have to worry about trying to coerce bash into doing something it's not really built to do.
Such a function can be created with:
pax> x() {
...> val=$(tr '[a-z]' '[A-Z]' <<< $1)
...> BC_LINE_LENGTH=0 bc <<< "ibase=16;obase=2;$val"
...> }
and used as follows:
pax> x ff
11111111
pax> x 42
1000010
pax> x cB
11001011
pax> x 457365384563453653276537456354635635326535635345
10001010111001101100101001110000100010101100011010001010011011001010011001001110110010100110111010001010110001101010100011000110101011000110101001100100110010100110101011000110101001101000101
Note the use of tr to coerce alphas into uppercase values. Without that, you're likely to run into problems with bc recognising them as valid hex digits.
Also note the setting of BC_LINE_LENGTH to prevent bc from auto-wrapping very large numbers, such as that last one where you would otherwise see:
pax> y 457365384563453653276537456354635635326535635345
10001010111001101100101001110000100010101100011010001010011011001010\
01100100111011001010011011101000101011000110101010001100011010101100\
0110101001100100110010100110101011000110101001101000101
I'm trying to write a script, In this script i'm passing a shell variable into an awk command, But when i run it nothing happens, i tried to run that line only in the shell, i found that no variable expansion happened like i expected. Here's the code :
1 #!/bin/bash
2
3 # Created By Rafael Adel
4
5 # This script is to start dwm with customizations needed
6
7
8 while true;do
9 datestr=`date +"%r %d/%m/%Y"`
10 batterystr=`acpi | grep -oP "([a-zA-Z]*), ([0-9]*)%"`
11 batterystate=`echo $batterystr | grep -oP "[a-zA-Z]*"`
12 batterypercent=`echo $batterystr | grep -oP "[0-9]*"`
13
14 for nic in `ls /sys/class/net`
15 do
16 if [ -e "/sys/class/net/${nic}/operstate" ]
17 then
18 NicUp=`cat /sys/class/net/${nic}/operstate`
19 if [ "$NicUp" == "up" ]
20 then
21 netstr=`ifstat | awk -v interface=${nic} '$1 ~ /interface/ {printf("D: %2.1fKiB, U: %2.1fKiB",$6/1000, $8/1000)}'`
22 break
23 fi
24 fi
25 done
26
27
28 finalstr="$netstr | $batterystr | $datestr"
29
30 xsetroot -name "$finalstr"
31 sleep 1
32 done &
33
34 xbindkeys -f /etc/xbindkeysrc
35
36 numlockx on
37
38 exec dwm
This line :
netstr=`ifstat | awk -v interface=${nic} '$1 ~ /interface/ {printf("D: %2.1fKiB, U: %2.1fKiB",$6/1000, $8/1000)}'`
Is what causes netstr variable not to get assigned at all. That's because interface is not replaced with ${nic} i guess.
So could you tell me what's wrong here? Thanks.
If you want to /grep/ with your variable, you have 2 choices :
interface=eth0
awk "/$interface/{print}"
or
awk -v interface=eth0 '$0 ~ interface{print}'
See http://www.gnu.org/software/gawk/manual/gawk.html#Using-Shell-Variables
it's like I thought, awk substitutes variables properly, but between //, inside regex ( or awk regex, depending on some awk parameter AFAIR), awk variable cannot be used for substitution
I had no issue grepping with variable inside an awk program (for simple regexp cases):
sawk1='repo\s+module2'
sawk2='#project2\s+=\s+module2$'
awk "/${sawk1}/,/${sawk2}/"'{print}' aFile
(Here the /xxx/,/yyy/ displays everything between xxx and yyy)
(Note the double-quoted "/${sawk1}/,/${sawk2}/", followed by the single-quoted '{print}')
This works just fine, and comes from "awk: Using Shell Variables in Programs":
A common method is to use shell quoting to substitute the variable’s value into the program inside the script.
For example, consider the following program:
printf "Enter search pattern: "
read pattern
awk "/$pattern/ "'{ nmatches++ }
END { print nmatches, "found" }' /path/to/data
The awk program consists of two pieces of quoted text that are concatenated together to form the program.
The first part is double-quoted, which allows substitution of the pattern shell variable inside the quotes.
The second part is single-quoted.
It does add the caveat though:
Variable substitution via quoting works, but can potentially be messy.
It requires a good understanding of the shell’s quoting rules (see Quoting), and it’s often difficult to correctly match up the quotes when reading the program.
Say I have generated the following binary file:
# generate file:
python -c 'import sys;[sys.stdout.write(chr(i)) for i in (0,0,0,0,2,4,6,8,0,1,3,0,5,20)]' > mydata.bin
# get file size in bytes
stat -c '%s' mydata.bin
# 14
And say, I want to find the locations of all zeroes (0x00), using a grep-like syntax.
The best I can do so far is:
$ hexdump -v -e "1/1 \" %02x\n\"" mydata.bin | grep -n '00'
1: 00
2: 00
3: 00
4: 00
9: 00
12: 00
However, this implicitly converts each byte in the original binary file into a multi-byte ASCII representation, on which grep operates; not exactly the prime example of optimization :)
Is there something like a binary grep for Linux? Possibly, also, something that would support a regular expression-like syntax, but also for byte "characters" - that is, I could write something like 'a(\x00*)b' and match 'zero or more' occurrences of byte 0 between bytes 'a' (97) and 'b' (98)?
EDIT: The context is that I'm working on a driver, where I capture 8-bit data; something goes wrong in the data, which can be kilobytes up to megabytes, and I'd like to check for particular signatures and where they occur. (so far, I'm working with kilobyte snippets, so optimization is not that important - but if I start getting some errors in megabyte long captures, and I need to analyze those, my guess is I would like something more optimized :) . And especially, I'd like something where I can "grep" for a byte as a character - hexdump forces me to search strings per byte)
EDIT2: same question, different forum :) grepping through a binary file for a sequence of bytes
EDIT3: Thanks to the answer by #tchrist, here is also an example with 'grepping' and matching, and displaying results (although not quite the same question as OP):
$ perl -ln0777e 'print unpack("H*",$1), "\n", pos() while /(.....\0\0\0\xCC\0\0\0.....)/g' /path/to/myfile.bin
ca000000cb000000cc000000cd000000ce # Matched data (hex)
66357 # Offset (dec)
To have the matched data be grouped as one byte (two hex characters) each, then "H2 H2 H2 ..." needs to be specified for as many bytes are there in the matched string; as my match '.....\0\0\0\xCC\0\0\0.....' covers 17 bytes, I can write '"H2"x17' in Perl. Each of these "H2" will return a separate variable (as in a list), so join also needs to be used to add spaces between them - eventually:
$ perl -ln0777e 'print join(" ", unpack("H2 "x17,$1)), "\n", pos() while /(.....\0\0\0\xCC\0\0\0.....)/g' /path/to/myfile.bin
ca 00 00 00 cb 00 00 00 cc 00 00 00 cd 00 00 00 ce
66357
Well.. indeed Perl is very nice 'binary grepping' facility, I must admit :) As long as one learns the syntax properly :)
This seems to work for me:
grep --only-matching --byte-offset --binary --text --perl-regexp "<\x-hex pattern>" <file>
Short form:
grep -obUaP "<\x-hex pattern>" <file>
Example:
grep -obUaP "\x01\x02" /bin/grep
Output (Cygwin binary):
153: <\x01\x02>
33210: <\x01\x02>
53453: <\x01\x02>
So you can grep this again to extract offsets. But don't forget to use binary mode again.
Someone else appears to have been similarly frustrated and wrote their own tool to do it (or at least something similar): bgrep.
One-Liner Input
Here’s the shorter one-liner version:
% perl -ln0e 'print tell' < inputfile
And here's a slightly longer one-liner:
% perl -e '($/,$\) = ("\0","\n"); print tell while <STDIN>' < inputfile
The way to connect those two one-liners is by uncompiling the first one’s program:
% perl -MO=Deparse,-p -ln0e 'print tell'
BEGIN { $/ = "\000"; $\ = "\n"; }
LINE: while (defined(($_ = <ARGV>))) {
chomp($_);
print(tell);
}
Programmed Input
If you want to put that in a file instead of a calling it from the command line, here’s a somewhat more explicit version:
#!/usr/bin/env perl
use English qw[ -no_match_vars ];
$RS = "\0"; # input separator for readline, chomp
$ORS = "\n"; # output separator for print
while (<STDIN>) {
print tell();
}
And here’s the really long version:
#!/usr/bin/env perl
use strict;
use autodie; # for perl5.10 or better
use warnings qw[ FATAL all ];
use IO::Handle;
IO::Handle->input_record_separator("\0");
IO::Handle->output_record_separator("\n");
binmode(STDIN); # just in case
while (my $null_terminated = readline(STDIN)) {
# this just *past* the null we just read:
my $seek_offset = tell(STDIN);
print STDOUT $seek_offset;
}
close(STDIN);
close(STDOUT);
One-Liner Output
BTW, to create the test input file, I didn’t use your big, long Python script; I just used this simple Perl one-liner:
% perl -e 'print 0.0.0.0.2.4.6.8.0.1.3.0.5.20' > inputfile
You’ll find that Perl often winds up being 2-3 times shorter than Python to do the same job. And you don’t have to compromise on clarity; what could be simpler that the one-liner above?
Programmed Output
I know, I know. If you don’t already know the language, this might be clearer:
#!/usr/bin/env perl
#values = (
0, 0, 0, 0, 2,
4, 6, 8, 0, 1,
3, 0, 5, 20,
);
print pack("C*", #values);
although this works, too:
print chr for #values;
as does
print map { chr } #values;
Although for those who like everything all rigorous and careful and all, this might be more what you would see:
#!/usr/bin/env perl
use strict;
use warnings qw[ FATAL all ];
use autodie;
binmode(STDOUT);
my #octet_list = (
0, 0, 0, 0, 2,
4, 6, 8, 0, 1,
3, 0, 5, 20,
);
my $binary = pack("C*", #octet_list);
print STDOUT $binary;
close(STDOUT);
TMTOWTDI
Perl supports more than one way to do things so that you can pick the one that you’re most comfortable with. If this were something I planned to check in as school or work project, I would certainly select the longer, more careful versions — or at least put a comment in the shell script if I were using the one-liners.
You can find documentation for Perl on your own system. Just type
% man perl
% man perlrun
% man perlvar
% man perlfunc
etc at your shell prompt. If you want pretty-ish versions on the web instead, get the manpages for perl, perlrun, perlvar, and perlfunc from http://perldoc.perl.org.
The bbe program is a sed-like editor for binary files. See documentation.
Example with bbe:
bbe -b "/\x00\x00\xCC\x00\x00\x00/:17" -s -e "F d" -e "p h" -e "A \n" mydata.bin
11:x00 x00 xcc x00 x00 x00 xcd x00 x00 x00 xce
Explanation
-b search pattern between //. each 2 byte begin with \x (hexa notation).
-b works like this /pattern/:length (in byte) after matched pattern
-s similar to 'grep -o' suppress unmatched output
-e similar to 'sed -e' give commands
-e 'F d' display offsets before each result here: '11:'
-e 'p h' print results in hexadecimal notation
-e 'A \n' append end-of-line to each result
You can also pipe it to sed to have a cleaner output:
bbe -b "/\x00\x00\xCC\x00\x00\x00/:17" -s -e "F d" -e "p h" -e "A \n" mydata.bin | sed -e 's/x//g'
11:00 00 cc 00 00 00 cd 00 00 00 ce
Your solution with Perl from your EDIT3 give me an 'Out of memory'
error with large files.
The same problem goes with bgrep.
The only downside to bbe is that I don't know how to print context that precedes a matched pattern.
One way to solve your immediate problem using only grep is to create a file containing a single null byte. After that, grep -abo -f null_byte_file target_file will produce the following output.
0:
1:
2:
3:
8:
11:
That is of course each byte offset as requested by "-b" followed by a null byte as requested by "-o"
I'd be the first to advocate perl, but in this case there's no need to bring in the extended family.
What about grep -a? Not sure how it works on truly binary files but it works well on text files that the OS thinks is binary.