I'm trying to follow section A.1.2 of RFC 6979 and am having some difficulty.
So h1 is as follows:
h1
AF 2B DB E1 AA 9B 6E C1 E2 AD E1 D6 94 F4 1F C7
1A 83 1D 02 68 E9 89 15 62 11 3D 8A 62 AD D1 BF
If that is run through bits2octets(h1) you're supposed to get this:
01 79 5E DF 0D 54 DB 76 0F 15 6D 0D AC 04 C0 32
2B 3A 20 42 24
I don't understand how.
Here's bits2octets defined in Java (from the RFC):
private byte[] bits2octets(byte[] in)
{
BigInteger z1 = bits2int(in);
BigInteger z2 = z1.subtract(q);
return int2octets(z2.signum() < 0 ? z1 : z2);
}
Here's bits2int:
private BigInteger bits2int(byte[] in)
{
BigInteger v = new BigInteger(1, in);
int vlen = in.length * 8;
if (vlen > qlen) {
v = v.shiftRight(vlen - qlen);
}
return v;
}
Heres q:
q = 0x4000000000000000000020108A2E0CC0D99F8A5EF
h1 is 32 bytes long. q is 21 bytes long.
So bits2int returns the first 21 bytes of h1. ie.
af2bdbe1aa9b6ec1e2ade1d694f41fc71a831d0268
Convert that to an integer and then subtract q and you get this:
af2bdbe1aa9b6ec1e2ade1d694f41fc71a831d0268
- 04000000000000000000020108A2E0CC0D99F8A5EF
------------------------------------------
ab2bdbe1aa9b6ec1e2addfd58c513efb0ce9245c79
The result is positive so it - z2 - is kept.
Then int2octets() is called.
private byte[] int2octets(BigInteger v)
{
byte[] out = v.toByteArray();
if (out.length < rolen) {
byte[] out2 = new byte[rolen];
System.arraycopy(out, 0,
out2, rolen - out.length,
out.length);
return out2;
} else if (out.length > rolen) {
byte[] out2 = new byte[rolen];
System.arraycopy(out, out.length - rolen,
out2, 0, rolen);
return out2;
} else {
return out;
}
}
q and v are the same size so ab2bdbe1aa9b6ec1e2addfd58c513efb0ce9245c79
is returned. But that's not what the test vector says:
bits2octets(h1)
01 79 5E DF 0D 54 DB 76 0F 15 6D 0D AC 04 C0 32
2B 3A 20 42 24
I don't get it. Did I mess up in my analysis somewhere?
The output is obtained as (0xaf2b...d1bf >> (256 - 163)) mod q = 0x0179...4224. Your mistake was assuming bits2int shifted bytes instead of bits.
Related
I want to replace the values of columns "q1_body" and "q2_body" of dataframe "result" with the values of "body" of the same id in dataframe "df", and the code is like:
def replace_body(x):
id1 = result.loc[x].qid1
result.loc[x].q1_body = df[df["qid"]==id1]["body"]
id2 = result.loc[x].qid2
result.loc[x].q2_body = df[df["qid"]==id2]["body"]
result.index.map(lambda x: replace_body(x))
When I run the code I got the following reminder in my ipython console and the program just stuck here:
//anaconda/lib/python3.6/site-packages/pandas/core/generic.py:3110:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
self[name] = value
Hope anyone can tell me what is wrong here.
Suppose the two dataframe are:
result:
qid1 q1_body qid2 q2_body
1a abc 2a bcd
1a abc 3a cde
2a bcd 3a cde
df:
qid body
1a sfgaks
2a shdfjk
3a adjkwf
And the expected output is like:
result:
qid1 q1_body qid2 q2_body
1a sfgaks 2a shdfjk
1a sfgaks 3a adjkwf
2a shdfjk 3a adjkwf
You need map by Series created by set_index:
s = df.set_index('qid')['body']
result['q1_body'] = result['qid1'].map(s)
result['q2_body'] = result['qid2'].map(s)
print (result)
qid1 q1_body qid2 q2_body
0 1a sfgaks 2a shdfjk
1 1a sfgaks 3a adjkwf
2 2a shdfjk 3a adjkwf
Here:
# Set index and get body as a series
s = df.set_index(qid)['body']
result['q1_body'] = s.loc[result['qid1']].values
result['q2_body'] = s.loc[result['qid2']].values
Result:
qid1 q1_body qid2 q2_body
0 1a sfgaks 2a shdfjk
1 1a sfgaks 3a adjkwf
2 2a shdfjk 3a adjkwf
Timing (10k rows, using auto-generated Lorem):
My method
#Jezareal's method
I am using Perl v5.16.2
I am using the Net::SMPP modules and it returns me some data.
If I show this data, I get this (simplified) :
$VAR1 = bless( {
'receipted_message_id' => '400002F6E09C61701222120140',
'30' => '400002F6E09C61701222120140'
}, 'Net::SMPP::PDU' );
Now, let's assume this data is in $pdu and I do this :
$message_id = $pdu->{30}; # or $pdu->{receipted_message_id}, same result
myfunction($message_id);
Then, I have myfunction defined as :
sub myfunction {
my $message_id = shift;
my $message_id_static = '400002F6E09C61701222120140';
print Dumper($message_id);
print Dumper($message_id_static);
print hexdump($message_id);
print hexdump($message_id_static);
if ($message_id eq $message_id_static)
{
print "match\n";
}
else
{
print "no match\n";
}
}
The output of the program is :
$VAR1 = '400002F6E09C61701222120140';
$VAR1 = '400002F6E09C61701222120140';
Data::Hexdumper: data length isn't an integer multiple of lines
so has been padded with NULLs at the end.
0x0000 : 34 30 30 30 30 32 46 36 45 30 39 43 36 31 37 30 : 400002F6E09C6170
0x0010 : 31 32 32 32 31 32 30 31 34 30 00 00 00 00 00 00 : 1222120140......
Data::Hexdumper: data length isn't an integer multiple of lines
so has been padded with NULLs at the end.
0x0000 : 34 30 30 30 30 32 46 36 45 30 39 43 36 31 37 30 : 400002F6E09C6170
0x0010 : 31 32 32 32 31 32 30 31 34 30 00 00 00 00 00 00 : 1222120140......
no match
Which doesn't make any sense to me... !
If I try to use $message_id to do a SQLite query, it fails miserably. If I use $message_id_static instead, it works perfectly.
So, is this a weird internal Perl bug, or am I missing something ?
This has been driving me nuts for hours...
EDIT :
Using the perl debugger, I get this :
DB<3> x $message_id_static
0 '400002F6E09C61701222120140'
DB<4> x $message_id
0 "400002F6E09C61701222120140\c#"
So at least I see there is a difference in the strings, but why isn't it seen by the hexdump, and what is that \c# ?
Thanks !
The \c# character is Ctrl-#, which is the ASCII NUL character at code point zero
You can't see it in your hexdump output because it is indistinguishable from the 00 padding at the end of the dump
If you set $Data::Dumper::Useqq = 1 then it will be visible in the output from print Dumper $message_id
You can remove it from the variable by using s/\0\z// or tr/\0//d, but you should really investigate why it is there in the first place
Since fpu operation is very costly, I would like to less use fpu operations as far as I can. In the meanwhile, I'm wondering what kind of operations within the float variable would counted as the operation that fpu involves?
Such as the following code would involve fpu unit?
struct my_float_struct {
int f;
} g;
void func(float a)
{
g.f = a;
}
would calling func will cause lazy FPU context switch?
In kernel you shouldn't use float at all (use lookup tables and multiplying by coefficients to save precision on divide operations instead). In user space all operations involving float type on RHS would lead to FPU instruction usage. If you have 2 integers on RHS but assigning result to float (on LHS) -- you will end up with integer operations, but also with casting to float instruction (like
cvtsi2ss).
It's actually very easy to answer your question just looking into disassemble for corresponding C code.
C code (build with gcc -Wall -O0 -g main.c):
int main(void)
{
volatile float f1, f2;
volatile int i;
f1 = i * 356;
f2 = f1 * f2;
return 0;
}
Disassemble (using objdump -DS a.out):
int main(void)
{
4004b6: 55 push %rbp
4004b7: 48 89 e5 mov %rsp,%rbp
volatile float f1, f2;
volatile int i;
f1 = i * 356;
4004ba: 8b 45 f4 mov -0xc(%rbp),%eax
4004bd: 69 c0 64 01 00 00 imul $0x164,%eax,%eax
4004c3: 66 0f ef d2 pxor %xmm2,%xmm2
4004c7: f3 0f 2a d0 cvtsi2ss %eax,%xmm2
4004cb: 66 0f 7e d0 movd %xmm2,%eax
4004cf: 89 45 fc mov %eax,-0x4(%rbp)
f2 = f1 * f2;
4004d2: f3 0f 10 4d fc movss -0x4(%rbp),%xmm1
4004d7: f3 0f 10 45 f8 movss -0x8(%rbp),%xmm0
4004dc: f3 0f 59 c8 mulss %xmm0,%xmm1
4004e0: 66 0f 7e c8 movd %xmm1,%eax
4004e4: 89 45 f8 mov %eax,-0x8(%rbp)
return 0;
4004e7: b8 00 00 00 00 mov $0x0,%eax
}
From here you can see that:
when multiplying integer by integer, but assigning result to float, it leads to imul instruction (integer multiplying), but also to cvtsi2ss (converting to float) instruction.
when multiplying 2 floats, it leads to mulss instruction, which float multiplying
Basically you can see immediately if FPU involved by instruction operates on FPU registers, like %xmm0, %xmm1, etc.
I have a binary data file and need to retrieve some data from it. From trial and error and the help of a hex editor, I have identified the regions of text that I need, but I'm not sure what encoding is being used.
Each character is using two bytes, but in my sample set the second byte is always empty.
1F00 : a
1C00 : b
1A00 : d
1B00 : e
1900 : g
1600 : h
1700 : i
1500 : k
1200 : l
1000 : n
1100 : o
0E00 : p
0F00 : q
0C00 : r
0D00 : s
0A00 : t
0B00 : u
0800 : v
0900 : w
5000 : .
5E00 : <- space
3F00 : A
3C00 : B
3D00 : C
3A00 : D
3B00 : E
2D00 : S
for example, the word hello is represented as
16 00 1B 00 12 00 12 00 11 00
Obviously the weird thing is that 0x41 is not A, and that the alphabet is not even consecutive. It is possible that some weird cypher was being used, but I doubt it.
Joop Eggen found the solution below - a simple xor!
You probably saw it already, but one can see the xorring.
This is a poor man's encryption, every char as int:
code = (plain ^ 0x7e) << 8
If you on linux try to use enca (it detects character set and encoding of text files and can also convert them to other encodings).
There is definitely a pattern:
04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F
a x a
b x b
c x c
d x d
e x e
f x f
g x g
h x h
i x i
j x j
k x k
l x l
m x m
n x n
o x o
p x p
q x q
r x r
s x s
t x t
u x u
v x v
w x w
x x x
y x y
z x z
04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F
I have a binary file , the definition of its content is as below : ( all data is stored
in little endian (ie. least significant byte first)) . The example numbers below are HEX
11 63 39 46 --- Time, UTC in seconds since 1 Jan 1970.
01 00 --- 0001 = No Fix, 0002 = SPS
97 85 ff e0 7b db 4c 40 --- Latitude, as double
a1 d5 ce 56 8d 26 28 40 --- Longitude, as double
f0 37 e1 42 --- Height in meters, as float
fe 2b f0 3a --- Speed in km/h, as float
00 00 00 00 --- Heading (degrees ?), as float
01 00 --- RCR, log reason. 0001=Time, 0004=Distance
59 20 6a f3 4a 26 e3 3f --- Distance in meters, as double,
2a --- ? Don't know
a8 --- Checksum, xor of all bytes above not including 0x2a
the data from the Binary file "in HEX" is as below
"F25D39460200269652F5032445401F4228D79BCC54C09A3A2743B4ADE73F2A83"
I appreciate if you can support me to translate this data line based on the instruction before.
Probably wrong, but here's a shot at it using Ruby:
hex = "F25D39460200269652F5032445401F4228D79BCC54C09A3A2743B4ADE73F2A83"
ints = hex.scan(/../).map{ |s| s.to_i(16) }
raw = ints.pack('C*')
fields = raw.unpack( 'VvEEVVVvE')
p fields
#=> [1178164722, 2, 42.2813707974677, -83.1970117467067, 1126644378, 1072147892, nil, 33578, nil]
p Time.at( fields.first )
#=> 2007-05-02 21:58:42 -0600
I'd appreciate it if someone well-versed in #pack and #unpack would show me a better way to accomplish the first three lines.
My Cygnus Hex Editor could load such a file and, using structure templates, display the data in its native formats.
Beyond that, it's just a matter of doing through each value and working out the translation for each byte.