Trying to decode a bytes array in Python 3.8 where the sending interface spec states the format is "float (serialized as double)".
Using a sample bytes array received from the interface, this is the result i get
>>> import struct
>>> bytes_arr = b'\xbf\xe9\x99\x99\xa0\x00\x00\x00'
>>> print(struct.unpack('d', bytes_arr))
(3.40792534166e-312,)
The parent application (displaying the same data i am receiving over the interface) states the answer to be -0.8. This is obviously rounded but a different decode.
Where am i going wrong?
Your value is in big-endian format:
>>> print(struct.unpack('>d', bytes_arr))
(-0.800000011920929,)
Related
I want to convert 16-digit hexadecimal numbers into doubles. I actually did the reverse of this before which worked fine:
import struct
import wrap
def double_to_hex(doublein):
return hex(struct.unpack('<Q', struct.pack('<d', doublein))[0])
for i in modified_list:
encoded_list.append(double_to_hex(i))
modified_list.clear()
encoded_msg = ''.join(encoded_list).replace('0x', '')
encoded_list.clear()
print_command('encode', encoded_message)
And now I want to sort of do the reverse. I tried this without success:
from textwrap import wrap
import struct
import binascii
MESSAGE = 'c030a85d52ae57eac0129263c4fffc34'
#Splitting up message into n 16-bit strings
MSGLIST = wrap(MESSAGE, 16)
doubles = []
print(MSGLIST)
for msg in MSGLIST:
doubles.append(struct.unpack('d', binascii.unhexlify(msg)))
print(doubles)
However, when I run this, I get crazy values, which are of course not what I put in:
[(-1.8561629252326087e+204,), (1.8922789420412524e-53,)]
Were your original numbers -16.657673995556173 and -4.642958715557189 ?
If so, then the problem is that your hex strings contain the big-endian (most-significant byte first) representation of the double, but the 'd' format string in your unpack call specifies conversion using your system's native format, which happens to be little-endian (least-significant byte first). The result is that unpack reads and processes the bytes of the unhexlify'ed string from the wrong end. Unsurprisingly, that will produce the wrong value.
To fix, do one of:
convert the hex string into little-endian format (reverse the bytes, so c030a85d52ae57ea becomes ea57ae525da830c0) before passing it to binascii.unhexlify, or
reverse the bytes produced by unhexlify (change binascii.unhexlify(msg) to binascii.unhexlify(msg)[::-1]) before you pass them to unpack, or
tell unpack to do the conversion using big-endian order (replace the format string 'd' with '>d')
I'd go with the last one, replacing the format string.
In python3.5 I need to convert the string to IPFIX supported field value for UDP packet. While I am sending string bytes as UDP packet I am unable to recover the string data again. In Wireshark, it says that "Malformed data".
I found that IPFIX supports only the "ASCII" for strings. So I have converted ASCII value to hex and then converted into bytes. But while converting hex("4B") to byte. I am not getting my hex value in bytes instead of I am getting the string in bytes("K").
I have tried the following in the python console. I need exact byte what I have entered. But it seems like b'\x4B' instead of '\x4B' I am getting 'K'. I am using python3.5
b'\x4B'
b'K'
Code: "K".encode("ascii")
Actual OP: b'K'
Expected OP: b'\x4B'
There are multiple ways to do this:
1. The hex method (python 3.5 and up)
>>> 'K'.encode('ascii').hex()
'4b' # type str
2. Using binascii
>>> binascii.hexlify('K'.encode('ascii'))
b'4b' # type bytes
3. Using str.format
>>> ''.join('{:02x}'.format(x) for x in 'K'.encode('ascii'))
'4b' # type str
4. Using format
>>> ''.join(format(x, '02x') for x in 'K'.encode('ascii'))
'4b' # type str
Note: Methods using format are not very performance efficient.
If you really care about the \x you will have to use format, eg:
>>> print(''.join('\\x{:02x}'.format(x) for x in 'K'.encode('ascii')))
\x4b
>>> print(''.join('\\x{:02x}'.format(x) for x in 'KK'.encode('ascii')))
\x4b\x4b
If you care about uppercase then you can use X instead of x, eg:
>>> ''.join('{:02X}'.format(x) for x in 'K'.encode('ascii'))
'4B'
>>> ''.join(format(x, '02X') for x in 'K'.encode('ascii'))
'4B'
Uppercase and with \x:
>>> print(''.join('\\x{:02X}'.format(x) for x in 'Hello'.encode('ascii')))
\x48\x65\x6C\x6C\x6F
If you want bytes instead of str then just encode the output to ascii again:
>>> print(''.join('\\x{:02X}'.format(x) for x in 'Hello'.encode('ascii')).encode('ascii'))
b'\\x48\\x65\\x6C\\x6C\\x6F'
How do I print a bytes string without the b' prefix in Python 3?
>>> print(b'hello')
b'hello'
Use decode:
>>> print(b'hello'.decode())
hello
If the bytes use an appropriate character encoding already; you could print them directly:
sys.stdout.buffer.write(data)
or
nwritten = os.write(sys.stdout.fileno(), data) # NOTE: it may write less than len(data) bytes
If the data is in an UTF-8 compatible format, you can convert the bytes to a string.
>>> print(str(b"hello", "utf-8"))
hello
Optionally, convert to hex first if the data is not UTF-8 compatible (e.g. data is raw bytes).
>>> from binascii import hexlify
>>> print(hexlify(b"\x13\x37"))
b'1337'
>>> print(str(hexlify(b"\x13\x37"), "utf-8"))
1337
>>> from codecs import encode # alternative
>>> print(str(encode(b"\x13\x37", "hex"), "utf-8"))
1337
According to the source for bytes.__repr__, the b'' is baked into the method.
One workaround is to manually slice off the b'' from the resulting repr():
>>> x = b'\x01\x02\x03\x04'
>>> print(repr(x))
b'\x01\x02\x03\x04'
>>> print(repr(x)[2:-1])
\x01\x02\x03\x04
To show or print:
<byte_object>.decode("utf-8")
To encode or save:
<str_object>.encode('utf-8')
I am a little late but for Python 3.9.1 this worked for me and removed the -b prefix:
print(outputCode.decode())
It's so simple...
(With that, you can encode the dictionary and list bytes, then you can stringify it using json.dump / json.dumps)
You just need use base64
import base64
data = b"Hello world!" # Bytes
data = base64.b64encode(data).decode() # Returns a base64 string, which can be decoded without error.
print(data)
There are bytes that cannot be decoded by default(pictures are an example), so base64 will encode those bytes into bytes that can be decoded to string, to retrieve the bytes just use
data = base64.b64decode(data.encode())
Use decode() instead of encode() for converting bytes to a string.
>>> import curses
>>> print(curses.version.decode())
2.2
Iam working in python 3.6
I receive from serial comunication an string '3F8E353F'. This is a float number 1.111. How can I convert this string to a float number?
Thank you
Ah yes. Since this is 32-bits, unpack it into an int first then:
x='3F8E353F'
struct.unpack('f',struct.pack('i',int(x,16)))
On my system this gives:
>>> x='3F8E353F'
>>> struct.unpack('f',struct.pack('i',int(x,16)))
(1.1109999418258667,)
>>>
Very close to the expected value. However, this can give 'backwards' results based on the 'endianness' of bytes in your system. Some systems store their bytes least significant byte first, others most significant byte first. See this reference page for the descriptors to format based on byte order.
I used struct.unpack('f',struct.pack('i',int(x,16))) to convert Hex value to float but for negative value I am getting below error
struct.error: argument out of range
To resolve this I used below code which converts Hex (0xc395aa3d) to float value (-299.33). It works for both positive as well for negative value.
x = 0xc395aa3d
struct.unpack('f', struct.pack('I', int(x,16) ))
Another way is to use bytes.fromhex()
import struct
hexstring = '3F8E353F'
struct.unpack('!f', bytes.fromhex(hexstring))[0]
#answer:1.1109999418258667
Note: The form '!' is available for those poor souls who claim they can’t remember whether network byte order is big-endian or little-endian (from struct docs).
Do you have any idea how to encode and decode a float number with base64 in Python.
I am trying to use
response='64.000000'
base64.b64decode(response)
the expected output is 'AAAAAAAALkA=' but i do not get any output for float numbers.
Thank you.
Base64 encoding is only defined for byte strings, so you have to convert your number into a sequence of bytes using struct.pack and then base64 encode that. The example you give looks like a base64 encoded little-endian double. So (for Python 2):
>>> import struct
>>> struct.pack('<d', 64.0).encode('base64')
'AAAAAAAAUEA=\n'
For the reverse direction you base64 decode and then unpack it:
>>> struct.unpack('<d', 'AAAAAAAALkA='.decode('base64'))
(15.0,)
So it looks like your example is 15.0 rather than 64.0.
For Python 3 you need to also use the base64 module:
>>> import struct
>>> import base64
>>> base64.encodebytes(struct.pack('<d', 64.0))
b'AAAAAAAAUEA=\n'
>>> struct.unpack('<d', base64.decodebytes(b'AAAAAAAALkA='))
(15.0,)