I'm converting strings to floats using float(x). However for some reason, one of the strings is "71.2\x0060". I've tried following this answer, but it does not remove the bytes character
>>> s = "71.2\x0060"
>>> "".join([x for x in s if ord(x) < 127])
'71.2\x0060'
Other methods I've tried are:
>>> s.split("\\x")
['71.2\x0060']
>>> s.split("\x")
ValueError: invalid \x escape
I'm not sure why this string is not formatted correctly, but I'd like to get as much precision from this string and move on.
Going off of wim's comment, the answer might be this:
>>> s.split("\x00")
['71.2', '60']
So I should do:
>>> float(s.split("\x00")[0])
71.2
Unfortunately the POSIX group \p{XDigit} does not exist in the re module. To remove the hex control characters with regular expressions anyway, you can try the following.
impore re
re.sub(r'[\x00-\x1F]', r'', '71.2\x0060') # or:
re.sub(r'\\x[0-9a-fA-F]{2}', r'', r'71.2\x0060')
Output:
'71.260'
'71.260'
r means raw. Take a look at the control characters up to hex 1F in the ASCII table: https://www.torsten-horn.de/techdocs/ascii.htm
Related
In a situation I had to store data as utf-8 and now when I want to fetch and decode('utf-8') data it's just simply does not work. Consider line below as an example:
\x0d\x0a\xd8\xb3\xd8\xa7\xd9\x82\xdb\x8c\xe2\x80\x8c\xd9\x86\xd8\xa7\xd9\x85\xd9\x87
You can simply copy the line below to convert the string above to the human readable format:
b"\x0d\x0a\xd8\xb3\xd8\xa7\xd9\x82\xdb\x8c\xe2\x80\x8c\xd9\x86\xd8\xa7\xd9\x85\xd9\x87".decode("utf-8")
However could not find a way to convert the string to bytestring without corrupting the string. I tried following methods but all of them failed:
.decode("utf-8")
.decode()
.bytes()
Up until this point I could not find solution in OS or other places. Appreciate any help.
x0d\x0a\xd8\xb3\xd8\xa7\xd9\x82\xdb\x8c\xe2\x80\x8c\xd9\x86\xd8\xa7\xd9\x85\xd9\x87
b'x0d\x0a\xd8\xb3\xd8\xa7\xd9\x82\xdb\x8c\xe2\x80\x8c\xd9\x86\xd8\xa7\xd9\x85\xd9\x87'
The above lines (both given in the question) are particular instances of String and Bytes literals (respectively):
\xhh Character with hex value hh (2, 3)
2 Unlike in Standard C, exactly two hex digits are
required.
3 In a bytes literal, hexadecimal and octal escapes denote
the byte with the given value. In a string literal, these escapes
denote a Unicode character with the given value.
Let's check the string defined in such a way (inside Python prompt):
>>> xstr = "\x0d\x0a\xd8\xb3\xd8\xa7\xd9\x82\xdb\x8c\xe2\x80\x8c\xd9\x86\xd8\xa7\xd9\x85\xd9\x87"
>>> xstr
'\r\nساÙ\x82Û\x8câ\x80\x8cÙ\x86اÙ\x85Ù\x87'
>>> print( xstr)
ساÙÛâÙاÙ
Ù
>>>
Apparently, the print( xstr) output does not resemble a word in any known language however all its characters belong (by definition) to Unicode range r'[\u0000-\u00ff]' i.e. the first 256 of characters in Unicode, and voila - it's iso-8859-1 aka 'latin1'.
We need to get an encoded version of the xstr string as a bytes object, e.g. using str.encode method or built-in bytes() function. Then
print( bytes(xstr,'latin1').decode()); print(xstr.encode("latin1").decode())
ساقینامه
ساقینامه
I am trying to convert hex numbers into decimals using unpack.
When I use:
from struct import *
unpack("<H",b"\xe2\x07")
The output is: 2018, which is what I want.
The thing is I have my hex data in a list as a string in the form of:
asd = ['e2','07']
My question is is there a simple way of using unpack without the backslashes, the x? Something like so:
unpack("<H","e207")
I know this doesn't work but I hope you get the idea.
For clarification I know I could get the data in the form of b'\x11' in the list but then it's interpreted as ASCII, which I don't want, that's why I have it in the format I showed.
You have hex-encoded data, in a text object. So, to go back to raw hex bytes, you can decode the text string. Please note that this is not the usual convention in Python 3.x (generally, text strings are already decoded).
>>> codecs.decode('e207', 'hex')
b'\xe2\x07'
A convenience function for the same thing:
>>> bytes.fromhex('e207')
b'\xe2\x07'
Now you can struct.unpack those bytes. Putting it all together:
>>> asd = ['e2','07']
>>> text = ''.join(asd)
>>> encoded = codecs.decode(text, 'hex')
>>> struct.unpack("<H", encoded)
(2018,)
Get unicode point of character Ä.
Python3 version.
>>> str="Ä"
>>> str.encode("unicode-escape")
b'\\xc4'
How to get the single backslash format b'\xc4' instead of b'\\xc4' as my output ?
It's not entirely clear to me what you want, so I'll give you a few options.
Get the (Unicode) code point of a character as an integer:
>>> ord('Ä')
196
Display the integer in hex notation:
>>> hex(ord('Ä'))
'0xc4'
or with string formatting:
>>> '{:X}'.format(ord('Ä'))
'C4'
However, you talk about backslashes and show the bytestring b'\xc4'.
This is the Latin-1 encoding of 'Ä' (all characters with a Unicode codepoint below 256 can be encoded with Latin-1, and their byte value equals the Unicode codepoint).
>>> 'Ä'.encode('latin-1')
b'\xc4'
This is a bytestring of length 1.
It is displayed in a way in which you could type this character, ie. using an escape sequence with backslash-x and a two-digit hex number.
The "unicode-escape" codec produces these four ASCII characters (\, x, c 4), but not as str, but as a bytes object (because str.encode() returns bytes by definition).
To get a backslash in a str/bytes literal, you need to type two backslashes, so the representation form also uses two backslashes:
>>> 'Ä'.encode('unicode-escape')
b'\\xc4'
The "unicode-escape" codec is very Python-specific and I don't see a lot of applications; maybe if you want to write your own pickle protocol or parse fragments of Python source code.
I have a string looking like this:
"\\xd6\\x83\\x8dd!VT\\x92\\xaaA\\x05\\xe0\\x9b\\x8b\\xf1"
and I want to remove the double escape characters in order to make it a proper binary. Is that even possible?
The source string looks pretty much like a bytes string, so you could do:
>>> import ast
>>> s = "\\xd6\\x83\\x8dd!VT\\x92\\xaaA\\x05\\xe0\\x9b\\x8b\\xf1"
>>> print(ast.literal_eval("b'''%s'''" % s))
b'\xd6\x83\x8dd!VT\x92\xaaA\x05\xe0\x9b\x8b\xf1'
I have a string like so:
>>> t
'\\u0048\\u0065\\u006c\\u006c\\u006f\\u0020\\u20ac\\u0020\\u00b0'
That I made using a function that converts unicode to the representative Python escape sequences. Then, when I want to convert it back, I can't get rid of the double backslash so that it is interpreted as unicode again. How can this be done?
>>> t = unicode_encode("
>>> t
'\\u0048\\u0065\\u006c\\u006c\\u006f\\u0020\\u20ac\\u0020\\u00b0'
>>> print(t)
\u0048\u0065\u006c\u006c\u006f\u0020\u20ac\u0020\u00b0
>>> t.replace('\\','X')
'Xu0048Xu0065Xu006cXu006cXu006fXu0020Xu20acXu0020Xu00b0'
>>> t.replace('\\', '\\')
'\\u0048\\u0065\\u006c\\u006c\\u006f\\u0020\\u20ac\\u0020\\u00b0'
Of course, I can't do this, either:
>>> t.replace('\\', '\')
File "<ipython-input-155-b46c447d6c3d>", line 1
t.replace('\\', '\')
^
SyntaxError: EOL while scanning string literal
Not sure if this is appropriate for your situation, but you could try using unicode_escape:
>>> t
'\\u0048\\u0065\\u006c\\u006c\\u006f\\u0020\\u20ac\\u0020\\u00b0'
>>> type(t)
<class 'str'>
>>> enc_t = t.encode('utf_8')
>>> enc_t
b'\\u0048\\u0065\\u006c\\u006c\\u006f\\u0020\\u20ac\\u0020\\u00b0'
>>> type(enc_t)
<class 'bytes'>
>>> dec_t = enc_t.decode('unicode_escape')
>>> type(dec_t)
<class 'str'>
>>> dec_t
'Hello € °'
Or in abbreviated form:
>>> t.encode('utf_8').decode('unicode_escape')
'Hello € °'
You take your string and encode it using UTF-8, and then decode it using unicode_escape.
Since a backslash is an escape character and you are searching for two backslashes you need to replace four backslashes with two - i.e.:
t.replace("\\\\", "\\")
This will replace every r"\\" with r"\". The r indicates raw string. So, for example, if you type print(r"\\") into idle or any python script (or print r"\\" in Python 2) you will get \\\\. This means that every "\\" is really just a r"\".
user1632861 suggested that you use .replace("\\", ""), but this replaces ever r"\" with nothing. Try the above method instead. :D
In this case, however, it appears as though you are reading/receiving data, and you probably want to use the correct encoding and then decode to unicode (as the person above me suggested).
You only got one backslash in your code, but backslashes are represent as \\. As you can see, when you use print(), there's only one backslash. So if you want to get rid of one of the two backslashes, don't do anything, it's not there. If you wanna get rid of both, just remove one. Again use \\ to represent one backslash: t.replace("\\", "")
So your string never has two backslashes in the first place, it shouldn't be the problem.