UnicodeDecodeError in decoding the torrent file - python-3.x

Question

I am trying to write the torrent app from scratch just for learning purpose.So after few hours of reading wiki, I wrote some code for decoding of torrent files which use 'Bencoding' "https://en.wikipedia.org/wiki/Bencode" . But unfortunately, I didn't notice about byte string and python string. My code work fine with python string like torrent data, but when I passed torrent byte data, I got the encoding error.
I tried " open(file, 'rb',encoding='utf-8', errors='ignore') ". It did change the byte string into python string.I also tried all the available answers on stakoverflow. But some data were lost as errors and so I can't properly decode the torrent data. Pardon my messy coding and please help... I also read the bencoder library, and it works directly on byte string, so if there are any way that I don't have to re-write the code, please...
with open(torrent_file1, 'rb') as _file:
data = _file.read()
def int_decode(meta, cur):
print('inside int_decode function')
cursor = cur
start = cursor + '1'
end = start
while meta[end] != 'e':
end += 1
value = int(meta[start:end])
cursor = end + 1
print(value, cursor)
return value, cursor
def chr_decode(meta, cur):
print('inside chr_decode function')
cursor = cur
start = cursor
end = start
while meta[end] != ':':
end += 1
chr_len = int(meta[start:end])
chr_start = end + 1
chr_end = chr_start + chr_len
value = meta[chr_start:chr_end]
cursor = chr_end
print(value, cursor)
return value, cursor
def list_decode(meta, cur):
print('inside the list decoding')
cursor = cur+1
new_list = list()
while cursor < (len(meta)):
if meta[cursor] == 'i':
item, cursor = int_decode(meta, cursor)
new_list.append(item)
elif meta[cursor].isdigit():
item, cursor = chr_decode(meta, cursor)
new_list.append(item)
elif meta[cursor] == 'e':
print('list is ended')
cursor += 1
break
return (new_list,cursor)
def dict_decode(meta, cur=0, key_=False, key_val=None):
if meta[cur] == 'd':
print('dict found')
new_dict = dict()
key = key_
key_value = key_val
cursor = cur + 1
while cursor < (len(meta)):
if meta[cursor] == 'i':
value, cursor = int_decode(meta, cursor)
if not key:
key = True
key_value = value
else:
new_dict[key_value] = value
key = False
elif meta[cursor].isdigit():
value, cursor = chr_decode(meta, cursor)
if not key:
key = True
key_value = value
else:
new_dict[key_value] = value
key = False
elif meta[cursor] == 'l':
lists, cursor = list_decode(meta, cursor)
if key:
new_dict[key_value] = lists
key = False
else:
print('list cannot be used as key')
elif meta[cursor] == 'd':
dicts, cursor = dict_decode(meta, cursor)
if not key:
key=True
key_value = dicts
else:
new_dict[key_value] = dicts
key=False
elif meta[cursor] == 'e':
print('dict is ended')
cursor += 1
break
return (new_dict,cursor)
test = 'di323e4:spami23e4:spam5:helloi23e4:spami232ei232eli32e4:doneei23eli1ei2ei3e4:harmee'
test2 = 'di12eli23ei2ei22e5:helloei12eli1ei2ei3eee'
test3 = 'di12eli23ei2ei22ee4:johndi12e3:dggee'
print(len(test2))
new_dict = dict_decode(data)
print(new_dict)
Traceback (most recent call last):
File "C:\Users\yewaiyanoo\Desktop\python\torrent\read_torrent.py", line 8, in
data = _file.read()
File "C:\Users\yewaiyanoo\AppData\Local\Programs\Python\Python37-32\lib\codecs.py", line 701, in read
return self.reader.read(size)
File "C:\Users\yewaiyanoo\AppData\Local\Programs\Python\Python37-32\lib\codecs.py", line 504, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 204: invalid start byte

Answer 1

Python 3 decodes text files when reading, encodes when writing. The default encoding is taken from locale.getpreferredencoding(False), which evidently for your setup returns 'ASCII'. See the open() function documenation:
In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.
Instead of relying on a system setting, you should open your text files using an explicit codec:
currentFile = open(file, 'rb',encoding='latin1', errors='ignore')
where you set the encoding parameter to match the file you are reading.
Python 3 supports UTF-8 as the default for source code.
The same applies to writing to a writeable text file; data written will be encoded, and if you rely on the system encoding you are liable to get UnicodeEncodingError exceptions unless you explicitly set a suitable codec. What codec to use when writing depends on what text you are writing and what you plan to do with the file afterward.
You may want to read up on Python 3 and Unicode in the Unicode HOWTO, which explains both about source code encoding and reading and writing Unicode data.