Exception in Python Trace Module: "Charmap Can't Encode..." - python-3.x

Question

I am using the python trace module in order to figure out why my program is exiting early.
However, there seems to be a bug with the trace module itself.
When i run py -m trace -t src/main.py > temp/trace.txt in powershell, I get the following error:
Traceback (most recent call last):
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\trace.py", line 740, in <module>
main()
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\trace.py", line 728, in main
t.runctx(code, globs, globs)
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\trace.py", line 450, in runctx
exec(cmd, globals, locals)
File "src/main.py", line 11, in <module>
from project_core.image_processing import *
File "G:\My Drive\Swamynathan Lab Image Processing\ImageProcessingProjectLatestCodeWithGitVersionTracking\project\project_core\image_processing.py", line 7, in <module>
import numpy as np
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\__init__.py", line 154, in <module>
from . import polynomial
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\polynomial\__init__.py", line 116, in <module>
from .polynomial import Polynomial
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\polynomial\polynomial.py", line 87, in <module>
from ._polybase import ABCPolyBase
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\polynomial\_polybase.py", line 18, in <module>
class ABCPolyBase(abc.ABC):
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\polynomial\_polybase.py", line 73, in ABCPolyBase
"0": "⁰",
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\polynomial\_polybase.py", line 73, in ABCPolyBase
"0": "⁰",
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\trace.py", line 575, in localtrace_trace
print("%s(%d): %s" % (bname, lineno,
File "C:\Users\micha\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2070' in position 32: character maps to <undefined>
Is there any straightforward way to fix this problem?

Answer 1

The problem is that the trace module is opening a file with the default encoding for your Python implementation. On Windows this is often cp1252 which you can see in the error dump. CP1252 is incapable of encoding every possible Unicode character, so sometimes you'll run into this problem.
Python will use the environment variable PYTHONIOENCODING to override the default. If you set it before you start Python to something that can encode all the Unicode characters such as utf-8 or utf-8-sig it will eliminate the error. Unfortunately this will affect all other text files opened by your program.