I use python 3.9.1 on macOS Big Sur with an M1 chip.
I would like to open the grib format file which is provided by Japan Meteorological Agency.
So, I tried to use the pygrib library as below:
import pygrib
gpv_file = pygrib.open("Z__C_RJTD_20171205000000_GSM_GPV_Rjp_Lsurf_FD0000-0312_grib2.bin")
But I got the error like this:
----> 1 gpv_file = pygrib.open("Z__C_RJTD_20171205000000_GSM_GPV_Rjp_Lsurf_FD0000-0312_grib2.bin")
pygrib/_pygrib.pyx in pygrib._pygrib.open.__cinit__()
pygrib/_pygrib.pyx in pygrib._pygrib._strencode()
UnicodeEncodeError: 'ascii' codec can't encode characters in position 50-51: ordinal not in range(128)
I asked other people to run the same code, and it somehow worked.
I am not sure what the problem is and how to fix it.
Related
The Python 3.4 and Python 3.8/3.9 are different when I try execute below statement:
print('\u212B')
Python 3.8/3.9 can print it correctly.
Å
Python 3.4 will report an exception:
Traceback (most recent call last):
File "test.py", line 9, in <module>
print('\u212B')
UnicodeEncodeError: 'gbk' codec can't encode character '\u212b' in position 0: illegal multibyte sequence
And according to this page, I can avoid the exception by overwrite sys.stdout via statement:
sys.stdout = io.TextIOWrapper(buffer=sys.stdout.buffer,encoding='utf-8')
But python 3.4 still print different charactor as below:
鈩?
So my questions are:
Why do different python versions have different behaviors on stand output print?
How can I print correct value Å in python 3.4?
Edit 1:
I guess the difference is caused by PEP 528 -- Change Windows console encoding to UTF-8. But I still don't understand the machanism of console encoding and how I can print correct character in Python 3.4.
Edit 2:
One more difference, sys.getfilesystemencoding() will get utf-8 in Python 3.8/3.9 and get mbcs in Python 3.4.
Why?
Regarding the rationale behind the stdout encoding you can read more in the answers here: Changing default encoding of Python?
In short, Python 3.4 is using your OS's encoding by default as the one for stdout whereas with Python 3.8 it is set to UTF-8.
How to fix this?
You can use a new method - reconfigure introduced with Python 3.7:
sys.stdout.reconfigure(encoding='utf-8')
Typically, you can try setting the environment variable PYTHONIOENCODING to utf-8:
set PYTHONIOENCODING=utf8
in most of the operating systems except Windows where another environment variable must be set for it to work:
set PYTHONLEGACYWINDOWSIOENCODING=1
You can fix it in the version of Python preceding v. 3.7 via installing win-unicode-console package that handles UTF issues transparently on Windows:
pip install win-unicode-console
If you are not running the code directly from a console there is a possibility that your IDE configuration is interfering.
I try to use Python 3.8.3 on Windows 10 with Pip 20.1.1
I would like to install and use Mitre Attack Joystick Master get from this :
https://github.com/mitre-attack/joystick
I check all file is on Utf-8, using StringGenerator 0.3.0, but nothing works every time the same error when I try to launch the command
error return
I'm trying to read a .spydata file into Spyder, which was written in a different platform (and probably with a different encoding), but spyder gives an error :
'ascii' codec can't decode byte 0xb5 in position 2: ordinal not in range(128)
I tried changing my encoding setting before loading spyder without success. Any ideas?
I fixed the encoding problem with this line on top of the file:
# -*- coding: utf-8 -*-
https://python.readthedocs.io/en/stable/howto/unicode.html
You can change de encoding if you need too. But I don't know if you can include the encoding line on the file you mentioned. Hope It helps.
I am using PyTorch with python3.
I tried the following while in ipdb mode:
regions = np.zeros([107,4], dtype='uint8')
torch.from_numpy(regions)
This prints the tensor.
However when trying:
regions = np.zeros([107,107,4], dtype='uint8')
torch.from_numpy(regions)
I get the following error:
*** UnicodeEncodeError: 'ascii' codec can't encode character '\u22ee' in position 72: ordinal not in range(128)
I'm am using:
numpy==1.11.3
torch==0.2.0.post4
torchvision==0.1.9
and python3.5.3
I'm able to run both versions of your code with no issues in a jupyter notebook. I'm gonna say that this kind of Unicode codec error is not due to .from_numpy()
I'm running Python 3.5.2 |Anaconda custom (x86_64)
numpy==1.13.0
torch==0.1.12.post2
torchvision==0.1.8
From the website http://pytorch.org/ they say:
Please ensure that you are on the latest pip and numpy packages.
I was not able to reproduce your errors with recent versions of numpy and pytorch. I think pip install -U numpy might solve the problem.
I'm using Linux Mint 18.1 and Python 3.5.2.
I have a library that currently works using Python 2.7. I need to use the library for a Python 3 project. I'm updating it and have run into a unicode problem that I can't seem to fix.
First, a file is created via tar cvjf tarfile.tbz2 (on a Linux system) and is later opened in the Python library as open(tarfile).
If I run the code as is, using Python 3, I get the following error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 11: invalid start byte
My first attempt at a fix was to open it as open(tarfile, encoding='utf-8') as I was under the impression that tar would just use what the file system gave it. When I do this, I get the same error (the byte value changes).
If I try with another encoding, say latin-1, I get the following error:
TypeError: Unicode-objects must be encoded before hashing
Which leads me to believe that utf-8 is correct, but I might be misunderstanding.
Can anyone provide suggestions?
I was going down the wrong path thinking this was some strange encoding problem. When it was just a simple problem with that fact that open() defaults to read as text (r). In Python 2 it's a no-op.
The fix is to open(tarfile, 'rb').
The head fake with unicode...should have seen this one coming. :facepalm: