Compressing a netCDF file with the pynco package - python-3.x

I'm interested in compressing a NetCDF file in python using the pynco package. From the command line, I would usually use:
nccopy -d 3 input_file.nc output_file.nc
In python, I have tried the following:
from nco import Nco
Nco.nccopy(input='input_file.nc', output='output_file.nc', options=['-d 1'])
This produces an attribute error 'Nco' has no attribute 'nccopy'. Is nccopy supported within pynco?

nccopy is a netCDF command, not an NCO command. However, ncks can do nearly everything nccopy does, and a whole lot else. To compress as in your example, try
Nco.ncks(input='input_file.nc', output='output_file.nc', options=['-7 -L 1'])

Just to list an alternative to pynco, you can also do this within python using the cdo package:
from cdo import *
cdo = Cdo()
cdo.copy(input='input.nc',output='output.nc',options='-f nc4 -z zip_7')
You can choose the compression level from light (zip_1) all the way up to most compressed (zip_9), I usually find that going above level 4 or 5 there is not much gain (i.e. additional size saving very minor) for a lot of pain (command takes a lot longer!). I think the -f nc4 to ensure netCDF4 output is perhaps not needed if the input is netCDF4 standard, but it doesn't hurt to put it anyway.
You can install cdo with
pip3 install cdo

Related

batch extraction of features - csv files not stored

I want to extract audio features from a large set of files using pyaudioanalysis, using the following command line (as suggested on the github project's page):
python3 audioAnalysis.py featureExtractionDir -i data/ -mw 1.0 -ms 1.0 -sw 0.050 -ss 0.050
This seems to indeed run the feature extraction - it takes time and return to prompt without error - but does not write any csv files.
Any hint about why this does not work would be greatly appreciated. Thank you.
Found the answer:
the files to be analyzed were actually ADPCM-compressed files, which apparently does not work. Therefore, this led to the following error (in the background, it was NOT returned, and only discovered while trying to run the command in a system call in R):
ValueError: Unknown wave file format: DVI_ADPCM. Supported formats: PCM, IEEE_FLOAT
Once files were uncompressed, the pyAudioAnalysis call worked as expected.

Nuitka: Integration of PYDs into .py and .exe

I have an app that uses PySimpleGUI, Matplotlib and some others (I will refer to them jointly as the 'Modules').
(1) I would like to use better than everage protection (VMProtect is in the list with some others) for my executable (app.exe). To achieve this I use Nuitka to convert the my app.py code into C++ and mingw64 to compile it into an executable:
python -m nuitka --standalone --nofollow-imports --plugin-enable=tk-inter --plugin-enable=numpy --plugin-enable=pylint-warnings --mingw64 --output-dir=X: app.py
(2) Also, I want to convert the Modules (=Modules.py) into *.pyd files (=Modules.pyd). I also use Nuitka to conver Modules.py into Modules.pyd. For example:
python -m nuitka --module "c:\Program Files\Python\Lib\site-packages\PySimpleGUI\PySimpleGUI.py"
(3) app.exe shall refer to Modules.pyd which will be distributed together.
My problem is that when I am trying to refer to Modules.pyd in app.py code, which is:
import PySimpleGUI.pyd as sg # PySimpleGUI.pyd is located in the same folder as app.py
I receve the following error:
ModuleNotFoundError: No module named 'PySimpleGUI.pyd'; 'PySimpleGUI' is not a package
I was unable to find a solution to this problem.
In practical terms I would like to achieve one of the following:
(i) cause app.py (which also means app.exe) to recognise Modules.pyd (I already have the latter)
(ii) find some other way to receive app.exe (NB: ready for protection) and Modules.pyd through any other conversion / compilation process.
What I do NOT want to have is app.exe and dozens of Modules.py in the same directory.
Any advice on how to achieve either (i) or (ii) will be most appreciated.
P.S.: all software and modules are up-to-date, all PATHS are set.
UPDATE (2021-09-20):
After further reserach I was able to find what seems to be a [partial] solution -- adding the following lines above imports in app.py taught app.exe (after it was converted via Nuitka) to recognise the Modules.pyd:
import sys
sys.path.append("./DLL/") # where ./DLL/ is a sub-folder with Modules.pyd within the folder where app.exe is located.
However: now I cannot conver dateutil.relativedelta into a proper *.pyd because no such file exists.
This is turning to be a question of how to configure Nuitka to create *.pyd files from module.class.
If anyone has any solution to this, please resond.

Allow my friends to run my python script? [duplicate]

This question already has answers here:
Create a single executable from a Python project [closed]
(3 answers)
Closed 1 year ago.
I'm building a Python application and don't want to force my clients to install Python and modules.
So, is there a way to compile a Python script to be a standalone executable?
You can use PyInstaller to package Python programs as standalone executables. It works on Windows, Linux, and Mac.
PyInstaller Quickstart
Install PyInstaller from PyPI:
pip install pyinstaller
Go to your program’s directory and run:
pyinstaller yourprogram.py
This will generate the bundle in a subdirectory called dist.
pyinstaller -F yourprogram.py
Adding -F (or --onefile) parameter will pack everything into single "exe".
pyinstaller -F --paths=<your_path>\Lib\site-packages yourprogram.py
running into "ImportError" you might consider side-packages.
pip install pynput==1.6.8
still runing in Import-Erorr - try to downgrade pyinstaller - see Getting error when using pynput with pyinstaller
For a more detailed walkthrough, see the manual.
You can use py2exe as already answered and use Cython to convert your key .py files in .pyc, C compiled files, like .dll in Windows and .so on Linux.
It is much harder to revert than common .pyo and .pyc files (and also gain in performance!).
You might wish to investigate Nuitka. It takes Python source code and converts it in to C++ API calls. Then it compiles into an executable binary (ELF on Linux). It has been around for a few years now and supports a wide range of Python versions.
You will probably also get a performance improvement if you use it. It is recommended.
Yes, it is possible to compile Python scripts into standalone executables.
PyInstaller can be used to convert Python programs into stand-alone executables, under Windows, Linux, Mac OS X, FreeBSD, Solaris, and AIX. It is one of the recommended converters.
py2exe converts Python scripts into only executable on the Windows platform.
Cython is a static compiler for both the Python programming language and the extended Cython programming language.
I would like to compile some useful information about creating standalone files on Windows using Python 2.7.
I have used py2exe and it works, but I had some problems.
It has shown some problems for creating single files in Windows 64 bits: Using bundle_files = 1 with py2exe is not working;
It is necessary to create a setup.py file for it to work. http://www.py2exe.org/index.cgi/Tutorial#Step2;
I have had problems with dependencies that you have to solve by importing packages in the setup file;
I was not able to make it work together with PyQt.
This last reason made me try PyInstaller http://www.pyinstaller.org/.
In my opinion, it is much better because:
It is easier to use.
I suggest creating a .bat file with the following lines for example (pyinstaller.exe must be in in the Windows path):
pyinstaller.exe --onefile MyCode.py
You can create a single file, among other options (https://pyinstaller.readthedocs.io/en/stable/usage.html#options).
I had only one problem using PyInstaller and multiprocessing package that was solved by using this recipe: https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Multiprocessing.
So, I think that, at least for python 2.7, a better and simpler option is PyInstaller.
And a third option is cx_Freeze, which is cross-platform.
pyinstaller yourfile.py -F --onefile
This creates a standalone EXE file on Windows.
Important note 1: The EXE file will be generated in a folder named 'dist'.
Important note 2: Do not forget --onefile flag
You can install PyInstaller using pip install PyInstaller
NOTE: In rare cases there are hidden dependencies...so if you run the EXE file and get missing library error (win32timezone in the example below) then use something like this:
pyinstaller --hiddenimport win32timezone -F "Backup Program.py"
I like PyInstaller - especially the "windowed" variant:
pyinstaller --onefile --windowed myscript.py
It will create one single *.exe file in a distination/folder.
You may like py2exe. You'll also find information in there for doing it on Linux.
Use py2exe.... use the below set up files:
from distutils.core import setup
import py2exe
from distutils.filelist import findall
import matplotlib
setup(
console = ['PlotMemInfo.py'],
options = {
'py2exe': {
'packages': ['matplotlib'],
'dll_excludes': ['libgdk-win32-2.0-0.dll',
'libgobject-2.0-0.dll',
'libgdk_pixbuf-2.0-0.dll']
}
},
data_files = matplotlib.get_py2exe_datafiles()
)
I also recommend PyInstaller for better backward compatibility such as Python 2.3 - 2.7.
For py2exe, you have to have Python 2.6.
For Python 3.2 scripts, the only choice is cx_Freeze. Build it from sources; otherwise it won't work.
For Python 2.x I suggest PyInstaller as it can package a Python program in a single executable, unlike cx_Freeze which outputs also libraries.
Since it seems to be missing from the current list of answers, I think it is worth mentioning that the standard library includes a zipapp module that can be used for this purpose. Its basic usage is just compressing a bunch of Python files into a zip file with extension .pyz than can be directly executed as python myapp.pyz, but you can also make a self-contained package from a requirements.txt file:
$ python -m pip install -r requirements.txt --target myapp
$ python -m zipapp -p "interpreter" myapp
Where interpreter can be something like /usr/bin/env python (see Specifying the Interpreter).
Usually, the generated .pyz / .pyzw file should be executable, in Unix because it gets marked as such and in Windows because Python installation usually registers those extensions. However, it is relatively easy to make a Windows executable that should work as long as the user has python3.dll in the path.
If you don't want to require the end user to install Python, you can distribute the application along with the embeddable Python package.
py2exe will make the EXE file you want, but you need to have the same version of MSVCR90.dll on the machine you're going to use your new EXE file.
See Tutorial for more information.
You can find the list of distribution utilities listed at Distribution Utilities.
I use bbfreeze and it has been working very well (yet to have Python 3 support though).
Not exactly a packaging of the Python code, but there is now also Grumpy from Google, which transpiles the code to Go.
It doesn't support the Python C API, so it may not work for all projects.
Using PyInstaller, I found a better method using shortcut to the .exe rather than making --onefile. Anyway, there are probably some data files around and if you're running a site-based app then your program depends on HTML, JavaScript, and CSS files too. There isn't any point in moving all these files somewhere... Instead what if we move the working path up?
Make a shortcut to the EXE file, move it at top and set the target and start-in paths as specified, to have relative paths going to dist\folder:
Target: %windir%\system32\cmd.exe /c start dist\web_wrapper\web_wrapper.exe
Start in: "%windir%\system32\cmd.exe /c start dist\web_wrapper\"
We can rename the shortcut to anything, so renaming to "GTFS-Manager".
Now when I double-click the shortcut, it's as if I python-ran the file! I found this approach better than the --onefile one as:
In onefile's case, there's a problem with a .dll missing for the Windows 7 OS which needs some prior installation, etc. Yawn. With the usual build with multiple files, no such issues.
All the files that my Python script uses (it's deploying a tornado web server and needs a whole freakin' website worth of files to be there!) don't need to be moved anywhere: I simply create the shortcut at top.
I can actually use this exact same folder on Ubuntu (run python3 myfile.py) and Windows (double-click the shortcut).
I don't need to bother with the overly complicated hacking of .spec file to include data files, etc.
Oh, remember to delete off the build folder after building. It will save on size.
Use Cython to convert to C, compile, and link with GCC.
Another could be, make the core functions in C (the ones you want to make hard to reverse), compile them and use Boost.Python to import the compiled code (plus you get a much faster code execution). Then use any tool mentioned to distribute.
I'm told that PyRun is also an option. It currently supports Linux, FreeBSD and Mac OS X.

Gromacs: .ene extension not recognized

I'm new to Gromacs (and protein-analysis coding in general, but have some experience with python-based code). I'm trying to convert a .ene file received from pyDock to a more readable format (it currently opens
When I try to use different commands that the gromacs guide says accept .ene files (gmx eneconv and gmx dump), for example
gmx eneconv -f project.ene -o converted.edr
and
gmx dump -e project.ene -om read.mdp
I get the error
File 'project.ene' cannot be used by GROMACS because it does not have a recognizable extension.
The following extensions are possible for this option:
.edr
I have updated my OS and re-installed gromacs. My installation is working according to the 'Getting Started' page.
I am also open to suggestions for other programs to open and read the .ene file type.
Thanks!
"ene" is short for "energy" which contains binary data for energy, temperature, volume, etc, and is only used in older version of Gromacs. In new version of Gromacs this is replaced by "edr" file which contains portable binary data. See "gmx dump -h", you will find the following lines:
Options to specify output files:
-o [<.edr>] (fixed.edr)
Energy file
If you insist on using ene file, you should install an older version of Gromacs.

SublimeCodeIntel isn't looking in python 3 paths

I installed SublimeCodeIntel package in sublime text, and I use it to code in python 3. However, the problem is that it uses python2 paths to do the import, so if a library is installed in python2 and not in python3, i won't find it when i use import or from X import y.
I did confirm that when I used the command SublimeCodeIntel: Dump Import Directories , so I saw both the files python and python3 in the directory ./codeintel/import_dir_stats and here is there content :
python
dedeeb56f744e507026fef17243da41f /home/bilal/.local/lib/python2.7/site-packages
6a1d0cac3d9e6148e2208b63a33a1e6f /home/bilal/.local/lib/python2.7/site-packages/impacket
16a4fccbb3beadfdfd72691ef8f7298c /home/bilal/.local/lib/python2.7/site-packages/mechanize
211d2b55059f6b634799fdae534decd9 /usr/lib/python2.7/dist-packages
be5448890caffe81686310f127d6efae /usr/lib/python2.7/dist-packages/_markerlib
cec69a0830a725e10ac4e364d44add8f /usr/lib/python2.7/dist-packages/appindicator
python3
dedeeb56f744e507026fef17243da41f /home/bilal/.local/lib/python2.7/site-packages
6a1d0cac3d9e6148e2208b63a33a1e6f /home/bilal/.local/lib/python2.7/site-packages/impacket
16a4fccbb3beadfdfd72691ef8f7298c /home/bilal/.local/lib/python2.7/site-packages/mechanize
211d2b55059f6b634799fdae534decd9 /usr/lib/python2.7/dist-packages
be5448890caffe81686310f127d6efae /usr/lib/python2.7/dist-packages/_markerlib
cec69a0830a725e10ac4e364d44add8f /usr/lib/python2.7/dist-packages/appindicator
I didn't put all the lines (because there is a lot), but the content of the two files is identical.
I don't understand from where is coming this problem, why SublimeCodeIntel is looking in python2 directories for the import ??!!
PS : I am using Ubuntu 15.10 (with Linux version > 4), and sublime text 3 build 3103.
Please help, I really consumed a lot of time and energy looking for this.
I would strongly recommend using the Anaconda plugin (no relation whatsoever to the Anaconda Python distribution) instead of SublimeCodeIntel. I struggled with SCI for a while on various machines, and could never get very good code completion out of it until I ran across Anaconda one day. The next day I removed SCI and have been completely satisfied ever since. It's super-easy to configure (just give it the path to your python executable and it reads sys.path and all the rest), and pretty much just works. It has linting with several different linters built-in (you can disable them if you want), has popups available for function signatures and other hints, works with virtualenvs right out of the box, and more.
(I didn't write it and I have no connection to the author(s) - it's just a great plugin!)

Resources