My python version is 3.5 through Anaconda on Windows 10 environment. I'm using Pyminizip because I need password protected for my zip files, and Zipfile doesn't support it yet.
I am able to zip single file through the function pyminizip.compress, and the encrypt function worked as expected. However, when trying to use pyminizip.compress_multiple I always encountered a Python crash (as pictures) and I believe it's due to the problem of my bad input format.
What I would like to know is: What's the acceptable format for input argument src file LIST path? From Pyminizip's documentation:
pyminizip.compress_multiple([u'pyminizip.so', 'file2.txt'], "file.zip", "1233", 4, progress)
Args:
1. src file LIST path (list)
2. dst file path (string)
3. password (string) or None (to create no-password zip)
4. compress_level(int) between 1 to 9, 1 (more fast) <---> 9 (more compress)
It seems the first argument src file LIST path should be a list containing all files required to be zipped. Accordingly, I tried to use compress_multiple to compress single file with command:
pyminizip.compress_multiple( ['Filename.txt'], 'output.zip', 'password', 4, optional)
and it lead to Python crash. So I try to add a full path into the args.
pyminizip.compress_multiple( [os.getcwd(), 'Filename.txt'], ... )
and still, it crashed again. So I think maybe I have to split the path like this
path = os.getcwd().split( os.sep )
pyminizip.compress_multiple( [path, 'Filename.txt'], ...)
still got a bad luck. Any ideas?
Pyminizip requires the path name (or relative path name from where the script is running from) in the files.
Your example:
pyminizip.compress_multiple( [os.getcwd(), 'Filename.txt'], ... )
gives a list of files of os.getcwd(), and then another file, 'Filename.txt'. You need to combine them into a single path using os.path.join()
in your filename example, you will need:
pyminizip.compress_multiple( [os.path.join(getcwd(), 'Filename.txt')],...)
conversly:
pyminizip.compress_multiple( [os.path.join(getcwd(), 'Filename1.txt'), os.path.join(getcwd(), 'Filename2.txt')],...)
From here - https://pypi.org/project/pyminizip/, the usage of compress_multiple is
pyminizip.compress_multiple([u'pyminizip.so', 'file2.txt'], [u'/path_for_file1', u'/path_for_file2'], "file.zip", "1233", 4, progress)
The second parameter is a bit confusing, but if used, it will create a zip file, which when uncompressed, will create a directory structure like:
Related
As far as I'm aware I'm using best practices to define paths (using raw strings) and how I go about joining them (using os.path.join()), e.g.
import os
fdir = r'C:\Code\...\samples'
fpath = os.path.join(fdir, 'fname.ext')
and doing so has not caused me any problems when running my code within a Python or command shell. If I print fpath to the console I get consistent use of \s in the path:
C:\Code...\samples\fname.ext
But when I run a Docker containerized version of the code and run the image I get the error:
FileNotFoundError: [Errno 2] No such file or directory:
'C:\Code\...\samples/fname.ext'
I don't understand why os.path.join() has used a / to join fdir and fname.ext when the rest of the path included \\. It doesn't do this when I run the code outside of the container.
I have tried using os.path.normpath():
fpath = os.path.join(fdir, 'fname.ext')
fpath = os.path.normpath(fpath)
as discussed here, and os.sep.join():
fpath = os.sep.join([fdir, 'fname.ext'])
as covered here, and Path().joinpath():
from pathlib import Path
fpath = Path(fdir).joinpath('fname.ext')
as well as Path() / 'path_to_add':
fpath = Path(fdir) / 'fname.ext'
as discussed here, but in every case I end up with the same result using os.path.join().
Can someone please help me to understand what is going on and how to create consistent paths that will work whether I run the code in Python in a Windows environment, or in a Docker container?
Update Nov. 16:
In trying to keep my question brief I think I've left out details that are crucial. Apologies to those who have kindly taken the time to offer suggestions based on my incomplete description of the problem.
My code needs to import/export files from/to directories that are defined within a user-specified configuration file.
So the configuration file has a section of code where the user defines variables and paths, e.g.
samplesDir = r"path-to-samples-directory"
The variables are stored in a dictionary of dictionaris and stored as a .json.
At the start of the code the user defines the key that selects the dictionary of interest so that at various parts in my code when a file needs to be imported/exported, the paths are at hand.
So back to my example, samplesDir is stored in the configuration dictionary, cfgDict, so all I need to do is append the file name:
sampleFpath = os.path.join(sampleDir, sampleFname)
and sampleFname is determined based on other variables.
Because of the dynamic nature of the variables (including directory paths and file paths), I think it rules out the use of static path defined in a .yml with Docker Compose.
Update Nov. 18:
It may help to include a few more details and some screenshots.
The above screenshot shows the file and folder structure of the src directory containing the source code, the main app.py script for command-line use, the Dockerfile, etc.
The configs folder contains JSON files that includes variables, paths to directories and files. The user can create configuration files either by copying an existing one and modifying the entries, or configuration files can be generated by calling config.py.
Within config.py I have pre-set variables and paths, so that the directory path to the configuration files (configs), sample files (sample_DROs) and others (e.g. fiducials) are all within src.
I don't anticipate any reason why the user would want to store the config files anywhere else, nor do I expect them to want to use different sample files (or move them elsewhere). However, they will undoubtedly create their own fiducials and may decide not to store them in the fiducials directory (i.e. somewhere not within the src directory).
Likewise I have pre-set the download directory (based on the parameters stored within the configuration files, files are fetched from a server and downloaded) to be the default Downloads directory:
rootDownloadDir = os.path.join(Path.home(), "Downloads", "xnat_downloads")
Those files are later imported, processed, and the outputs are (by default) exported into sub-directories within rootDownloadDir.
Within Dockerfile I set the working directory of the container to be that of the source code and copy all of the contents of src (with the exception of some directories defined in .dockerignore):
WORKDIR C:/Code/WP1.3_multiple_modalities/src
...
COPY . .
so that the structure of the container mimics that of WORKDIR:
Hence I have allowed for flexibility in import/export directories, and they are by default a combination of paths within and outside of the src directory. And so, the code executed within the container will need to access files both within and outside of src.
That said, I don't know what rootDownloadDir will look like when os.path.join(Path.home(), "Downloads", "xnat_downloads") is run within the container.
This has got me thinking - Is it bad practice to set the download directory outside of src?
Returning to the original error:
the sample file is in the container:
From the actual behavior I can suppose that the container is based on Unix-like image. Path separator is / in such systems.
To build an environment-independent path which works inside and outside of the container you need the following steps:
Mounting of host folder to container directory.
Environment variable inside and outside the container.
I can show an example of how this is achievable via docker-compose tool and its configuration file docker-compose.yml:
# docker-compose.yml file
version: '3'
services:
<service_name>: # your service name here
image: <image_name> # name of image your container is built on
environment:
- SAMPLES_PATH=/samples
volumes:
- C:\Code\somepath\samples:/samples
In your python code you can use the following structure:
import os
fdir = os.getenv('SAMPLES_PATH', r'C:\Code\...\samples')
fpath = os.path.join(fdir, 'fname.ext')
I have the following code;
#!/usr/bin/env python3
import yaml
with open('config.yml', 'r') as config_file:
config = yaml.load(config_file)
The file is called __init__.py which is in the directory ~/bin/myprogram/myprogram/ and in the same directory, I have a file called config.yml
My symlink is as follows;
user$ ls -la /usr/local/bin/
lrwxr-xr-x 1 user admin 55 27 Nov 13:25 myprogram -> /Users/user/bin/myprogram/myprogram/__init__.py
Every time I run myprogram, I get the error FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'. I believe this is because the config.yml is not in /usr/local/bin/. What is the best way to work around this issue?
You can use __file__ to access the location of the __init__.py file when executing code in that file. It returns the full path, but care has to be taken as it may be the .pyc (or .pyo) version. Since you are using Python3 I would use the pathlib module:
import yaml
from pathlib import Path
my_path = Path(__file__).resolve() # resolve to get rid of any symlinks
config_path = my_path.parent / 'config.yaml'
with config_path.open() as config_file:
config = yaml.safe_load(config_file)
Please note:
If you have to use PyYAML, use safe_load(), even PyYAML's own documentation indicates .load() can be unsafe. It almost never necessary to use that. And in the unlikely event that safe_load() cannot load your config, e.g. if it has !!python/... tags, you should explicitly add register the classes that you actually need to the SafeLoader).
Since September 2006 the recommended extension for YAML files has been .yaml
This is very similar question to this one, but for Python instead of powershell. It was also discussed here, and here, but no working solutions are posted.
So, is there a way to create a directory in Python that bypasses the 260 char limit on windows? I tried multiple ways of prepending \\?\, but could not make it work.
In particular, the following most obvious code
path = f'\\\\?\\C:\\{"a"*300}.txt'
open(path, 'w')
fails with an error
OSError: [Errno 22] Invalid argument: '\\\\?\\C:\\aaaaa<...>aaaa.txt'
Thanks to eryksun I realized that I was trying to create a file with too long of a name. After some experiments, this is how one can create a path that exceeds 260 chars on windows (provided file system allows it):
from pathlib import Path
folder = Path('//?/c:/') / ('a'*100) / ('b'*100)
file = folder / ('c' * 100)
folder.mkdir(parents=True, exist_ok=True)
file.open('w')
Update 2
I think #faré is right, it's an output translation problem.
So I declared the evironment variable ASDF_OUTPUT_TRANSLATIONS and set it to E:/. Now (asdf:require-system "my-system") yields a different error: Uneven number of components in source to destination mapping: "E:/" which led me to this SO-topic.
Unfortunately, his solution doesn't work for me. So I tried the other answer and set ASDF_OUTPUT_TRANSLATIONS to (:output-translations (t "E:/")). Now I get yet another error:
Invalid source registry (:OUTPUT-TRANSLATIONS (T "E:/")).
One and only one of
:INHERIT-CONFIGURATION or
:IGNORE-INHERITED-CONFIGURATION
is required.
(will be skipped)
Original Posting
I have a simple system definition but can't get ASDF to load it.
(asdf-version 3.1.5, sbcl 1.3.12 (upgraded to 1.3.18 AMD64), slime 2.19, Windows 10)
What I have tried so far
Following the ASDF manual: "4.1 Configuring ASDF to find your systems"
There it says:
For Windows users, and starting with ASDF 3.1.5, start from your
%LOCALAPPDATA%, which is usually ~/AppData/Local/ (but you can ask in
a CMD.EXE terminal echo %LOCALAPPDATA% to make sure) and underneath
create a subpath config/common-lisp/source-registry.conf.d/
That's exactly what I did:
Echoing %LOCALAPPDATA% which evaluates to C:\Users\my-username\AppData\Local
Underneath I created the subfolders config\common-lisp\source-registry.conf.d\ (In total: C:\Users\my-username\AppData\Local\config\common-lisp\source-registry.conf.d\
The manual continues:
there create a file with any name of your choice but with the type conf, for instance 50-luser-lisp.conf; in this file, add the following line to tell ASDF to recursively scan all the subdirectories under /home/luser/lisp/ for .asd files: (:tree "/home/luser/lisp/")
That’s enough. You may replace /home/luser/lisp/ by wherever you want to install your source code.
In the source-registry.conf.d folder I created the file my.conf and put in it (:tree "C:/Users/my-username/my-systems/"). This folder contains a my-system.asd.
And here comes the weird part:
If I now type (asdf:require-system "my-system") in the REPL I get the following error:
Can't create directory C:\Users\my-username\AppData\Local\common-lisp\sbcl-1.3.12-win-x86\C\Users\my-username\my-systems\C:\
So the problem is not that ASDF doesn't find the file, it does -- but (whatever the reason) it tries to create a really weird subfolder hierarchy which ultimately fails because at the end it tries to create the folder C: but Windows doesn't allow foldernames containing a colon.
Another approach: (push path asdf:*central-registry*)
If I try
> (push #P"C:/Users/my-username/my-systems/" asdf:*central-registry*)
(#P"C:/Users/my-username/my-systems/"
#P"C:/Users/my-username/AppData/Roaming/quicklisp/quicklisp/")
> (asdf:require-system "my-system")
I get the exact same error.
I don't know what to do.
Update
Because of the nature of the weird path ASDF was trying to create I thought maybe I could bypass the problem by specifying a relative path instead of an absolute one.
So I tried
(:tree "\\Users\\my-username\\my-systems")
in my conf file. Still the same error.
Ahem. It looks like an output-translations problem.
I don't have a Windows machine right now, but this all used to work last time I tried.
Can you setup some ad hoc output-translations for now that will make it work?
I have variable address = /data/train/1.jpg, and I'm trying to read file by
im = Image.open(address)
FileNotFoundError: [Errno 2] No such file or directory: '/data/train/1.jpg'
By some reasons I can't use full name of file.
I started jupyter notebook from folder which actually contains file 1.jpg in /data/train/.
How can I fix it?
Relative addressing means from the perspective of the current working directory. So if you script is in the same directory that the data folder is in, your path to the file would be ./data/train/1.jpg. Note the ./, which means the current directory.
use relative path, this one is absolute
address = './data/train/1.jpg'
im = Image.open(address)
in this case . means current location while slash means the environments root (view and explanation) on this depends on your OS