I want to create a script to sync files between two directories, and was going to utilise Dirsync and Python 3 for this.
from dirsync import sync
sync('C:/03py/Sync/Sync1','C:/03py/Sync/Sync2','sync', twoway=True, create=True)
After running the file for the first time, the folders are synced. I then put a dummy file and folder into the target directory and reran the above script, hoping the file and folder would be copied back into the source directory. However I get the following:
Only in C:/03py/Sync/Sync2
<< TESTTWOFOLDER
<< _TESTTWOWAY.txt
I am not certain if I am using the above commands correctly.
i don't know if this helps...
from dirsync import sync
source_path = 'C:\\wamp\\www\\first-python-app\\http'
target_path = 'C:\\wamp\\www\\first-python-app\\dev'
sync(target_path, source_path, 'sync', twoway=True, purge=True)
sync(source_path, target_path, 'sync')
Related
Might just be an edge case but I extracted a zip file to a directory using the zip file module. When extracting, zip file names the directory it extracts to.
If there is a way I get to specify the name of the folder Zip file creates to extract the files to? I am hitting an error because I am using the same folder zipped up to test zip file and it keeps using the old folder name which already exists so it throws an error. Here is my code:
orginalFolderName = jobFolder + name
with zipfile.ZipFile(directory,"r") as zip_ref:
zip_ref.extractall(jobFolder)
os.rename(orginalFolderName, newFoldername)
directory = newFoldername
with zipfile.ZipFile(filepath) as z:
z.extractall(dest_folder)
filepath - Complete path of zipfile
dest_folder - destination folder
As far as I'm aware I'm using best practices to define paths (using raw strings) and how I go about joining them (using os.path.join()), e.g.
import os
fdir = r'C:\Code\...\samples'
fpath = os.path.join(fdir, 'fname.ext')
and doing so has not caused me any problems when running my code within a Python or command shell. If I print fpath to the console I get consistent use of \s in the path:
C:\Code...\samples\fname.ext
But when I run a Docker containerized version of the code and run the image I get the error:
FileNotFoundError: [Errno 2] No such file or directory:
'C:\Code\...\samples/fname.ext'
I don't understand why os.path.join() has used a / to join fdir and fname.ext when the rest of the path included \\. It doesn't do this when I run the code outside of the container.
I have tried using os.path.normpath():
fpath = os.path.join(fdir, 'fname.ext')
fpath = os.path.normpath(fpath)
as discussed here, and os.sep.join():
fpath = os.sep.join([fdir, 'fname.ext'])
as covered here, and Path().joinpath():
from pathlib import Path
fpath = Path(fdir).joinpath('fname.ext')
as well as Path() / 'path_to_add':
fpath = Path(fdir) / 'fname.ext'
as discussed here, but in every case I end up with the same result using os.path.join().
Can someone please help me to understand what is going on and how to create consistent paths that will work whether I run the code in Python in a Windows environment, or in a Docker container?
Update Nov. 16:
In trying to keep my question brief I think I've left out details that are crucial. Apologies to those who have kindly taken the time to offer suggestions based on my incomplete description of the problem.
My code needs to import/export files from/to directories that are defined within a user-specified configuration file.
So the configuration file has a section of code where the user defines variables and paths, e.g.
samplesDir = r"path-to-samples-directory"
The variables are stored in a dictionary of dictionaris and stored as a .json.
At the start of the code the user defines the key that selects the dictionary of interest so that at various parts in my code when a file needs to be imported/exported, the paths are at hand.
So back to my example, samplesDir is stored in the configuration dictionary, cfgDict, so all I need to do is append the file name:
sampleFpath = os.path.join(sampleDir, sampleFname)
and sampleFname is determined based on other variables.
Because of the dynamic nature of the variables (including directory paths and file paths), I think it rules out the use of static path defined in a .yml with Docker Compose.
Update Nov. 18:
It may help to include a few more details and some screenshots.
The above screenshot shows the file and folder structure of the src directory containing the source code, the main app.py script for command-line use, the Dockerfile, etc.
The configs folder contains JSON files that includes variables, paths to directories and files. The user can create configuration files either by copying an existing one and modifying the entries, or configuration files can be generated by calling config.py.
Within config.py I have pre-set variables and paths, so that the directory path to the configuration files (configs), sample files (sample_DROs) and others (e.g. fiducials) are all within src.
I don't anticipate any reason why the user would want to store the config files anywhere else, nor do I expect them to want to use different sample files (or move them elsewhere). However, they will undoubtedly create their own fiducials and may decide not to store them in the fiducials directory (i.e. somewhere not within the src directory).
Likewise I have pre-set the download directory (based on the parameters stored within the configuration files, files are fetched from a server and downloaded) to be the default Downloads directory:
rootDownloadDir = os.path.join(Path.home(), "Downloads", "xnat_downloads")
Those files are later imported, processed, and the outputs are (by default) exported into sub-directories within rootDownloadDir.
Within Dockerfile I set the working directory of the container to be that of the source code and copy all of the contents of src (with the exception of some directories defined in .dockerignore):
WORKDIR C:/Code/WP1.3_multiple_modalities/src
...
COPY . .
so that the structure of the container mimics that of WORKDIR:
Hence I have allowed for flexibility in import/export directories, and they are by default a combination of paths within and outside of the src directory. And so, the code executed within the container will need to access files both within and outside of src.
That said, I don't know what rootDownloadDir will look like when os.path.join(Path.home(), "Downloads", "xnat_downloads") is run within the container.
This has got me thinking - Is it bad practice to set the download directory outside of src?
Returning to the original error:
the sample file is in the container:
From the actual behavior I can suppose that the container is based on Unix-like image. Path separator is / in such systems.
To build an environment-independent path which works inside and outside of the container you need the following steps:
Mounting of host folder to container directory.
Environment variable inside and outside the container.
I can show an example of how this is achievable via docker-compose tool and its configuration file docker-compose.yml:
# docker-compose.yml file
version: '3'
services:
<service_name>: # your service name here
image: <image_name> # name of image your container is built on
environment:
- SAMPLES_PATH=/samples
volumes:
- C:\Code\somepath\samples:/samples
In your python code you can use the following structure:
import os
fdir = os.getenv('SAMPLES_PATH', r'C:\Code\...\samples')
fpath = os.path.join(fdir, 'fname.ext')
I am trying to delete the folder with name "repro" and its contents in my build drop location. I have configured my delete files steps as below
Source Folder: $(BuildDropLocation)\$(BuildNumber)\CTrest\lime
Contents:
**/repro/*
repro folder resides here
$(BuildDropLocation)\$(BuildNumber)\CTrest\lime\version\package\code**repro**..
Is there something that I am missing here?
Here is the doc for the command: Delete Files task. Examples of contents:
**/temp/* deletes all files in any sub-folder named temp.
**/temp* deletes any file or folder with a name that begins with temp.
I think, **/repro* will be more suitable in your case.
I'm trying to create a folder within the temp folder that doesn't have a random name.
Here is how I was trying to create the folder within the temp folder.
if not DirExists(ExpandConstant('{%tmp}\Utilities\SDK')) then
CreateDir(ExpandConstant('{%tmp}\Utilities\SDK'));
Log('Temp\Utilities\SDK Folder Has Been Created.');
I had a look at this thread, but even with the %, unfortunately, it still doesn't create the folder.The script compiles and runs as expected, however the folder doesn't create even though it says it has in the log file, I understand that the log file will say that because its told too, however, if the folder was unable to be created, wouldnt it crash? or return a false if an if statement was present?
With CreateDir() You must create dirs one after the other and not a dir structure at once.
if not DirExists(ExpandConstant('{tmp}\Utilities')) then
CreateDir(ExpandConstant('{tmp}\Utilities'));
if not DirExists(ExpandConstant('{tmp}\Utilities\SDK')) then
CreateDir(ExpandConstant('{tmp}\Utilities\SDK'));
if DirExists(ExpandConstant('{tmp}\Utilities\SDK')) then
Log('Temp\Utilities\SDK Folder Has Been Created.') else
Log('Temp\Utilities\SDK Folder ERROR : NOT Created.');
Inno Setup has a function to create a dir structure at once
function ForceDirectories(Dir: string): Boolean;
Example:
if not DirExists(ExpandConstant('{tmp}\Utilities\SDK')) then
ForceDirectories(ExpandConstant('{tmp}\Utilities\SDK'));
Also keep in mind :
{tmp} all is related to the Inno Setup Temp folder is-XXXXX.tmp
C:\Users\...\AppData\Local\Temp\is-XXXXX.tmp
{%temp} is the users Temp folder
C:\Users\...\AppData\Local\Temp
I think you want the Windows Temp and not the tmp from InnoSetup
{tmp}:
Temporary directory used by Setup or Uninstall. This is not the value of the user's TEMP environment variable. It is a subdirectory of the user's temporary directory which is created by Setup or Uninstall at startup (with a name like "C:\WINDOWS\TEMP\IS-xxxxx.tmp"). All files and subdirectories in this directory are deleted when Setup or Uninstall exits. During Setup, this is primarily useful for extracting files that are to be executed in the [Run] section but aren't needed after the installation.
So I think you want to do somethink like this:
if not DirExists(ExpandConstant('{%temp}\Utilities\SDK')) then
CreateDir(ExpandConstant('{%temp}\Utilities\SDK'));
Log('Temp\Utilities\SDK Folder Has Been Created.');
I need to zip some files in amazon s3 without needing to write them to file locally first. Ideally my code worked in development but i don't have many write privileges in production.
folder = output_dir
files = fs.glob(folder)
f = BytesIO()
zip = zipfile.ZipFile(f, 'a', zipfile.ZIP_DEFLATED)
for file in files:
filename = os.path.basename(file)
image = fs.get(file, filename)
zip.write(filename)
zip.close()
the proplem is at this line in production
image = fs.get(file, filename)
Because i don't have write privileges.
My last resort is to write to /tmp/ directory which i have privileges to.
Is there a way to zip files from a url path or directly in the cloud?
I ended up using python tempfile which ended up being a perfect solution.
Using NamedTemporaryFile gave me the guarantee to create named and system visible temporary files that could be deleted automatically. No manual work.