I have a hello tool which contains only exe files (without source).
Hello tool structure:
bin
helloBin.exe
helloRoot.exe
conanfile.py
conanfile.py content:
class ToolHelloConan(ConanFile):
name = "ToolHello"
version = "0.1"
settings = "os", "compiler", "build_type", "arch"
def package(self):
self.copy("*")
def package_info(self):
self.cpp_info.libs = self.collect_libs()
I've exported the hello tool to local cache: conan export-pkg . ToolHello/0.1#user/testing. This copied all exe in local_cache/ToolHello/0.1/user/testing/package/hash/bin. The bin in local cache looks like this:
bin
helloBin.exe
helloRoot.exe
I've defined a tool integration project which contains only the conanfile.txt
[requires]
ToolHello/0.1#user/testing
[generators]
virtualrunenv
After running conan install . in tool integration project and activating the virtual run environment, I am able to call only the helloRoot.exe because it's located right in the bin directory, but I'm not able to execute the bin/bin/helloBin.exe
Question: How do I run exe files which are not located directly in the local_cache/ToolHello/0.1/user/testing/package/hash/bin, but in local_cache/ToolHello/0.1/user/testing/package/hash/bin/directory?
You need to define the bindirs that are not the default one (bin). Add this to your conanfile.py:
def package_info(self):
self.cpp_info.bindirs = ["bin", "bin/directory"]
If you need to also include the root of the package folder, you might need to use:
def package_info(self):
self.cpp_info.bindirs = ["", "bin", "bin/directory"]
Related
Building my workspace with colcon, some OSM files which are found in a directory "OSM" in a sub_package in the workspace are not found in built space. So when I go to the install space, the files are not there. I am not sure how to do this and if I should put it in the setup.py.
I tried putting this in the setup.py file in arguments of setup():
setup(
name=package_name,
version='0.0.0',
packages=[package_name, submodules, osm],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name), glob('launch/*.launch.py')),
('.package_name/sub_package', glob('OSM_folder/*.osm')),
],
.
.
.
) # close setup()
but it did not work.
I am using ROS2 Galactic.
Directory structure:
package_name
┃
┣━━━━setup.py
┣━━━━package.xml
┣━━━━resource/
┣━━━━launch/
┗━━━━package_name
┗━sub_package_name
┗━OSM
┣━__init__.py
┗━some_osm_files.osm
I have the OSM directory in the built workspace but it has only the init.py file
I solved it. I am not sure if this is the right way of doing this or is there like another better/proper way or not, but here we go.
In the setup.py file, I added the line
(os.path.join('lib/python3.8/site-packages/package_name/sub_package/OSM'),glob(package_name+'/sub_package_name/OSM/*.osm')), in the data_files variable.
The first part of the new line which is os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM') determines the new location of the files in the install folder after building the workspace.
The second part which is glob(package_name+'/sub_package_name/OSM/*.osm') determines the files original location in the project workspace.
So the result is that it takes the files from the location mentioned in the second part and puts them in the location mentioned in the first part.
The resulting block is:
setup(
name=package_name,
version='0.0.0',
packages=[package_name, submodules, osm],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name), glob('launch/*.launch.py')),
(os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM'), glob(package_name+'/sub_package_name/OSM/*.osm')),
],
.
.
.
)
As far as I'm aware I'm using best practices to define paths (using raw strings) and how I go about joining them (using os.path.join()), e.g.
import os
fdir = r'C:\Code\...\samples'
fpath = os.path.join(fdir, 'fname.ext')
and doing so has not caused me any problems when running my code within a Python or command shell. If I print fpath to the console I get consistent use of \s in the path:
C:\Code...\samples\fname.ext
But when I run a Docker containerized version of the code and run the image I get the error:
FileNotFoundError: [Errno 2] No such file or directory:
'C:\Code\...\samples/fname.ext'
I don't understand why os.path.join() has used a / to join fdir and fname.ext when the rest of the path included \\. It doesn't do this when I run the code outside of the container.
I have tried using os.path.normpath():
fpath = os.path.join(fdir, 'fname.ext')
fpath = os.path.normpath(fpath)
as discussed here, and os.sep.join():
fpath = os.sep.join([fdir, 'fname.ext'])
as covered here, and Path().joinpath():
from pathlib import Path
fpath = Path(fdir).joinpath('fname.ext')
as well as Path() / 'path_to_add':
fpath = Path(fdir) / 'fname.ext'
as discussed here, but in every case I end up with the same result using os.path.join().
Can someone please help me to understand what is going on and how to create consistent paths that will work whether I run the code in Python in a Windows environment, or in a Docker container?
Update Nov. 16:
In trying to keep my question brief I think I've left out details that are crucial. Apologies to those who have kindly taken the time to offer suggestions based on my incomplete description of the problem.
My code needs to import/export files from/to directories that are defined within a user-specified configuration file.
So the configuration file has a section of code where the user defines variables and paths, e.g.
samplesDir = r"path-to-samples-directory"
The variables are stored in a dictionary of dictionaris and stored as a .json.
At the start of the code the user defines the key that selects the dictionary of interest so that at various parts in my code when a file needs to be imported/exported, the paths are at hand.
So back to my example, samplesDir is stored in the configuration dictionary, cfgDict, so all I need to do is append the file name:
sampleFpath = os.path.join(sampleDir, sampleFname)
and sampleFname is determined based on other variables.
Because of the dynamic nature of the variables (including directory paths and file paths), I think it rules out the use of static path defined in a .yml with Docker Compose.
Update Nov. 18:
It may help to include a few more details and some screenshots.
The above screenshot shows the file and folder structure of the src directory containing the source code, the main app.py script for command-line use, the Dockerfile, etc.
The configs folder contains JSON files that includes variables, paths to directories and files. The user can create configuration files either by copying an existing one and modifying the entries, or configuration files can be generated by calling config.py.
Within config.py I have pre-set variables and paths, so that the directory path to the configuration files (configs), sample files (sample_DROs) and others (e.g. fiducials) are all within src.
I don't anticipate any reason why the user would want to store the config files anywhere else, nor do I expect them to want to use different sample files (or move them elsewhere). However, they will undoubtedly create their own fiducials and may decide not to store them in the fiducials directory (i.e. somewhere not within the src directory).
Likewise I have pre-set the download directory (based on the parameters stored within the configuration files, files are fetched from a server and downloaded) to be the default Downloads directory:
rootDownloadDir = os.path.join(Path.home(), "Downloads", "xnat_downloads")
Those files are later imported, processed, and the outputs are (by default) exported into sub-directories within rootDownloadDir.
Within Dockerfile I set the working directory of the container to be that of the source code and copy all of the contents of src (with the exception of some directories defined in .dockerignore):
WORKDIR C:/Code/WP1.3_multiple_modalities/src
...
COPY . .
so that the structure of the container mimics that of WORKDIR:
Hence I have allowed for flexibility in import/export directories, and they are by default a combination of paths within and outside of the src directory. And so, the code executed within the container will need to access files both within and outside of src.
That said, I don't know what rootDownloadDir will look like when os.path.join(Path.home(), "Downloads", "xnat_downloads") is run within the container.
This has got me thinking - Is it bad practice to set the download directory outside of src?
Returning to the original error:
the sample file is in the container:
From the actual behavior I can suppose that the container is based on Unix-like image. Path separator is / in such systems.
To build an environment-independent path which works inside and outside of the container you need the following steps:
Mounting of host folder to container directory.
Environment variable inside and outside the container.
I can show an example of how this is achievable via docker-compose tool and its configuration file docker-compose.yml:
# docker-compose.yml file
version: '3'
services:
<service_name>: # your service name here
image: <image_name> # name of image your container is built on
environment:
- SAMPLES_PATH=/samples
volumes:
- C:\Code\somepath\samples:/samples
In your python code you can use the following structure:
import os
fdir = os.getenv('SAMPLES_PATH', r'C:\Code\...\samples')
fpath = os.path.join(fdir, 'fname.ext')
I'm currently using the code below to find a configuration file in the current project
kRootPath = os.path.dirname(os.path.abspath(__file__))
kConfRootPath = os.path.join(kRootPath, '..', 'ConfigFiles' )
This currently works for my original project. I now have multiple projects that want to use this include file. I need the location of the file that called the include instead of the include itself.
Details:
Project 1
Config
1a.cfg
1b.cfg
Code
code1a.py
code1b.py
theInclude.py
Project 2
Config
2a.cfg
2b.cfg
Code
code2a.py
code2b.py
If all code files include theInclude.py then the include file should return the configure Path for:
project 1 if included from code1a.py or code1b.py
project 2 if included from code2a.py or code2b.py
I am using an Automation project with following structure
C:\Project_Name\Scripts
Under script folder, I have my python automation modules, libraries and other required data folders.
But I want this project not only restricted to C: drive. I should seamlessly able to execute by scripts from any drive, any parent directory of "Project_Name" folder.
I have tried the following. But it is always not true if I try to trigger my test from jenkins.
os.path.join(os.path.dirname(os.path.abspath(__file__)))
Think about this tree:
Project
|
|--Scripts
| |
| |--__init__.py
| |--use_text.py
| |--text.txt
|
|--run_me.py
and imagine we have to get text.txt path in use_text.py
Step 1. in run_me.py get the path of project directory, and save it as a environ using os. (example:)
import os
from pathlib import Path
path = Path(__file__).parent # replace it with your preferred library's version
os.environ['PROJECT_BASE_DIR'] = str(path)
from Scripts.use_text import use
use()
now in use_text.py
import os
text_path = 'Scripts/text.txt'
project_path = os.getenv('PROJECT_BASE_DIR')
text_real_path = os.path.join(project_path, text_path)
def use():
# do whatever you want with text_real_path
I got a way to get the real path of the file no matter where the project directory been placed.
import os
os.path.join((os.path.realpath(__file__)).split("<Base Directory Name>")[0], "<relative path till the file>")
this worked as well, with simple path.
I'm trying to make a python package and I have most of the things already setup by when I try to install the library from Github here, it installs everything except for the folder called champs and it's files
This is my File directory structure
LeagueYue
champs
-Lname_num.json
-Lname_Uname.json
-num_Uname.json
-__init__.py
-champion_files.py
-external.py
-match.py
-rank.py
-status.py
-summoner.py
-requirements.txt
-setup.py
All the files are installed except for the folder and the files inside champs
As this question answers:
There are 2 ways to add the static files:
1) Include_package_data=True + MANIFEST.in
A MANIFEST.in file in the same directory of setup.py, that looks like this:
include src/static/*
include src/Potato/*.txt
2) Package_data in setup.py
package_data = {
'static': ['*'],
'Potato': ['*.txt']
}
Specify the files inside the setup.py.
Two of the files could probably be derived at runtime from num_Uname.json, but that's fine.
I do not yet see a data_files directive in https://github.com/CharmingMother/LeagueLib/blob/async/setup.py
Thomas Cokelaer suggests using an expression like
datafiles = [(datadir, list(glob.glob(os.path.join(datadir, '*'))))]
and then
setup(
...
data_files = datafiles,
)
in http://thomas-cokelaer.info/blog/2012/03/how-to-embedded-data-files-in-python-using-setuptools/
In your case this could be as simple as:
data_files = [('', ['champs/num_Uname.json'])],
Martin Thoma explains you should access them using filepath = pkg_resources.resource_filename(__name__, path)
in How to read a (static) file from inside a Python package?
When I Read The Fine Manual, this setup.cfg alternative surfaces:
[options.data_files]
...
data = data/img/logo.png, data/svg/icon.svg
suggesting a line like . = champs/num_Uname.json or champs = num_Uname.json