Is there a way to tell scons to use a particular file to setup the default environment? I am using TI DSPs and the compiler is something different than cc; I'd like to have one "environment file" that defines where the compiler is, and what the default flags are, and then be able to use this for several projects.
Any suggestions?
You can use the normal python utilities to read a file or process XML and then import it into your env. If you don't have some external file that you need to import into SCons, then you can simply encode the environment in the scons file. If, for some reason, your environment is defined in a Perl dictionary ( as in my case...) you can either try to use PyPerl or convert the Perl dictionary into YAML and then read the YAML into python. ( I was able to do the later, but not the former).
Let's say you simply have a file that you need to read which has environment variables in the form:
ENV_VAR1 ENV_VAL1
ENV_VAR2 ENV_VAL2
...
You could import this into your SConstruct.py file like:
import os
env_file = open('PATH_TO_ENV_FILE','r')
lines = env.file.readlines()
split_regex = re.compile('^(?P<env_var>[\w_]+) *(?P<env_val>.*)')
for line in lines:
regex_search = split_regex.search(line)
if regex_search:
env_var = regex_search.group('env_var')
env_val = regex_search.group('env_val').strip()
os.environ[env_var] = env_val
base_env = Environment(ENV=os.environ)
# even though the below lines seem redundant, it was necessary in my build
# flow...
for key in os.environ.iterkeys():
base_env[key] = os.environ[key]
If you want to stick this ugliness inside a different file and then import it from your main SConstruct.py file, you can add the following to enable access to the 'Environment' class from your other file:
from SCons.Environment import *
Then in your main SConstruct.py file, import the env file like:
from env_loader import *
SInclusion file:
...
myenv = Environment(...)
...
SConstruct file:
...
execfile('SInclusion')
...
myenv.Object(...)
...
Related
I am using Python 3.9.2 with sphinx-build 3.5.2. I have created a project with the following directory and file structure
core_utilities
|_core_utilities
| |_ __init__.py
| |_read_files.py
|_docs
|_ sphinx
|_Makefile
|_conf.pg
|_source
| |_conf.py
| |_index.rst
| |_read_files.rst
|_build
The read_files.py file contains the following code. NOTE: I simplified this example so it would not have distracting information. Within this code their is one class that contains one member function to read a text file, look for a key word and read the variable to the right of the keyword as a numpy.float32 variable. The stand-alone function written to read in a csv file with specific data types and save it to a pandas dataframe.
# Import packages here
import os
import sys
import numpy as np
import pandas as pd
from typing import List
class ReadTextFileKeywords:
def __init__(self, file_name: str):
self.file_name = file_name
if not os.path.isfile(file_name):
sys.exit('{}{}{}'.format('FATAL ERROR: ', file_name, ' does not exist'))
# ----------------------------------------------------------------------------
def read_double(self, key_words: str) -> np.float64:
values = self.read_sentence(key_words)
values = values.split()
return np.float64(values[0])
# ================================================================================
# ================================================================================
def read_csv_columns_by_headers(file_name: str, headers: List[str],
data_type: List[type],
if not os.path.isfile(file_name):
sys.exit('{}{}{}'.format('FATAL ERROR: ', file_name, ' does not exist'))
dat = dict(zip(headers, data_type))
df = pd.read_csv(file_name, usecols=headers, dtype=dat, skiprows=skip)
return df
The conf.py file as the following information in it.
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('../../../core_utilities'))
# -- Project information -----------------------------------------------------
project = 'Core Utilities'
copyright = 'my copyright'
author = 'my name'
# The full version, including alpha/beta/rc tags
release = '0.1.0'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.todo', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc',
'sphinx.ext.autosummary', 'sphinx.ext.githubpages']
autodoc_member_order = 'groupwise'
autodoc_default_flags = ['members', 'show-inheritance']
autosummary_generate = True
autodock_mock_imports = ['numpy']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'nature'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
The index.rst file has the following information.
. Core Utilities documentation master file, created by
sphinx-quickstart
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Core Utilities's documentation!
==========================================
.. toctree::
:maxdepth: 2
:caption: Contents:
read_files
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
and the read_files.rst file has the following information
**********
read_files
**********
The ``read_files`` module contains methods and classes that allow a user to read different
types of files in different ways. Within the module a user will find functionality that
allows them to read text files and csv files by columns and keywords. This module
also contains functions to read sqlite databases, xml files, json files, yaml files,
and html files
.. autoclass:: test.ReadTextFileKeywords
:members:
.. autofunction:: test.read_csv_columns_by_headers
In addition, I am using a virtual environment that is active when I try to compile this with command line. I run the following command to compile html files with sphinx.
sphinx-build -b html source build
When I run the above command it fails with the following error.
WARNING: autodoc: failed to import class 'ReadTextFileKeywords' from module 'read_files'; the following exception was raised; No module named pandas.
If I delete the line from pandas import pd and then delete the function read_csv_columns_by_headers along with the call to the function in the read_files.rst file, everything compiles fine. It appears that for some reason sphinx is able to find numpy, but it for some reason does not seem to recognize pandas, although both exist in the virtual environment and were loaded with a pip3 install statement. Does anyone know why sphinx is able to find other modules, but not pandas?
Add pandas to autodoc
autodoc_mock_imports = ['numpy','pandas']
I have a main config.py file and then specific client config files, e.g. client1_config.py.
What I'd like to do is import all variables within my client1_config.py file into my config.py file. The catch is I want to do this flexibly at runtime according to an environment variable. It would look something like this:
import os
import importlib
client = os.environ['CLIENT']
client_config = importlib.import_module(
'{client}_config'.format(
client=client))
from client_config import *
This code snippet returns the following error: ModuleNotFoundError: No module named 'client_config'
Is it possible (and how) to achieve what I'm trying to do or Python does not support this kind of importing at all?
The call to import_module already imports the client configuration. from client_config import * assumes that client_config is the name of the module you are trying to import, just as import os will import the module os even if you create a variable os beforehand:
os = "sys"
import os # still imports the os module, not the sys module
In the following, assume that we have a client1_config.py which just contains one variable:
dummy = True
To add its elements to the main namespace of config.py so that you can access them directly, you can do the following:
import importlib
client = "client1"
# Import the client's configuration
client_config = importlib.import_module(f"{client}_config")
print(client_config.dummy) # True
# Add all elements from client_config
# to the main namespace:
globals().update({v: getattr(client_config, v)
for v in client_config.__dict__
if not v.startswith("_")})
print(dummy) # True
However, I would suggest to access the client's configuration as config.client for clarity and to avoid the client's configuration file overwriting values in the main configuration file.
I desesperatly searching for 1 liner load config package in Python 3...
In python 2, there is :
https://pypi.python.org/pypi/config/0.3.7
from config import Config; cfg = Config(file('D://config.py', 'r'))
There is
https://pypi.python.org/pypi/loadconfig
but it does not seem loading file...
Main goal is to install this package in all the python envs and load
one big config file for all envs.
Use case:
You have a single piece of python, you want to run on several hardware/OS and python envs (conda,...) without changing the code...
Method 1:
1) Define a generic config files for all Envs needed.
2) At RunTime, determine the environnment (linux, username, ) and
define the absolute path root repository directory with config file.
3) Single piece of code, running in different envs without changing the code, the envs...
From similar functionnalities than config
https://pypi.python.org/pypi/config/0.3.7
Security is not guaranted
config.py should not contains harmful python code.
from attrdict import AttrDict; f= open('USER_ROOT/config.py'); cfg = AttrDict(eval(f.read())); f.close()
cfg.key1
cfg.mykey2
cfg.mykey4
How do I load a python module, that is not built in. I'm trying to create a plugin system for a small project im working on. How do I load those "plugins" into python? And, instaed of calling "import module", use a string to reference the module.
Have a look at importlib
Option 1: Import an arbitrary file in an arbiatrary path
Assume there's a module at /path/to/my/custom/module.py containing the following contents:
# /path/to/my/custom/module.py
test_var = 'hello'
def test_func():
print(test_var)
We can import this module using the following code:
import importlib.machinery
myfile = '/path/to/my/custom/module.py'
sfl = importlib.machinery.SourceFileLoader('mymod', myfile)
mymod = sfl.load_module()
The module is imported and assigned to the variable mymod. We can then access the module's contents as:
mymod.test_var
# prints 'hello' to the console
mymod.test_func()
# also prints 'hello' to the console
Option 2: Import a module from a package
Use importlib.import_module
For example, if you want to import settings from a settings.py file in your application root folder, you could use
_settings = importlib.import_module('settings')
The popular task queue package Celery uses this a lot, rather than giving you code examples here, please check out their git repository
I found a solution to add files in library.zip via: Extend py2exe to copy files to the zipfile where pkg_resources can load them.
I can access to my file when library.zip is not include the exe.
I add a file : text.txt in directory: foo/media in library.zip.
And I use this code:
import pkg_resources
import zipfile
from cStringIO import StringIO
my_data = pkg_resources.resource_string(__name__,"library.zip")
filezip = StringIO(my_data)
zip = zipfile.ZipFile(filezip)
data = zip.read("foo/media/text.txt")
I try to use pkg_resources but I think that I don't understand something because I could open directly "library.zip".
My question is how can I do this when library.zip is embed in exe?
Best Regards
Jean-Michel
I cobbled together a reasonably neat solution to this, but it doesn't use pkg_resources.
I need to distribute productivity tools as standalone EXEs, that is, all bundled into the one .exe file. I also need to send out notifications when these tools are used, which I do via the Logging API, using file-based configuration. I emded the logging.cfg fileto make it harder to effectively switch-off these notifications i.e. by deleting the loose file... which would probably break the app anyway.
So the following is the interesting bits from my setup.py:
LOGGING_CFG = open('main/resources/logging.cfg').read()
setup(
name='productivity-tool',
...
# py2exe extras
console=[{'script': productivity_tool.__file__.replace('.pyc', '.py'),
'other_resources': [(u'LOGGINGCFG', 1, LOGGING_CFG)]}],
zipfile=None,
options={'py2exe': {'bundle_files': 1, 'dll_excludes': ['w9xpopen.exe']}},
)
Then in the startup code for productivity_tool.py:
from win32api import LoadResource
from StringIO import StringIO
from logging.config import fileConfig
...
if __name__ == '__main__':
if is_exe():
logging_cfg = StringIO(LoadResource(0, u'LOGGINGCFG', 1))
else:
logging_cfg = 'main/resources/logging.cfg'
fileConfig(logging_cfg)
...
Works a treat!!!
Thank you but I found the solution
my_data = pkg_resources.resource_stream("__main__",sys.executable) # get lib.zip file
zip = zipfile.ZipFile(my_data)
data = zip.read("foo/media/doc.pdf") # get my data on lib.zip
file = open(output_name, 'wb')
file.write(data) # write it on a file
file.close()
Best Regards
You shouldn't be using pkg_resources to retrieve the library.zip file. You should use it to retrieve the added resource.
Suppose you have the following project structure:
setup.py
foo/
__init__.py
bar.py
media/
image.jpg
You would use resource_string (or, preferably, resource_stream) to access image.jpg:
img = pkg_resources.resource_string(__name__, 'media/image.jpg')
That should "just work". At least it did when I bundled my media files in the EXE. (Sorry, I've since left the company where I was using py2exe, so don't have a working example to draw on.)
You could also try using pkg_resources.resource_filename(), but I don't think that works under py2exe.