We have a large git structure and inside that we wanted to create some python libraries, where some is referencing the other. Since we pull the git all at once we don't want setup tools reference each other via the VCS repository reference.
Currently what I was able to do was this:
from setuptools import setup
import setuptools.command.install
import pip._internal
import os
try:
era_root = os.environ.get("ERA_ROOT")
if not os.path.isdir(era_root):
raise ValueError("ERA_ROOT must point to a valid directory")
except KeyError as k:
raise KeyError("Please specify the ERA_ROOT environment variable")
protobufsDependency = "{}/Products/PythonLibs/Protobufs".format(
era_root)
class my_install(setuptools.command.install.install):
def run(self):
pip._internal.main(['install', '-I', protobufsDependency])
setuptools.command.install.install.run(self)
setup(name='replication_tools',
version='1.0',
packages=["FakeAgent", "EventGenerator"],
cmdclass={"install": my_install},
)
However this is problematic because if someone is installing this library with --user or some other params I would have to manually name them all in the pip._internal.main . So my question is is there some way I can reference with setuptools.setup the protobufsDependency directly as if it was a url package?
Something like (as this doesnt work but something like that)
setup(name='replication_tools',
version='1.0',
packages=["FakeAgent", "EventGenerator"],
install_requires=[
protobufsDependency
]
)
Ok it seems that I can use direct reference link
setup(name='replication_tools',
version='1.0',
packages=["FakeAgent", "EventGenerator"],
install_requires=[
"protobufs-generated # file://C:/PathToMyLib"
]
)
Related
I have a pathlib.Path object and want to find the absolute path to a parent folder, called "BASE". However, I do not know how much further up the device tree the "BASE" folder is. I just know the pathlib.Path will contain a folder called "BASE".
Example:
import pathlib
# the script is located at:
my_file_dir = pathlib.Path(__file__).parent
# some pathlib function to find the absolute path to BASE
# input_1: /some/path/BASE/some/more/layers/script.py
# output_1: /some/path/BASE
# input_2: /usr/BASE/script.py
# output_2: /usr/BASE
In earlier python versions I would use os.path, os.path.split() and search for the string to get the directory. Now, it seems, pathlib should be used for these things. But how?
EDIT:
Here is how I solved it using pathlib.
def get_base_dir(current_dir: Path, base: str) -> Path:
base_parts = current_dir.parts[:current_dir.parts.index(base)+1]
return Path(*base_parts)
Honestly, #victor__von__doom's solution is better.
I think you could do this more simply using some basic string processing on the command line args:
import sys
script_path, my_file_dir = sys.argv[0], ''
try:
my_file_dir = script_path[:script_path.index('BASE')]
except ValueError:
print("No path found back to BASE"); exit()
sys seems like a much easier option than pathlib here.
Since JupyterLab 3.x jupyter-server is used instead of the classic notebook server, and the following code does not list servers served with jupyter_server:
from notebook import notebookapp
notebookapp.list_running_servers()
None
What still works for the file/notebook name is:
from time import sleep
from IPython.display import display, Javascript
import subprocess
import os
import uuid
def get_notebook_path_and_save():
magic = str(uuid.uuid1()).replace('-', '')
print(magic)
# saves it (ctrl+S)
# display(Javascript('IPython.notebook.save_checkpoint();')) # Javascript Error: IPython is not defined
nb_name = None
while nb_name is None:
try:
sleep(0.1)
nb_name = subprocess.check_output(f'grep -l {magic} *.ipynb', shell=True).decode().strip()
except:
pass
return os.path.join(os.getcwd(), nb_name)
But it's not pythonic nor fast
How to get the current running server instances - and so e.g. the current notebook file?
Migration to jupyter_server should be as easy as changing notebook to jupyter_server, notebookapp to serverapp and changing the appropriate configuration files - the server-related codebase is largely unchanged. In the case of listing servers simply use:
from jupyter_server import serverapp
serverapp.list_running_servers()
I have been using "ipython --script" to automatically save a .py file for each ipython notebook so I can use it to import classes into other notebooks. But this recenty stopped working, and I get the following error message:
`--script` is deprecated. You can trigger nbconvert via pre- or post-save hooks:
ContentsManager.pre_save_hook
FileContentsManager.post_save_hook
A post-save hook has been registered that calls:
ipython nbconvert --to script [notebook]
which behaves similarly to `--script`.
As I understand this I need to set up a post-save hook, but I do not understand how to do this. Can someone explain?
[UPDATED per comment by #mobius dumpling]
Find your config files:
Jupyter / ipython >= 4.0
jupyter --config-dir
ipython <4.0
ipython locate profile default
If you need a new config:
Jupyter / ipython >= 4.0
jupyter notebook --generate-config
ipython <4.0
ipython profile create
Within this directory, there will be a file called [jupyter | ipython]_notebook_config.py, put the following code from ipython's GitHub issues page in that file:
import os
from subprocess import check_call
c = get_config()
def post_save(model, os_path, contents_manager):
"""post-save hook for converting notebooks to .py scripts"""
if model['type'] != 'notebook':
return # only do this for notebooks
d, fname = os.path.split(os_path)
check_call(['ipython', 'nbconvert', '--to', 'script', fname], cwd=d)
c.FileContentsManager.post_save_hook = post_save
For Jupyter, replace ipython with jupyter in check_call.
Note that there's a corresponding 'pre-save' hook, and also that you can call any subprocess or run any arbitrary code there...if you want to do any thing fancy like checking some condition first, notifying API consumers, or adding a git commit for the saved script.
Cheers,
-t.
Here is another approach that doesn't invoke a new thread (with check_call). Add the following to jupyter_notebook_config.py as in Tristan's answer:
import io
import os
from notebook.utils import to_api_path
_script_exporter = None
def script_post_save(model, os_path, contents_manager, **kwargs):
"""convert notebooks to Python script after save with nbconvert
replaces `ipython notebook --script`
"""
from nbconvert.exporters.script import ScriptExporter
if model['type'] != 'notebook':
return
global _script_exporter
if _script_exporter is None:
_script_exporter = ScriptExporter(parent=contents_manager)
log = contents_manager.log
base, ext = os.path.splitext(os_path)
py_fname = base + '.py'
script, resources = _script_exporter.from_filename(os_path)
script_fname = base + resources.get('output_extension', '.txt')
log.info("Saving script /%s", to_api_path(script_fname, contents_manager.root_dir))
with io.open(script_fname, 'w', encoding='utf-8') as f:
f.write(script)
c.FileContentsManager.post_save_hook = script_post_save
Disclaimer: I'm pretty sure I got this from SO somwhere, but can't find it now. Putting it here so it's easier to find in future (:
I just encountered a problem where I didn't have rights to restart my Jupyter instance, and so the post-save hook I wanted couldn't be applied.
So, I extracted the key parts and could run this with python manual_post_save_hook.py:
from io import open
from re import sub
from os.path import splitext
from nbconvert.exporters.script import ScriptExporter
for nb_path in ['notebook1.ipynb', 'notebook2.ipynb']:
base, ext = splitext(nb_path)
script, resources = ScriptExporter().from_filename(nb_path)
# mine happen to all be in Python so I needn't bother with the full flexibility
script_fname = base + '.py'
with open(script_fname, 'w', encoding='utf-8') as f:
# remove 'In [ ]' commented lines peppered about
f.write(sub(r'[\n]{2}# In\[[0-9 ]+\]:\s+[\n]{2}', '\n', script))
You can add your own bells and whistles as you would with the standard post save hook, and the config is the correct way to proceed; sharing this for others who might end up in a similar pinch where they can't get the config edits to go into action.
I am working with Python 3.4 on Windows 7. My setup file is as follows:
from cx_Freeze import setup, Executable, sys
exe=Executable(
script="XYZ.py",
base="Win32Gui",
)
includefiles=[]
includes=[]
excludes=[]
packages=[]
setup(
version = "1.0",
description = "XYZ",
author = "MAX",
name = "AT",
options = {'build_exe': {'excludes':excludes,'packages':packages,'include_files':includefiles}},
executables = [exe]
)
from distutils.core import setup
import py2exe, sys, os, difflib
sys.argv.append('py2exe')
setup(
options = {'py2exe': {'bundle_files': 1}},
console = [{'script': "XYZ.py"}],
zipfile = None,
)
When the obtained exe is run, an error pops up saying:
...
File "C:\Python34\Lib\site-packages\win32com\client\CLSIDToClass.py", line 46, in GetClass
return mapCLSIDToClass[clsid]
KeyError: '{00020970-0000-0000-C000-000000000046}'
I just can't figure out the problem here. Help, please.
Thanks.
I've just figured out that the problem with EnsureDispatch is within gencache module, it assumes to be in read-only mode when an executable is built with cx_freeze.
The following lines allow cache to be built inside AppData\Local\Temp\gen_py\#.#\ directory in Windows 7 x64:
from win32com.client import gencache
if gencache.is_readonly:
gencache.is_readonly = False
gencache.Rebuild() #create gen_py folder if needed
References:
py2exe/pyinstaller and DispatchWithEvents answer
py2exe.org: UsingEnsureDispatch
P. S. Performance is much better with static dispatch
You are using static proxy which is generated on your disk and which has the compiled executable trouble finding. If you do not know what the static proxy is, you are probably using win32com.client.gencache.EnsureDispatch which generates static proxy automatically.
The easiest way to fix the problem is to use dynamic proxy by using win32com.client.dynamic.Dispatch. Static proxy has some benefits, but there is a high possibility that you do not need it.
You can find more information about static and dynamic proxies to COM objects here: http://timgolden.me.uk/python/win32_how_do_i/generate-a-static-com-proxy.html
I found a solution to add files in library.zip via: Extend py2exe to copy files to the zipfile where pkg_resources can load them.
I can access to my file when library.zip is not include the exe.
I add a file : text.txt in directory: foo/media in library.zip.
And I use this code:
import pkg_resources
import zipfile
from cStringIO import StringIO
my_data = pkg_resources.resource_string(__name__,"library.zip")
filezip = StringIO(my_data)
zip = zipfile.ZipFile(filezip)
data = zip.read("foo/media/text.txt")
I try to use pkg_resources but I think that I don't understand something because I could open directly "library.zip".
My question is how can I do this when library.zip is embed in exe?
Best Regards
Jean-Michel
I cobbled together a reasonably neat solution to this, but it doesn't use pkg_resources.
I need to distribute productivity tools as standalone EXEs, that is, all bundled into the one .exe file. I also need to send out notifications when these tools are used, which I do via the Logging API, using file-based configuration. I emded the logging.cfg fileto make it harder to effectively switch-off these notifications i.e. by deleting the loose file... which would probably break the app anyway.
So the following is the interesting bits from my setup.py:
LOGGING_CFG = open('main/resources/logging.cfg').read()
setup(
name='productivity-tool',
...
# py2exe extras
console=[{'script': productivity_tool.__file__.replace('.pyc', '.py'),
'other_resources': [(u'LOGGINGCFG', 1, LOGGING_CFG)]}],
zipfile=None,
options={'py2exe': {'bundle_files': 1, 'dll_excludes': ['w9xpopen.exe']}},
)
Then in the startup code for productivity_tool.py:
from win32api import LoadResource
from StringIO import StringIO
from logging.config import fileConfig
...
if __name__ == '__main__':
if is_exe():
logging_cfg = StringIO(LoadResource(0, u'LOGGINGCFG', 1))
else:
logging_cfg = 'main/resources/logging.cfg'
fileConfig(logging_cfg)
...
Works a treat!!!
Thank you but I found the solution
my_data = pkg_resources.resource_stream("__main__",sys.executable) # get lib.zip file
zip = zipfile.ZipFile(my_data)
data = zip.read("foo/media/doc.pdf") # get my data on lib.zip
file = open(output_name, 'wb')
file.write(data) # write it on a file
file.close()
Best Regards
You shouldn't be using pkg_resources to retrieve the library.zip file. You should use it to retrieve the added resource.
Suppose you have the following project structure:
setup.py
foo/
__init__.py
bar.py
media/
image.jpg
You would use resource_string (or, preferably, resource_stream) to access image.jpg:
img = pkg_resources.resource_string(__name__, 'media/image.jpg')
That should "just work". At least it did when I bundled my media files in the EXE. (Sorry, I've since left the company where I was using py2exe, so don't have a working example to draw on.)
You could also try using pkg_resources.resource_filename(), but I don't think that works under py2exe.