Given the following Python module layout:
app/
├── drivers
│ ├── mydriver
│ │ ├── driver.py
│ │ └── __init__.py
│ └── __init__.py
├── __init__.py
└── main.py
I am trying to dynamically import the "mydriver" module in main.py:
import os
import importlib
driver_dir = os.path.join(os.path.dirname(__file__), 'drivers')
loader_details = (
importlib.machinery.ExtensionFileLoader,
importlib.machinery.EXTENSION_SUFFIXES
)
finder = importlib.machinery.FileFinder(driver_dir, loader_details)
spec = finder.find_spec('mydriver')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# The following line produces AttributeError: module 'mydriver' has no attribute 'driver'
driver = getattr(module, 'driver')
drivers/mydriver/__init__.py contains the following:
from . import driver
print("TEST")
So the result is the Attribute error as written in the inline comment. The "print()" from __init__.py is also not being executed.
Any hints why the module is apparently not being evaluated?
While I haven't found a root cause, I did find a (not so pretty) workaround. For some reason, the module can not be executed if it was found using the FileFinder. It does however execute if I do the following:
sys.path.insert(0, driver_dir)
spec = importlib.util.find_spec('mydriver')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
So in short, I don't know what Python wants from a Finder to also execute modules, not just files. Well, at least I have working code for now...
Related
I have a a top level directory and then few subdirectories underneath it which in turn have few subdirectories and files.
the tree view looks like this:-
treeview
├── abc
│ ├── pqr
│ │ ├── 1
│ │ ├── 2
│ │ └── 3
│ └── sty
└── xyz
Now I want my script to ignore walking into "abc" and "1"(this is a directory and not a file).
I have come up with the below script:-
import os
import shutil
files_to_keep = []
files_grandlist = []
dirs_to_keep = []
dirs_grandlist = []
rootpath="./treeview"
exclude = ['./treeview/abc', './treeview/abc/pqr/1']
exclude_temp = set(['abc','1'])
for path in exclude:
for root, dirs, files in os.walk(path):
dirs[:] = [d for d in dirs if d not in exclude_temp]
dirs_to_keep.append(root)
print("================dirs to keep======")
for i in dirs_to_keep:
print(i)
for root, dirs, files in os.walk(rootpath):
dirs_grandlist.append(root)
print("===============grandlist==========")
for i in dirs_grandlist:
print(i)
print("filter============================")
for i in list(set(dirs_grandlist)^set(dirs_to_keep)):
print(i)
This gives me output like below when run:
================dirs to keep======
./treeview/abc
./treeview/abc/pqr
./treeview/abc/pqr/2
./treeview/abc/pqr/3
./treeview/abc/sty
./treeview/abc/pqr/1
===============grandlist==========
./treeview
./treeview/abc
./treeview/abc/pqr
./treeview/abc/pqr/1
./treeview/abc/pqr/2
./treeview/abc/pqr/3
./treeview/abc/sty
./treeview/xyz
filter============================
./treeview
./treeview/xyz
The idea is to capture a list of directories/subdirectories under treeview top level directory and then capture the same information for a list of excluded directories.
the output under "filter============================" line should give me a list of directories which I want to remove from the filesystem.
Appreciate any help here.
I have 2 locations with folders and multiple subfolders as follows:
Location 1
2022_10
├── FolderA
│ └── Version 3
│ └── FolderA.docx
├── FolderB
│ └── Version 2
| └── FolderB.docx
├── FolderC
│ └── Version 2
│ └── FolderC.docx
├── FolderD
│ └── Version 3
└── FolderD.docx
Location 2
2022_10
├── FolderA
│ └── Version 1
│ └── FolderA.docx
│ └── Version 2
│ └── FolderA.docx
├── FolderB
│ └── Version 1
| └── FolderB.docx
├── FolderC
│ └── Version 1
│ └── FolderC.docx
├── FolderD
│ └── Version 1
│ └── FolderA.docx
│ └── Version 2
│ └── FolderA.docx
Location 1 has the latest version of subfolders that I need to copy to location 2 (centralized repository), but respecting the folder structure and the previous folder versions that already exist there.
At the end, my objective is to have location 2 ingesting only the latest version as follows, if the version already exist the script should overwrite the latest version from Location 1 into Location 2.
Location 2
2022_10
├── FolderA
│ └── Version 1
│ └── FolderA.docx
│ └── Version 2
│ └── FolderA.docx
│ └── Version 3
│ └── FolderA.docx
├── FolderB
│ └── Version 1
| └── FolderB.docx
│ └── Version 2
| └── FolderB.docx
With some help I got a script up and running which replicates the Location 1 structure, that part is done, but now I'm thinking on the best way to have the same script to accomplish the copy between locations as well perhaps using shutil with a multi-option menu: -A for generating location 1 structure, and -C to do the copy operation from Location 1 to Location 2.
Here is the code I have:
#!/usr/bin/env python3
import docx, os, glob, re, shutil, sys
from pathlib import Path
#Taking the folder to process from user input (second argument is considered)
folder = sys.argv[1]
#Function to create a new folder if the path does not exist
def create_dir(path):
is_exist = os.path.exists(path)
if not is_exist:
os.makedirs(path)
for file in glob.glob(os.path.join(folder, '*.docx')):
main_folder = os.path.join(folder,Path(file).stem)
file_name = os.path.basename(file)
#Getting the version information from every word
doc = docx.Document(file).paragraphs[6].text
#Getting the version number line = Version Number: (.*) and extracting the number only portion
version_number = re.search("(Version Number: (.*))", doc).group(1)
version_subfolder = version_number.split(':')[1].strip()
# path to actual sub_folder with version_no
version_subfolder = os.path.join(main_folder, version_subfolder)
# destination path
dest_file_path = os.path.join(version_subfolder, file_name)
for i in [main_folder,version_subfolder]:
create_dir(i) # function call
# to move the file to the corresponding version folder (overwrite if exists)
if os.path.exists(dest_file_path):
os.remove(dest_file_path)
shutil.move(file, version_subfolder)
else:
shutil.move(file, version_subfolder)
I wish to deploy a package to PyPi using setuptools. However, the core part of the package is actually written in Fortran, and I am using f2py to wrap it in python. Basically the project's structure looks like this:
my_project
license.txt
README.md
setup.py
my_project
init.py
myfunc.py
hello.so
The module myfunc.py imports hello.so (import my_project.hello) which can then be used by functions inside myfunc.py. This works perfectly on my machine.
Then I tried standard setuptools installation: sudo python3 setup.py install on my Ubuntu, and it gets installed perfectly. But unfortunately, while importing, it throws ModuleNotFoundError: No module named 'hello'.
Now, from what I understand, on Linux based systems, for python, the shared libraries *.so are stored in /usr/lib/python3/dist-packages/. So I manually copied this hello.so there, and I got a working package! But of course this works only locally. What I would like to do is to tell setuptools to include hello.so inside the python-egg and automatically do the copying etc so that when a user uses pip3 install my_package, they will have access to this shared library automatically. I can see that numpy has somehow achieved that but even after looking at their code, I haven't been able to decode how they did it. Can someone help me with this? Thanks in advance.
You can achieve this with a setup.py file like this (simplified version, keep only the relevant parts for building external modules)
import os
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
class f2py_Extension(Extension):
def __init__(self, name, sourcedirs):
Extension.__init__(self, name, sources=[])
self.sourcedirs = [os.path.abspath(sourcedir) for sourcedir in sourcedirs]
self.dirs = sourcedirs
class f2py_Build(build_ext):
def run(self):
for ext in self.extensions:
self.build_extension(ext)
def build_extension(self, ext):
# compile
for ind,to_compile in enumerate(ext.sourcedirs):
module_loc = os.path.split(ext.dirs[ind])[0]
module_name = os.path.split(to_compile)[1].split('.')[0]
os.system('cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name))
setup(
name="foo",
ext_modules=[f2py_Extension('fortran_external',['foo/one.F90','foo/bar/two.F90'])],
cmdclass=dict(build_ext=f2py_Build),
)
The essential parts for building an external module are ext_modules and cmdclass in setup(...). ext_modules is just a list of Extension instances, each of which describes a set of extension modules. In the setup.py above, I tell ext_modules I want to create two external modules with two source files foo/test.F90 and foo/bar/two.F90. Based on ext_modules, cmdclass is responsible for compiling the two modules, in our case, the command for compiling the module is
'cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name)
Project structure before installation
├── foo
│ ├── __init__.py
│ ├── bar
│ │ └── two.F90
│ └── one.F90
└── setup.py
Project structure after python setup.py install
├── build
│ └── bdist.linux-x86_64
├── dist
│ └── foo-0.0.0-py3.7-linux-x86_64.egg
├── foo
│ ├── __init__.py
│ ├── __pycache__
│ │ └── __init__.cpython-37.pyc
│ ├── bar
│ │ ├── two.F90
│ │ └── two.cpython-37m-x86_64-linux-gnu.so
│ ├── one.F90
│ └── one.cpython-37m-x86_64-linux-gnu.so
├── foo.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ └── top_level.txt
└── setup.py
The two source files one.F90 and two.F90 are very simple
one.F90
module test
implicit none
contains
subroutine add(a)
implicit none
integer :: a
integer :: b
b = a + 1
print *, 'one',b
end subroutine add
end module test
two.F90
module test
implicit none
contains
subroutine add(a)
implicit none
integer :: a
integer :: b
b = a + 2
print *, 'two',b
end subroutine add
end module test
After I installed the package, I can successfully run
>>> from foo.bar.two import test
>>> test.add(5)
two 7
and
>>> from foo.one import test
>>> test.add(5)
one 6
Here is an approach based on F2PY's documentation (the example there covers building multiple F2PY modules, and multiple source files per module), making use of numpy.distutils, that supports Fortran source files.
The structure of a minimal example with multiple F2PY extension modules is based on a src directory layout. It is not necessary/required, but has the advantage that the test routine cannot run unless the package has been installed successfully.
Source layout
my_project
|
+-- src
| |
| +-- my_project
| |
| +-- __init__.py
| +-- mod1.py
| +-- funcs_m.f90
| +-- two
| |
| +-- pluss2.f90
| +-- times2.f90
|
+-- test_my_project.py
+-- setup.py
setup.py
from setuptools import find_packages
from numpy.distutils.core import setup, Extension
ext1 = Extension(name='my_project.modf90',
sources=['src/my_project/funcs_m.f90'],
f2py_options=['--quiet'],
)
ext2 = Extension(name='my_project.oldf90',
sources=['src/my_project/two/plus2.f90', 'src/my_project/two/times2.f90'],
f2py_options=['--quiet'],
)
setup(name="my_project",
version="0.0.1",
package_dir={"": "src"},
packages=find_packages(where="src"),
ext_modules=[ext1, ext2])
__init__.py
The __init__.py file is empty. (Can e.g. import the F2PY modules here if desired)
mod1.py
def add(a, b):
""" add inputs a and b, and return """
return a + b
funcs_m.f90
module funcs_m
implicit none
contains
subroutine add(a, b, c)
integer, intent(in) :: a
integer, intent(in) :: b
integer, intent(out) :: c
c = a + b
end subroutine add
end module funcs_m
plus2.f90
subroutine plus2(x, y)
integer, intent(in) :: x
integer, intent(out) :: y
y = x + 2
end subroutine plus2
times2.f90
subroutine times2(x, y)
integer, intent(in) :: x
integer, intent(out) :: y
y = x * 2
end subroutine times2
test_my_project.py
import my_project.mod1
import my_project.oldf90
import my_project.modf90
print("mod1.add: 1 + 2 = ", my_project.mod1.add(1, 2))
print("modf90.funcs_m.add: 1 + 2 = ", my_project.modf90.funcs_m.add(1, 2))
x = 1
x = my_project.oldf90.plus2(x)
print("oldf90.plus2: 1 + 2 = ", x)
x = my_project.oldf90.times2(x)
print("oldf90.plus2: 3 * 2 = ", x)
Installing
Now, one can use pip to install the package. There are several advantages to using pip (including ease of upgrading, or uninstalling) as opposed to setup.py install (but this can still be used for building the package for distribution!). From the directory containing setup.py:
> python -m pip install .
Testing
And then, to test the just installed package
> python test_my_project.py
mod1.add: 1 + 2 = 3
modf90.funcs_m.add: 1 + 2 = 3
oldf90.plus2: 1 + 2 = 3
oldf90.plus2: 3 * 2 = 6
This setup has been tested with success on Windows 10 (with ifort), on Ubuntu 18.04 (with gfortran) and on MacOS High Sierra (with gfortran), all with Python 3.6.3.
I am creating a CRUD web app using python and flask.
Here is the link to see the output:
http://zcds4327.pythonanywhere.com/
Here is the repository's link:
https://github.com/OCTRACORE/cs_proj_pro_1
Tree structure:
.
├── Password_creator.py
├── __pycache__
│ ├── Password_creator.cpython-38.pyc
│ ├── exec_prog.cpython-38.pyc
│ ├── global_data.cpython-38.pyc
│ └── table.cpython-38.pyc
├── exec_prog.py
├── global_data.py
├── passwd_file
├── requirements.txt
├── start.sh
├── static
│ ├── AddPasswdInp.css
│ ├── dispManipMenu.css
│ ├── favicon_io
│ │ ├── android-chrome-192x192.png
│ │ ├── apple-touch-icon.png
│ │ ├── favicon-32x32.png
│ │ ├── favicon.ico
│ │ └── site.webmanifest
│ ├── input.css
│ ├── show.css
│ └── showOutput.css
├── table.py
└── template
├── add.html
├── file-manip.html
├── input.html
├── modify.html
├── output_disp.html
├── semantic-error.html
└── str_main.html
Parent directory of the code:cs_proj_pro_1
Code snippet (from the file exec_prog.py):
#app.route('/generate/file-manip-upload',methods = ["GET","POST"])
def upload_file(): #for uploading the file
decrypted_pass_output_lst = []
decrypted_desc_output_lst = []
decrypted_password = []
decrypted_data = ""
Upload_key = "" #for storing the key
if request.method == "POST":
Uploadpassfile= request.files['Uploadpassfile'] #requesting the name of the required file
Uploadpassfile_name = secure_filename(Uploadpassfile.filename) #storing the name of the password saving file
Uploadkeyfile = request.files['Uploadkeyfile'] #requesting the name of the key storing file
Uploadkeyfile_name = secure_filename(Uploadkeyfile.filename) #storing the name of the key storing file
descfileUpload = request.files['descfileUpload'] #requesting the name of the file storing descriptions
descfileUpload_name = secure_filename(descfileUpload.filename) #storing the name of the file string descriptions
#print(Uploadkeyfile_name,Uploadpassfile_name,descfileUpload_name)
with open(Uploadkeyfile_name,"rb") as k: # reading the containing the key
Upload_key = k.read()
key_init_sec = Fernet(Upload_key)
with open(Uploadpassfile_name,"rb") as k: # reading the file containing the passwords by decrypting them
for i in k:
decrypted_pass_output_lst.append(str(key_init_sec.decrypt(i),encoding="utf-8"))
with open(descfileUpload_name,"rb") as k: #reading the file containing the descriptions by decrypting them
for i in k:
decrypted_desc_output_lst.append(str(key_init_sec.decrypt(i),encoding="utf-8"))
x = len(decrypted_pass_output_lst)
(GlobalData.password).clear() #clearing the GlobalData.password list to avoid piling up of data
for s in range(x): #for inserting the decrypted data inside the GlobalData.password list for display
data_decrypted = Table(decrypted_pass_output_lst[s],decrypted_desc_output_lst[s])
(GlobalData.password).append(data_decrypted)
Working directory:
/home/ZCDS4327/proj/cs_proj_pro_1
Environment used:
PythonAnywhere
When I try to save the passwords and their corresponding descriptions, I need to go to the file manipulation page by clicking on the save/upload / upload/save link present at the home page and output page respectively.
The routes of the pages are:
1.) http://zcds4327.pythonanywhere.com/generate/file-manip-disp
2.) http://zcds4327.pythonanywhere.com/generate/show-output
After that, I have to select three files from the local machine. One file will be used for storing the passwords by encrypting them. The other one will store the description of all the passwords by encrypting them. The third one will store the key used for encrypting both the files. After that, I have to click the submit button to save them.
The files finally get stored inside the ZCDS4327 folder.
To show the outputs again in the web app in the decrypted way I have to select the files containing the required data through the `uploading section of the page through the required buttons, then click on submit to receive the output.
I have created a virtual environment in my local machine which runs a replica of this program inside the localhost server. If I try to upload all the three files which have contents from the app hosted by the localhost server to the app hosted by the pythonanywhere server, I receive an internal server error. My aim is that the user is able to upload the files from whichever part of their system they want.
The error log
2020-10-07 08:47:16,087: Exception on /generate/file-manip-upload [POST]
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/ZCDS4327/proj/cs_proj_pro_1/exec_prog.py", line 155, in upload_file
with open(Uploadkeyfile_name,"rb") as k: # reading the containing the key
FileNotFoundError: [Errno 2] No such file or directory: 'keyfile.txt'
Here, I am the user as well as the developer.
So what shall I do in order to resolve this error? Every suggestion will be appreciated.
Notes:
The localhost server has been taken as an example. The problem, as I believe will be encountered in the case of any other server too.
The files saved by the user will get downloaded to their system. The code for that is not yet written and I do not think that in this case, it is required to know that.
How it's organized
Currently, this is my folder structure:
├ Websites
│ ├ Prototypes
│ │ ├ __init__.py
│ │ ├ Website.py
│ │ └ XML.py
│ ├ __init__.py
│ └ FunHouse.py
└ scrape_sites.py
This is FunHouse.py:
from Websites.Prototypes.Website import Website, Search
class FunHouse(Website):
def doStuff():
# does stuff
This is my __init__.py in the Websites folder:
from Websites.Prototypes.Website import Website
from Websites.FunHouse import FunHouse
def choose_site(website):
if website == "FunHouse":
return FunHouse()
else:
return Website()
And in my scrape_sites.py file is the following:
import Websites
# Some code that loads a text file and sets website_string to "FunHouse"
website = Websites.choose_site(website_string)
website.doStuff()
My Question
If I want to add a website, I have to edit __init__.py. Is there any way to make it so that I don't have to edit the __init__.py file whenever I add a new website? So if I create Google.py, I can just throw it into the Websites folder and it will be available to call?
You can add the __all__ variable to your __init__.py file once.
__all__ = ["Google", "Yahoo", "Bing"]
Now, each of these modules will be available to you in your code to use.
I figured it out. I don't even need the __init__.py files:
def choose_site(website_str):
mod = __import__('Websites.' + website_str, fromlist=[website_str])
obj = getattr(mod, website_str)
return obj()
This relies on the class name being the same as the Python filename, because if you pass in "Google" in the argument, it will import Websites.Google.Google
If you're looking at this and wanted to have more flexibility, this will work:
def choose_class(module_name, filename, class_name):
"""
:param module_name: The folder
:param filename: The .py file
:param class_name: The class in the .py file
:return: The selected class
"""
mod = __import__(module_name+ '.' + filename, fromlist=[filename])
obj = getattr(mod, class_name)
return obj()