I have a folder structure for project with small application and some additional scripts, which are not directly used by application (they are used for some data science on data in data folder):
project
data
doc
src
└───scripts1
script1_1.py
script1_2.py
scripts2
script2_1.py
application
common
__init__.py
config.py
enrichment
__init__.py
module1.py
module2.py
__init__.py
app.py
Script app.py is always entry point and is used for starting the application. I want to be able to use relative imports in this folder structure. For example I want to import Configuration class, which is located in config.py inside of module1.py. But when I run app.py and have this import statement in module1.py:
from ..common.config import Configuration
I get the following error:
File ".../project/src/application/enrichment/module1.py", line 6, in <module>
from ..common.config import Configuration
ValueError: attempted relative import beyond top-level package
I would also need to import enrichment modules in app.py, guess it should work similarly:
from .enrichment.module1 import <func or class>
I have read multiple threads on module importing, but I am still not able to reconstruct the behavior, where I am able to use these relative imports and not get the ValueError. In one older project I used path importing in __init__.py files, but I hope it can be solved somehow better, because it was a bit magic for me. Really thanks for any help.
Related
In our Airflow dags project, we have created a package called utils. In this package, we have created many Python files.
Recently I created a new file called github_util.py where I have written some code to interact with GitHub APIs.
There is another Python file called mail_forms.py in the utils folder.
I am importing github_util.py in mail_forms.py using
from utils.github_util import task_failed_github_issue
The content of github_util.py is
def task_failed_github_issue(context):
print("only function")
In utils package, we have an empty init.py file.
When I deploy this code in Airflow, I am getting below error:
ModuleNotFoundException: No module names 'utils'.
We have files called backend_util.py and entity_util in the same folder 'utils'. entity_util is also imported into backend file as below:
from utils.entity_util import read_definitions
This import is working correctly. I am not able to understand why this import works and mine does not.
I referred to many links on Stack Overflow and other websites, but none of the solution worked for me.
Here's my directory structure:
app
-Folder1
-class-container.py
-queries.py
-script.py
-Folder2
-Folder3
-main.py
In my class-container file I import the SQL queries from queries.py with from Folder1.queries import sql_queries
When I import and use the class in main.py everything runs smoothly. But when I do the same from script.py I get a ModuleNotFoundError... no module named Folder1. I've tried relative imports and changing it to from queries import sql_queries but every time I change it, it breaks one or the other. Any insight would be valuable. Thanks!
I want my project to work in two different situations. It should work as a standalone library, but also as sub package of a larger project. The main use case is that of a standalone library, where its internal imports should be of the form
from my_library import sub_package
When using the code as sub package of a larger project, these imports don't work as there is no global name my_library. Instead, I would have to use relative or absolute imports, for example
from large_project.my_library import sub_package
Let's assume I wrote my library as shown in the first example. How can I overwrite importing behavior when running as part of a larger project to automatically adjust import paths?
Thanks to #MatrixTai's suggestion of adding the parent directory of the package to the the module path, I came up with this dynamic solution. At the top of my_library/__init__.py:
# Make package global even if used as a sub package to enable short imports.
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
We have to navigate up two directories here to do from the my_library/__init__.py to my_library and from there to its parent direction, where imports will find the library.
You don't have much choice.
If you want to reference the my_library.py anywhere, there is 2 method (as I known) can do similar work.
1: add system path. Like those module you installed by pip. pip module is installed in /Python/Scripts. You can add a new path or simply put my_library.py into one of the path. For adding, that's in Computer(right-click)-> Properties -> Environment Variable -> Choose Path and Click Edit
(Though you may not want to use this.)
2: Changing __init__.py, but still at least one line you must add in my_library.py.
For example,
/Directory
/large_project
-__init__.py #call this sub_init
-my_library.py
-__init__.py #call this main_init, this fake
-main.py
In main_init,
import sys
sys.path.append('\\Directory\\large_project')
As main_init is not executed when you execute main.py (this is fake), so in main.py
import __init__
from my_library import sub_package
But as well you can take this main_init as the starter of library, like declaring __all__, etc.
I am trying to write a python library, where some files depend on other files, for example:
I have folder structure:
../libname
../libname/core.py
../libname/supplementary1.py
../libname/supplementary2.py
../libname/__init__.py
where libname is where I import from.
the core.py file begins with:
import supplementary1
import supplementary2
...some code...
and this works fine, if I test it in the main of the core.py
Let's say I want to use libname as library in my project. My folder structure is then:
./libname
./main.py
where main.py calls functions from core.py, which in fact need functions from supplementary1 and supplementary2.
Currently, it throws me an error, saying there is no supplementary1, if I try (in main.py)
from core.py import function1
My question is, how do I import files from my library then? I mean one option would be to copy all the code from e.g. supplementary1 to the core.py, but I wish to maintain my code elegantly separated, if possible.
So in other words, how does one import a file, which already imports some files from a local library?
Thank you very much.
In import ... and from ... import ... you need to write not the filename, but module name. Instead of core.py you should say libname.core, meaning "module core, from package libname" (libname will be searched in all module paths, that normally includes the directory of the script you've started, i.e. where your main.py is).
tl;dr: a simple answer to your question is to write from libname.core import function1 instead.
Also, I'd suggest to use relative imports and instead of import supplementary1 write from . import supplementary1 - here, from . means "from the current package - where this file (module) resides in".
Consider reading Python documentation on modules - there are a lot of examples and explanations there.
In pycharm I have a directory named my_hw3 with a few python files beneath it named Question_1,Question_2 etc.
To check if my code is right my prof gave a few test files that require me to import a few things.
The part of the test code that imports reads as.
import numpy as np
import numpy.random as npr
from my_hw3 import *
The part I struggle to understand is the from my_hw3 import *.
What do I need to change in pycharm in order for my python files to recognize the reference as I seem to always be given the error "unresolved reference".
!http://imgur.com/H7iEV8J
The directory my_hw3 will be considered an importable package only if there is an __init__.py in it.
If the Question_n files are Question_n.py files, then you may want to:
from my_hw3.Question_1 import *
You will probably get better answers if you provide an outline of the complete directory structure for your project.