In our Airflow dags project, we have created a package called utils. In this package, we have created many Python files.
Recently I created a new file called github_util.py where I have written some code to interact with GitHub APIs.
There is another Python file called mail_forms.py in the utils folder.
I am importing github_util.py in mail_forms.py using
from utils.github_util import task_failed_github_issue
The content of github_util.py is
def task_failed_github_issue(context):
print("only function")
In utils package, we have an empty init.py file.
When I deploy this code in Airflow, I am getting below error:
ModuleNotFoundException: No module names 'utils'.
We have files called backend_util.py and entity_util in the same folder 'utils'. entity_util is also imported into backend file as below:
from utils.entity_util import read_definitions
This import is working correctly. I am not able to understand why this import works and mine does not.
I referred to many links on Stack Overflow and other websites, but none of the solution worked for me.
Related
Disclaimer: I am studying Python. I am given a task of reusing functions.
the function is simple:
app/utils/calculators.py
def calculate_session(session_type, session_code):
return 3 # just to save space
Now i need to use the function from a different file but for the life of me, I failed to import it. I have already added init.py to utils directory, to the app directory as well.
app/tasks/process_sessions.py
from utils.calculators import calculate_session
but when I run it, it fails saying module not found. I am in a virtual environment and all the files go in app directory.
What am I missing?
You'll want to specify the full path of the file (including the root directory) and import the specific function. Example below:
from app.utils.calculators import calculate_session
This should then import your function, regardless of whether you're using a virtual environment.
I have a folder structure for project with small application and some additional scripts, which are not directly used by application (they are used for some data science on data in data folder):
project
data
doc
src
└───scripts1
script1_1.py
script1_2.py
scripts2
script2_1.py
application
common
__init__.py
config.py
enrichment
__init__.py
module1.py
module2.py
__init__.py
app.py
Script app.py is always entry point and is used for starting the application. I want to be able to use relative imports in this folder structure. For example I want to import Configuration class, which is located in config.py inside of module1.py. But when I run app.py and have this import statement in module1.py:
from ..common.config import Configuration
I get the following error:
File ".../project/src/application/enrichment/module1.py", line 6, in <module>
from ..common.config import Configuration
ValueError: attempted relative import beyond top-level package
I would also need to import enrichment modules in app.py, guess it should work similarly:
from .enrichment.module1 import <func or class>
I have read multiple threads on module importing, but I am still not able to reconstruct the behavior, where I am able to use these relative imports and not get the ValueError. In one older project I used path importing in __init__.py files, but I hope it can be solved somehow better, because it was a bit magic for me. Really thanks for any help.
Before reading my question I would like to clarify that running the program within the python shell will run without any issues. I need to run some code through jupyter notebook and only then does importing within jupyter notebook will result in a No Module found error. This issue is described further below.
I have a project with the following hierarchy:
-ProjectFolder
-test.ipynb
-modules
-Game.py
-Grid.py
I was able to successfully import both Game.py and Grid.py into my jupyter notebook test file by adding the two lines to import each python file like below:
test.ipynb
from modules import Game
from modules import Grid
Unfortunately for this project there is a hard requirement. I must import the Grid Class within the Game.py file. Because both Game.py and Grid.py reside in the same folder I thought I could simply add the line import Grid at the top of Game.py without any issues. I did this by adding the line importGrid:
Game.py
import Grid
...
class Grid(object):
// grid structure logic
With that line, my test.ipynb now imports only the Game class from Game.py like so:
test.ipynb
from modules import Game
...
class Game(object):
// game logic
When running test in jupyter I unfortunately get the error No module named 'Grid'. Could anyone please tell me what I am doing wrong in this case? I have tried several things like relative imports trying from .Grid import Grid but this didnt work either and had a module not found error as well"
I am trying to write a python library, where some files depend on other files, for example:
I have folder structure:
../libname
../libname/core.py
../libname/supplementary1.py
../libname/supplementary2.py
../libname/__init__.py
where libname is where I import from.
the core.py file begins with:
import supplementary1
import supplementary2
...some code...
and this works fine, if I test it in the main of the core.py
Let's say I want to use libname as library in my project. My folder structure is then:
./libname
./main.py
where main.py calls functions from core.py, which in fact need functions from supplementary1 and supplementary2.
Currently, it throws me an error, saying there is no supplementary1, if I try (in main.py)
from core.py import function1
My question is, how do I import files from my library then? I mean one option would be to copy all the code from e.g. supplementary1 to the core.py, but I wish to maintain my code elegantly separated, if possible.
So in other words, how does one import a file, which already imports some files from a local library?
Thank you very much.
In import ... and from ... import ... you need to write not the filename, but module name. Instead of core.py you should say libname.core, meaning "module core, from package libname" (libname will be searched in all module paths, that normally includes the directory of the script you've started, i.e. where your main.py is).
tl;dr: a simple answer to your question is to write from libname.core import function1 instead.
Also, I'd suggest to use relative imports and instead of import supplementary1 write from . import supplementary1 - here, from . means "from the current package - where this file (module) resides in".
Consider reading Python documentation on modules - there are a lot of examples and explanations there.
I wrote a simple set of python3 files for emulating a small set of mongodb features on a 32 bit platform. I fired up PyCharm and put together a directory that looked like:
minu/
client.py
database.py
collection.py
test_client.py
test_database.py
test_client.py
My imports are simple. For example, client.py has the following at the top:
from collection import Collection
Basically, client has a Client class, collection has a Collection class, and database has a Database class. Not too tough.
As long as I cd into the minu directory, I can fire up a python3 interpreter and do things like:
>>> from client import Client
>>> c = Client(pathstring='something')
And everything just works. I can run the test_files as well, which use the same sorts of imports.
I'd like to modularize this, so I can use it another project by just dropping the minu directory alongside my application's .py files and just have everything work. When I do this though, and am running python3 from another directory, the local imports don't work. I placed an empty init.py in the minu directory. That made it so I could import minu. But the others broke. I tried using things like from .collection import Collection (added the dot), but then I can't run things in the original directory anymore, like I could before. What is the simple/right way to do this?
I have looked around a bit with Dr. Google, but none of the examples really clarify it well, feel free to point out the one I missed
In this file ...minu/__init__.py import the submodules you wish to expose externally.
If the __init__.py file contains the following lines, and the client.py file has a variable foo.
import client
import collection
import database
Then from above the minu directory, the following will work:
from minu.client import foo