I have the following directory structure:
/home/pi
- project/
- p1v1.py
- tools1/
- __init__.py
- tools1a/
- __init__.py
- file1.py
- file2.py
- tools1a1/
- __init__.py
- file3.py
- file4.py
- tools1a2/
- __init__.py
- file5.py
- file6.py
I am trying to import all the modules from the file1.py into my project file p1v1.py
from file1 import *
but end up with either an
ImportError: attempted relative import with no known parent package
or an
ValueError: Attempted relative import in non-package
depending on what I use in p1v1.py because the functions in file1.py depend on file3.py and file4.py. I would like to use explicit imports (for clarity), but I'm not sure how to do this. Any advice would be appreciated.
Thank you!
Through trial and error eventually figured out how to solve this:
import sys
sys.path.insert(0,'..')
from tools1.tools1a.file1 import function as f1
Note: For this to work, I needed to be editing and executing my script p1v1.py out of the working directory /home/pi/project/. Hope this helps others with a similar problem!
Related
Similar to ImportError: Cannot import name X, I have a main.py file and, in the same folder, a few other python files bill.py and flatmate.py.
bill.py and flatmate.py each contain classes, nothing more.
Directory structure:
parentfolder
- main.py
- bill.py
- flatmate.py
- files
- pic.png
- doc.pdf
Sample code:
main.py:
import bill # won't work
from flatmate import Flatmate # won't work
...
bill.py:
class Bill:
def __init__(self, ...):
...
flatmate.py:
class Flatmate:
def __init__(self, ...):
...
I get an ImportError: cannot import name 'Bill' from 'bill' (c:\Users\lavid\Desktop\Python Experimente\Python OOP Kurs\App-2-Flatmates-Bill\bill.py). I am pretty sure I don't have circular references. What could be the cause of this problem?
Make sure that all files are next each others
try:
from . import module
if this doesn't work , add your files dir to the Question
Saving the referenced files before accessing them helped a lot.
I have a project with two applications. They both use a mongo-engine database model file. Also they have to start in different Docker containers, but use the same Mongo database in the fird container. Now my app structure looks like this:
app_root/
app1/
database/
models.py
main.py
app2/
database/
models.py
main.py
And it works fine, BUT I have to support two same files database/models.py. I dont want to do this and I make the next structure:
app_root/
shared/
database/
models.py
app1/
main.py
app2/
main.py
Unfortunately it doesnt work for me, because when I try this in my main.py:
from ..shared.database.models import *
I get
Exception has occurred: ImportError
attempted relative import with no known parent package
And when I try
from app_root.shared.database.models import *
I get
Exception has occurred: ModuleNotFoundError No module named 'app_root'
Please, what do I do wrong?
In the file you perform the import, try adding this:
import os
import sys
sys.path.append(os.path.abspath('../../..'))
from app_root.shared.database.models import *
I am creating an ETL pipeline using Apache Airflow and I am trying to create generalized custom operators. There seems to be no problem with the operators but they are not being imported into my DAG python file.
This is my directory structure.
my_project\
.env
Pipfile
Pipfile.lock
.gitignore
.venv\
airflow\
dags\
logs\
plugins\
__init__.py
helpers\
operators\
__init__.py
data_quality.py
load_fact.py
load_dimension.py
stage_redshift
This is what is present in the __init__.py file under plugins folder.
from __future__ import division, absolute_import, print_function
from airflow.plugins_manager import AirflowPlugin
import airflow.plugins.operators as operators
import airflow.plugins.helpers as helpers
# Defining the plugin class
class SparkifyPlugin(AirflowPlugin):
name = "sparkify_plugin"
operators = [
operators.StageToRedshiftOperator,
operators.LoadFactOperator,
operators.LoadDimensionOperator,
operators.DataQualityOperator
]
helpers = [
helpers.SqlQueries
]
I'm importing these operators into my DAG file as following
from airflow.operators.sparkify_plugin import (StageToRedshiftOperator,
LoadFactOperator,
LoadDimensionOperator,
DataQualityOperator)
I am getting an error as follows
ERROR - Failed to import plugin /Users/user_name/Documents/My_Mac/Projects/sparkify_etl_sql_to_sql/airflow/plugins/operators/stage_redshift.py
Can you help me understand why this is happening?
I figured out how to register my custom operators with Airflow without dedicating a python file to use AirflowPlugin class.
I achieved this by declaring them in my __init__.py file under plugins directory.
This is how I did it.
My project folder structure is as follows
my_project\
.env
Pipfile
Pipfile.lock
.gitignore
.venv\
airflow\
dags\
logs\
plugins\
__init__.py
helpers\
operators\
__init__.py
data_quality.py
load_fact.py
load_dimension.py
stage_redshift
My code in plugins/__init__.py
from airflow.plugins_manager import AirflowPlugin
import operators
import helpers
# Defining the plugin class
class SparkifyPlugin(AirflowPlugin):
name = "sparkify_plugin"
operators = [
operators.StageToRedshiftOperator,
operators.LoadFactOperator,
operators.LoadDimensionOperator,
operators.DataQualityOperator
]
helpers = [
helpers.SqlQueries
]
My code in plugins/operators/__init__.py
from operators.stage_redshift import StageToRedshiftOperator
from operators.load_fact import LoadFactOperator
from operators.load_dimension import LoadDimensionOperator
from operators.data_quality import DataQualityOperator
__all__ = [
'StageToRedshiftOperator',
'LoadFactOperator',
'LoadDimensionOperator',
'DataQualityOperator'
]
I am importing these custom operators in my dag file(dags/etl.py) as:
from airflow.operators.spark_plugin import LoadDimensionOperator
spark_plugin is what the name attribute in SparkifyPlugin class (stored in plugins/__init__.py) holds.
Airflow automatically registers these custom operators.
Hope it helps someone else in the future.
In case you are having some import errors, try running python __init__.py for each module as described by #absolutelydevastated. Make sure that the one in plugins directory runs without throwing errors.
I used Pycharm and it did throw me a few errors when running __init__.py files in the plugins/operators directory.
Fixing the one in plugins directory and ignoring the errors thrown by plugins/operators/__init__.py fixed my issue.
If you check out: Writing and importing custom plugins in Airflow
The person there was having a similar problem with their plugin, which they fixed by including a file under airflow/plugins named for their plugin, rather than defining it in the __init__.py file.
Several questions have been posted on the topic of relative paths in python, and it looks like this continues to be a fairly confusing topic, and no final word mentioned on it. I had found the most illustrative example in https://github.com/wweiradio/pkg-relative-import
However, when I made a few changes to those files, I ended up again with some error messages. After some amount of debugging, I came up with the following files and directory structure which works. (In the hierarchy mentioned below, I have mentioned the imports in each file, and which imports work and which dont.)
NOTE: (1) the code works, whether or not __init__.py files exist. So I have deleted all those files. (2) the files mentioned contain only the mentioned import lines, and maybe some print statements, and nothing else.
I run the code as:
cd top_level; python3 start1.py --> this works
cd top_level/src; python3 start2.py --> this does not work, as expected
I learnt the following lessons from this exercise:
(1) no relative paths on the invocation directory is allowed, or anywhere outside it.
(2) any further imports should be explicitly relative (eg .sub3, and not sub3),
or absolute with reference to the top level (eg src.sub2.sub3)
(3) relative paths have nothing to do with packages, nor the presence or absence of __init__.py.
eg, in the below code I have deleted all init.py, and it still works
(4) current dir is ., its parent is .., its grandparent is ..., etc
Can you confirm if these lessons are correct, and are there some other factors that I need to remember while using relative paths?
directory structure:
- top_level/
- top.py
- start1.py --> import src.sub.relative
- src/
- parent.py
- start2.py --> import sub.relative
- sub/
- relative.py --> from .. import parent
from ..sub2 import relative2
# these imports give error if start2.py is called
# this is understood, so we call only start1.py
- sub2/
- relative2.py --> import top
from .. import parent
#from sub3 import relative3 # --> doesnt work
from .sub3 import relative3 # --> works
from src.sub2.sub3 import relative3 # --> works
#import sub3.relative3 # --> doesnt work
import src.sub2.sub3.relative3 # --> works
- sub3/
- relative3.py --> import top
from ... import parent
I have a file structure as shown below,
MainFolder
__init__.py
FirstFolder
__init__.py
firstFile.py
SecondFolder
__init__.py
secondFile.py
Inside firstFile.py, I have a class named Math and I want to import this class in secondFile.py.
Code for firstFile.py
class Math(object):
def __init__(self, first_value, second_value):
self.first_value = first_value
self.second_value = second_value
def addition(self):
self.total_add_value = self.first_value + self.second_value
print(self.total_add_value)
def subtraction(self):
self.total_sub_value = self.first_value - self.second_value
print(self.total_sub_value)
Code for secondFile.py
from FirstFolder.firstFile import Math
Math(10, 2).addition()
Math(10, 2).subtraction()
When I tried running secondFile.py I get this error: ModuleNotFoundError: No module named 'First'
I am using Windows and the MainFolder is located in my C drive, under C:\Users\Name\Documents\Python\MainFolder
Possible solutions that I have tried are, creating the empty __init__.py for all main and sub folders, adding the dir of MainFolder into path under System Properties environment variable and using import sys & sys.path.append('\Users\Name\Documents\Python\MainFolder').
Unfortunately, all these solutions that I have found are not working. If anyone can highlight my mistakes to me or suggest other solutions, that would be great. Any help will be greatly appreciated!
There are potentially two issues. The first is with your import statement. The import statement should be
from FirstFolder.firstFile import Math
The second is likely that your PYTHONPATH environment variable doesn't include your MainFolder.
On linux and unix based systems you can do this temporarily on the commandline with
export PYTHONPATH=$PYTHONPATH:/path/to/MainFolder
On windows
set PYTHONPATH="%path%;C:\path\to\MainFolder"
If you want to set it permanently, use setx instead of set