I am not able to import certain files/classes
My directory structure is below:
Main Project Directory
├── config
│ ├── config.py
│ ├── constants.py
│ └── __init__.py
└── utils
├── __init__.py
├── logger_config.py
├── sql_helper.py
├── update_data_on_request.py
└── utility_functions.py
I want to import config.py from config into utility_functions.
Currently init.py is blank.
Error I face was ModuleNotFoundError: No module named 'config'
Try to use from config.config import * in utility_functions.py
say I have a folder structure that looks like this:
.
├── CODEOWNERS
├── Makefile
├── README.rst
├── __pycache__
│ └── conftest.cpython-37-pytest-6.2.2.pyc
├── airflow
│ ├── __pycache__
│ │ └── __init__.cpython-37.pyc
│ ├── dev
│ │ └── dags
│ └── <some_file_I_need>
how do I import the file I need from the airflow local package (not the third party dependency named airflow). I have a dependency called airflow unfortunately and that gets imported when I do: import airflow.dev... and that errors out.
I am trying to test Python modules. Currently, the file structure is like the following displayed:
project
├── __init__.py
├── __pycache__
│ └── __init__.cpython-37.pyc
├── p.py
├── package1
│ ├── __init__.py
│ ├── __pycache__
│ ├── module1.py
│ └── module2.py
└── package2
├── __init__.py
├── __pycache__
└── module3.py
In the module3.py, I would like to import the functions defined in module2.py. I believe I just need to use from package1.module2 import function_name. However, that doesn't work and the error is ModuleNotFoundError: No module named 'package1'.
BTW, I am using Python3.6 for the testing.
You need to add the directory path to your sys.path or PYTHONPATH, the following code would work.
In your module3.py, add this:
import os, sys
ab_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../package1'))
sys.path.append(ab_path)
Then you can do from module2 import function_name after it.
I have an api with a structure like this
my-api
├── Dockerfile
├── __init__.py
├── app
│ ├── __init__.py
│ ├── api
│ │ ├── __init__.py
│ │ └── mymodule.py
│ ├── config.py
│ ├── main.py
│ └── requirements.txt
└── docker-compose.yml
I'm trying to import mymodule.py into main.py per the docs but when I do so get the error
File "./app/main.py", line 3
from my-api.app.api import allocate
^
SyntaxError: invalid syntax
My import looks like this
from my-api.app.api import mymodule
and my Dockerfile CMD is
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "7833"]
Is there something I'm doing wrong here?
I have read through at least a dozen different stackoverflow questions that all present the same basic problem and have the same basic answer: either the module isn't installed correctly or the OP is doing the import wrong.
In this case, I am trying to do from google.cloud import secretmanager_v1beta1.
It works in my airflow container when I run airflow dags or if I run pytest tests/dags/test_my_dag.py. However, if I run cd dags; python -m my_dag or cd dags; python my_dag.py I get this error:
from google.cloud import secretmanager as secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)
I can add from google.cloud import bigquery in the line right above this line and that works OK. It appears to literally just be a problem with this particular package.
Why does it matter if pytest and airflow commands succeed? Because, I have another environment where I am trying to run dataflow jobs from the command-line and I get this same error. And unfortunately I don't think I can bypass this error in that environment for several reasons.
UPDATE 6
I have narrowed down the error to an issue with the google.cloud namespace and the secretmanager package within that namespace in the __init__.py file.
If I add from google.cloud import secretmanager to airflow/dags/__init__.py and then try to run python -m dags.my_dag.py, I receive this error but with a slightly different stacktrace:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/local/lib/python3.7/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "/workspace/airflow/dags/__init__.py", line 3, in <module>
from google.cloud import secretmanager
ImportError: cannot import name 'secretmanager' from 'google.cloud' (unknown location)
OLD INFORMATION
I am 95% sure that it's still a path problem and that pytest and airflow are fixing something I'm not aware of that isn't handled when I try to manually run the python script.
Things I have tried:
cd /airflow; python setup.py develop --user
cd /airflow; pip install -e . --user
cd /airflow/dags; pip install -r requirements.txt --user
UPDATE
As per requests in the comments, here are the contents of requirements.txt:
boto3>=1.7.84
google-auth==1.11.2
google-cloud-bigtable==1.2.1
google-cloud-bigquery==1.24.0
google-cloud-spanner==1.14.0
google-cloud-storage==1.26.0
google-cloud-logging==1.14.0
google-cloud-secret-manager>=0.2.0
pycloudsqlproxy>=0.0.15
pyconfighelper>=0.0.7
pymysql==0.9.3
setuptools==45.2.0
six==1.14.0
And I accidentally omitted the --user flags from the pip and python installation command examples above. In my container environment everything is installed into the user's home directory using --user and NOT in the global site-packages directory.
UPDATE 2
I've added the following code to the file that is generating the error:
print('***********************************************************************************')
import sys
print(sys.path)
from google.cloud import secretmanager_v1beta1 as secretmanager
print('secretmanager.__file__: {}'.format(secretmanager.__file__))
From airflow list_dags:
['/home/app/.local/bin', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/workspace/airflow', '/usr/local/lib/python3.7/site-packages', '/workspace/airflow/dags', '/workspace/airflow/config', '/workspace/airflow/plugins']
secretmanager.__file__: /home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg/google/cloud/secretmanager_v1beta1/__init__.py
From python my_dag.py:
['/workspace/airflow/dags', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/app/.local/lib/python3.7/site-packages', '/home/app/.local/lib/python3.7/site-packages/Jeeves-0.0.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/google_cloud_secret_manager-0.2.0-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/pyconfighelper-0.0.7-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/click-7.1.1-py3.7.egg', '/home/app/.local/lib/python3.7/site-packages/icentris_ml_airflow-0.0.0-py3.7.egg', '/usr/local/lib/python3.7/site-packages']
UPDATE 3
tree airflow/dags
airflow/dags
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-37.pyc
│ ├── bq_to_cs.cpython-37.pyc
│ ├── bq_to_wrench.cpython-37.pyc
│ ├── fetch_cloudsql_tables-bluesun.cpython-37.pyc
│ ├── fetch_cloudsql_tables.cpython-37.pyc
│ ├── fetch_app_tables-bluesun.cpython-37.pyc
│ ├── fetch_app_tables.cpython-37.pyc
│ ├── gcs_to_cloudsql.cpython-37.pyc
│ ├── gcs_to_s3.cpython-37.pyc
│ ├── lake_to_staging.cpython-37.pyc
│ ├── schedule_dfs_sql_to_bq-bluesun.cpython-37.pyc
│ ├── schedule_dfs_sql_to_bq.cpython-37.pyc
│ ├── app_to_bq_initial_load-bluesun.cpython-37.pyc
│ ├── app_to_lake-bluesun.cpython-37.pyc
│ └── app_to_lake.cpython-37.pyc
├── bq_to_wrench.py
├── composer_variables.json
├── my_ml_airflow.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ └── top_level.txt
├── lake_to_staging.py
├── libs
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-37.pyc
│ │ ├── checkpoint.cpython-37.pyc
│ │ └── utils.cpython-37.pyc
│ ├── checkpoint.py
│ ├── io
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ │ └── __init__.cpython-37.pyc
│ │ └── gcp
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ │ ├── __init__.cpython-37.pyc
│ │ │ └── storage.cpython-37.pyc
│ │ └── storage.py
│ ├── shared -> /workspace/shared/
│ └── utils.py
├── requirements.txt
├── table_lists
│ └── table-list.json
└── templates
└── sql
├── lake_to_staging.contacts.sql
├── lake_to_staging.orders.sql
└── lake_to_staging.users.sql
11 directories, 41 files
UPDATE 4
I tried fixing it so that sys.path looked the same when running python dags/my_dag.py as it does when running airflow list_dags or pytest test_my_dag.py.
Still get the same error.
Looking at a more recent version of documentation, I noticed that you should be able to just do from google.cloud import secretmanager. Which gave me the same result (works with airflow and pytest, not when trying to run directly).
At this point, my best guess is that it has something to do with namespace magic, but I'm not sure?
It have to be installed via terminal: pip install google-cloud-secret-manager
Because package name is not secretmanager but google-cloud-secret-manager
Similar to Noah's answer, this fixed the issue for me without having to import an unneeded module:
import google.cloud.secretmanager as secretmanager
After much trial and error, the issue is that currently one cannot import secretmanager from the google.cloud namespace if one has not previously imported another package from google.cloud.
Ex.
mod.py
from google.cloud import secretmanager # Fails with error
mod2.py
from google.cloud import bigquery
from google.cloud import secretmanager # Works because the first import initialized the namespace
I'm using Python 3.8.2 and google-cloud-secret-manager 2.7.1. The following line fixed the issue for me:
from google.cloud import secretmanager_v1beta1 as secretmanager