My file structure look like this
Module
|
|--Common
| |
| utils.py
| credentials.ini
|-- Folder2
|--Folder3
|--Folder4
|
Folder5
|
comp.py
I need to import utils.py functions in the comp.py file, but the problem is that utils itself needs the credentials.ini file for it to work.
I solved the problem in utils.py by giving it a absolute path like this path=join(dirname(os.path.realpath(__file__)), 'credentials.ini')
and in comp.py file I added this path to env using
import sys
sys.path.append("../../")
While this worked when I ran comp.py but I need to schedule it on airflow for it to run. Whenever airflow schedules comp.py to run it can't find the utils.py (Airflow and the module package are in different paths). Any idea how I can resolve it? I don't want to manually add the utils.py path to the env.
P.S The whole directly is initialized as a package. I have added __init__.py to the main module folder as well as all the subdirectories in it.
EDIT: Fixed Formatting
Airflow loads DAGs in a sandboxed environment and it does not handle all the various ways importing works when you run Python file as script. This is due to security and the way how different components of the distributed system work.
See https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html but more detailed information especially the "development" versoin of the documentation that will be released in 2.2 (especially the "best practices"):
http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/modules_management.html#best-practices-for-module-loading
There are some best practices to follow:
Place all your python files in one of the modules that are already on pythonpath
Always use absolute imports, do not use "relative" references
Don't rely on your current working directory setting (likely this is what your problem was really - your current working directory was different than you expected).
In your case what will likely work is:
write a method in your 'utils.py" - for example "get_credentials_folder()".
in this method use __file__ to derive the path of the "utils.py" and find the absolute path of the expected folder containing it (use pardir and abspath)
add that absolute path you get to the sys.path
I've been working a on a CRUD SAM application using a Python 3.8 runtime. My lambda functions reference a lambda layer that has code shared amongst the functions. My application builds/deploys and I can invoke the functions locally, however, in composing unit tests (using pytest), I'm not sure how to get around my imports referencing a layer in line that doesn't match the file structure.
File structure:
.
├── template.yaml
├── _layer-folder
│ └── _python
│ └── shared_code.py
├── _lambda
│ ├── some_function.py
│ └── _tests
│ └── test-some-function.py
When running my tests for my lambda functions, I get an import error when I reference a module in that lives in the shared layer. For example:
from some_module_in_a_layer import some_layer_function
Is there a way to configure pytest to reference the correct file directory when running the tests?
I ended up resolving this by appending to the system path when testing or running locally within my __init__.py file.
if os.environ.get("ENVIRONMENT") == "test":
sys.path.append(os.getcwd() + '/path/to/layer')
It's a pain indeed to test layers properly and it kind of depends on your method of running the test (e.g. in Pycharm or using the terminal). In PyCharm you can add the layers directory as a source (right click use as source). To run from terminal you can add it to your PYTHONPATH before running, but it's quite ugly.
So PYTHONPATH='/path/to/layer' python main.py
It's not pretty but I don't know another way to fix that tbh.
I added this line in my test and it worked perfectly.
test_file.py
import pytest
sys.path.append(os.getcwd() + '/../layer_location')
Inside layer location my file system looks like this:
layer-location\
__init__.py
layer_code.py
The import in my non-test code is then:
code_under_test.py
from layer_code import foo
foo()
I have a python 3.6 code-base which needs to be installed in the environment's site-packages directory in order to be called and used. After moving it to docker, I decided that I should set up a shared volume between the docker container and the host machine in order to avoid copying and installing the code on the container and having to rebuild every time I made a change to the code and wanted to run it. In order to achieve this, I had to change a lot of the import statements from relative to absolute. Here is the structure of the application:
-root
-src
-app
-test
In order to run the application from the root directory without installing it, I had to change a lot of the import statements from
from app import something
to:
import src.app.something
The problem is that I use poetry to build the app on an azure build agent, and tox to run the tests. The relevant part of my pyproject.toml file looks like this:
[tool.poetry]
name = "app"
version = "0.1.0"
packages = [{include = 'app', from='src'}]
The relevant part of my tox.ini file looks like this:
[tox]
envlist = py36, bandit, black, flake8, safety
isolated_build = True
[testenv:py36]
deps =
pytest
pytest-cov
pytest-env
pytest-mock
fakeredis
commands =
pytest {posargs} -m "not external_service_required" --junitxml=junit_coverage.xml --cov report=html --cov-report=xml:coverage.xml
I'm not an expert in tox or poetry, but from what I could tell, the problem was that the src directory wasn't being included in the build artifact, only the inner app directory was, so I added a parent directory and changed the directory structure to this:
-root
-app
-src
-app
-test
And then changed the poetry configuration to the following in order to include the src directory
[tool.poetry]
name = "app"
version = "0.1.0"
packages = [{include = 'src', from='app'}]
Now when I change the imports in the tests from this:
from app import something
to this:
from app.src.app import something
The import is recognized in Pycharm, but when I try to run tox -r, the I get the following error:
E ModuleNotFoundError: No module named 'app'
I don't understand how tox installs the application, and what kind of package structure I need to specify in order to be able to call the code both from the code-base directory and from site packages. I looked at some example projects, and noticed that they don't use the isolated_build flag, but rather the skip_dist flag, but somehow they also install the application in site packages before running their tests.
Any help would be much appreciated.
Specs:
poetry version: 1.1.6
python version:3.6.9
tox version:3.7
environment: azure windows build agent
You have to change the imports back to from app import something, the src part is, with respect to the code as a deliverable, completely transient. Same goes for adding in another app directory, your initial project structure was fine.
You were right about going from relative imports to absolute ones though, so all that is necessary thereafter is telling your python runtime within the container that root/src should be part of the PYTHONPATH:
export PYTHONPATH="{PYTHONPATH}:/path/to/app/src"
Alternatively, you can also update the path within your python code right before importing your package:
import sys
sys.path.append("/path/to/root/src")
import app # can be found now
Just to state the obvious, meddling with the interpreter in this way is a bit hacky, but as far as I'm aware it should work without any issues.
I used easy_install to install pytest on a Mac and started writing tests for a project with a file structure likes so:
repo/
|--app.py
|--settings.py
|--models.py
|--tests/
|--test_app.py
Run py.test while in the repo directory, and everything behaves as you would expect.
But when I try that same thing on either Linux or Windows (both have pytest 2.2.3 on them), it barks whenever it hits its first import of something from my application path. For instance, from app import some_def_in_app.
Do I need to be editing my PATH to run py.test on these systems?
I'm not sure why py.test does not add the current directory in the PYTHONPATH itself, but here's a workaround (to be executed from the root of your repository):
python -m pytest tests/
It works because Python adds the current directory in the PYTHONPATH for you.
Recommended approach for pytest>=7: use the pythonpath setting
Recently, pytest has added a new core plugin that supports sys.path modifications via the pythonpath configuration value. The solution is thus much simpler now and doesn't require any workarounds anymore:
pyproject.toml example:
[tool.pytest.ini_options]
pythonpath = [
"."
]
pytest.ini example:
[pytest]
pythonpath = .
The path entries are calculated relative to the rootdir, thus . adds repo directory to sys.path in this case.
Multiple path entries are also allowed: for a layout
repo/
├── src/
| └── lib.py
├── app.py
└── tests
├── test_app.py
└── test_lib.py
the configuration
[tool.pytest.ini_options]
pythonpath = [
".", "src",
]
or
[pytest]
pythonpath = . src
will add both app and lib modules to sys.path, so
import app
import lib
will both work.
Original answer (not recommended for recent pytest versions; use for pytest<7 only): conftest solution
The least invasive solution is adding an empty file named conftest.py in the repo/ directory:
$ touch repo/conftest.py
That's it. No need to write custom code for mangling the sys.path or remember to drag PYTHONPATH along, or placing __init__.py into dirs where it doesn't belong (using python -m pytest as suggested in Apteryx's answer is a good solution though!).
The project directory afterwards:
repo
├── conftest.py
├── app.py
├── settings.py
├── models.py
└── tests
└── test_app.py
Explanation
pytest looks for the conftest modules on test collection to gather custom hooks and fixtures, and in order to import the custom objects from them, pytest adds the parent directory of the conftest.py to the sys.path (in this case the repo directory).
Other project structures
If you have other project structure, place the conftest.py in the package root dir (the one that contains packages but is not a package itself, so does not contain an __init__.py), for example:
repo
├── conftest.py
├── spam
│ ├── __init__.py
│ ├── bacon.py
│ └── egg.py
├── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
src layout
Although this approach can be used with the src layout (place conftest.py in the src dir):
repo
├── src
│ ├── conftest.py
│ ├── spam
│ │ ├── __init__.py
│ │ ├── bacon.py
│ │ └── egg.py
│ └── eggs
│ ├── __init__.py
│ └── sausage.py
└── tests
├── test_bacon.py
└── test_egg.py
beware that adding src to PYTHONPATH mitigates the meaning and benefits of the src layout! You will end up with testing the code from repository and not the installed package. If you need to do it, maybe you don't need the src dir at all.
Where to go from here
Of course, conftest modules are not just some files to help the source code discovery; it's where all the project-specific enhancements of the pytest framework and the customization of your test suite happen. pytest has a lot of information on conftest modules scattered throughout their docs; start with conftest.py: local per-directory plugins
Also, SO has an excellent question on conftest modules: In py.test, what is the use of conftest.py files?
I had the same problem. I fixed it by adding an empty __init__.py file to my tests directory.
Yes, the source folder is not in Python's path if you cd to the tests directory.
You have two choices:
Add the path manually to the test files. Something like this:
import sys, os
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + '/../')
Run the tests with the env var PYTHONPATH=../.
Run pytest itself as a module with:
python -m pytest tests
This happens when the project hierarchy is, for example, package/src package/tests and in tests you import from src. Executing as a module will consider imports as absolute rather than relative to the execution location.
You can run with PYTHONPATH in project root
PYTHONPATH=. py.test
Or use pip install as editable import
pip install -e . # install package using setup.py in editable mode
I had the same problem in Flask.
When I added:
__init__.py
to the tests folder, the problem disappeared :)
Probably the application couldn't recognize folder tests as a module.
I created this as an answer to your question and my own confusion. I hope it helps. Pay attention to PYTHONPATH in both the py.test command line and in the tox.ini.
https://github.com/jeffmacdonald/pytest_test
Specifically: You have to tell py.test and tox where to find the modules you are including.
With py.test you can do this:
PYTHONPATH=. py.test
And with tox, add this to your tox.ini:
[testenv]
deps= -r{toxinidir}/requirements.txt
commands=py.test
setenv =
PYTHONPATH = {toxinidir}
I fixed it by removing the top-level __init__.py in the parent folder of my sources.
I started getting weird ConftestImportFailure: ImportError('No module named ... errors when I had accidentally added __init__.py file to my src directory (which was not supposed to be a Python package, just a container of all source).
It is a bit of a shame that this is an issue in Python... But just adding this environment variable is the most comfortable way, IMO:
export PYTHONPATH=$PYTHONPATH:.
You can put this line in you .zshrc or .bashrc file.
I was having the same problem when following the Flask tutorial and I found the answer on the official Pytest documentation.
It's a little shift from the way I (and I think many others) are used to do things.
You have to create a setup.py file in your project's root directory with at least the following two lines:
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
where PACKAGENAME is your app's name. Then you have to install it with pip:
pip install -e .
The -e flag tells pip to install the package in editable or "develop" mode. So the next time you run pytest it should find your app in the standard PYTHONPATH.
I had a similar issue. pytest did not recognize a module installed in the environment I was working in.
I resolved it by also installing pytest into the same environment.
Also if you run pytest within your virtual environment make sure pytest module is installed within your virtual environment. Activate your virtual environment and run pip install pytest.
For me the problem was tests.py generated by Django along with tests directory. Removing tests.py solved the problem.
I got this error as I used relative imports incorrectly. In the OP example, test_app.py should import functions using e.g.
from repo.app import *
However liberally __init__.py files are scattered around the file structure, this does not work and creates the kind of ImportError seen unless the files and test files are in the same directory.
from app import *
Here's an example of what I had to do with one of my projects:
Here’s my project structure:
microbit/
microbit/activity_indicator/activity_indicator.py
microbit/tests/test_activity_indicator.py
To be able to access activity_indicator.py from test_activity_indicator.py I needed to:
start test_activity_indicatory.py with the correct relative import:
from microbit.activity_indicator.activity_indicator import *
put __init__.py files throughout the project structure:
microbit/
microbit/__init__.py
microbit/activity_indicator/__init__.py
microbit/activity_indicator/activity_indicator.py
microbit/tests/__init__.py
microbit/tests/test_activity_indicator.py
According to a post on Medium by Dirk Avery (and supported by my personal experience) if you're using a virtual environment for your project then you can't use a system-wide install of pytest; you have to install it in the virtual environment and use that install.
In particular, if you have it installed in both places then simply running the pytest command won't work because it will be using the system install. As the other answers have described, one simple solution is to run python -m pytest instead of pytest; this works because it uses the environment's version of pytest. Alternatively, you can just uninstall the system's version of pytest; after reactivating the virtual environment the pytest command should work.
I was getting this error due to something even simpler (you could even say trivial). I hadn't installed the pytest module. So a simple apt install python-pytest fixed it for me.
'pytest' would have been listed in setup.py as a test dependency. Make sure you install the test requirements as well.
Since no one has suggested it, you could also pass the path to the tests in your pytest.ini file:
[pytest]
...
testpaths = repo/tests
See documentation: https://docs.pytest.org/en/6.2.x/customize.html#pytest-ini
Side effect for Visual Studio Code: it should pick up the unit test in the UI.
We have fixed the issue by adding the following environment variable.
PYTHONPATH=${PYTHONPATH}:${PWD}/src:${PWD}/test
As pointed out by Luiz Lezcano Arialdi, the correct solution is to install your package as an editable package.
Since I am using Pipenv, I thought about adding to his answer a step-by-step how to install the current path as an edible with Pipenv, allowing to run pytest without the need of any mangling code or lose files.
You will need to have the following minimal folder structure (documentation):
package/
package/
__init__.py
module.py
tests/
module_test.py
setup.py
setup.py mostly has the following minium code (documentation):
import setuptools
setuptools.setup(name='package', # Change to your package name
packages=setuptools.find_packages())
Then you just need to run pipenv install --dev -e . and Pipenv will install the current path as an editable package (the --dev flag is optional) (documentation).
Now you should be able to run pytest without problems.
If this pytest error appears not for your own package, but for a Git-installed package in your package's requirements.txt, the solution is to switch to editable installation mode.
For example, suppose your package's requirements.txt had the following line:
git+https://github.com/foo/bar.git
You would instead replace it with the following:
-e git+https://github.com/foo/bar.git#egg=bar
If nothing works, make sure your test_module.py is listed under the correct src directory.
Sometimes it will give ModuleNotFoundError not because modules are misplaced or export PYTHONPATH="${PWD}:${PYTHONPATH}" is not working, its because test_module.py is placed into a wrong directory under the tests folder.
it should be 1-to-1 mapping relation recursively instead of the root folder should be named as "tests" and the name of the file that include test code should starts with "test_",
for example,
./nlu_service/models/transformers.py
./tests/models/test_transformers.py
This was my experience.
Very often the tests were interrupted due to module being unable to be imported.
After research, I found out that the system is looking at the file in the wrong place and we can easily overcome the problem by copying the file, containing the module, in the same folder as stated, in order to be properly imported.
Another solution proposal would be to change the declaration for the import and show MutPy the correct path of the unit. However, due to the fact that multiple units can have this dependency, meaning we need to commit changes also in their declarations, we prefer to simply move the unit to the folder.
My solution:
Create the conftest.py file in the test directory containing:
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + "/relative/path/to/code/")
This will add the folder of interest to the Python interpreter path without modifying every test file, setting environment variable or messing with absolute/relative paths.
I realise this is a real beginner question but for some reason, I haven't been able to find the right search terms for an answer.
I am setting up a nodeJS site, with Gulp running my builds. Part of this is Typescript & SCSS compilation, with the outputs being inputted to dist/js. So my files look something like this:
.
├── dev
| ├── app.ts
| └── utils
| └──file1.ts
| └──someFunction
| └──file2.ts
└─── dist
└── js
├── file1.js
└── file2.js
So a reference from file1.ts to ./someFunction/file2 should be ./file2 after compilation (i.e. referencing from the dist file. However, I am getting errors because gulp and typescript aren't changing the references (I didn't expect them too as I haven't made any attempt to tell them too!). How is this typically handled?
I normally try to keep the same paths between development and production to avoid the problem you are having. Another option is to combine the files into one destination file and minimize (uglify) it for inclusion. If you do not want to do that, there are gulp plugins to replace text (gulp-replace, gulp-html-replace, etc...) so you could remove specific paths.
You can have the Typescript compiler place your files in the dist/js directory without a path using --outdir (Typescript compiler options)