Related
I'm a total newbie and I'm trying to do this project this is my first time, and it's almost done. I tried every method mentioned in this SO thread to move secret key from settings. In every method i got some kind of error, even from this official django doc mathod. I couldn't find where I'm making mistake.
When the secret key is inside the settings.py, everything is working super smooth. But I need to push my code in git, so i have to hide it from settings.py.
Right now im adding the details when i tried using django-environ, to keep secret key outside of settings.py.
im putting the contents inside the root project folder.
im using miniconda: 4.10.1. here is my requirement.txt.
# platform: linux-64
_libgcc_mutex=0.1=main
_openmp_mutex=4.5=1_gnu
appdirs=1.4.4=py_0
asgiref=3.3.4=pyhd3eb1b0_0
attrs=21.2.0=pyhd3eb1b0_0
black=19.10b0=py_0
ca-certificates=2021.5.30=ha878542_0
certifi=2021.5.30=py39hf3d152e_0
click=8.0.1=pyhd3eb1b0_0
django=3.2.4=pyhd3eb1b0_0
django-environ=0.4.5=py_1
importlib-metadata=3.10.0=py39h06a4308_0
krb5=1.17.1=h173b8e3_0
ld_impl_linux-64=2.35.1=h7274673_9
libedit=3.1.20210216=h27cfd23_1
libffi=3.3=he6710b0_2
libgcc-ng=9.3.0=h5101ec6_17
libgomp=9.3.0=h5101ec6_17
libpq=12.2=h20c2e04_0
libstdcxx-ng=9.3.0=hd4cf53a_17
mypy_extensions=0.4.1=py39h06a4308_0
ncurses=6.2=he6710b0_1
openssl=1.1.1k=h7f98852_0
pathspec=0.7.0=py_0
pip=21.1.2=py39h06a4308_0
psycopg2=2.8.6=py39h3c74f83_1
python=3.9.5=h12debd9_4
python_abi=3.9=1_cp39
pytz=2021.1=pyhd3eb1b0_0
readline=8.1=h27cfd23_0
regex=2021.4.4=py39h27cfd23_0
setuptools=52.0.0=py39h06a4308_0
six=1.16.0=pyh6c4a22f_0
sqlite=3.35.4=hdfb4753_0
sqlparse=0.4.1=py_0
tk=8.6.10=hbc83047_0
toml=0.10.2=pyhd3eb1b0_0
typed-ast=1.4.2=py39h27cfd23_1
typing_extensions=3.7.4.3=pyha847dfd_0
tzdata=2020f=h52ac0ba_0
wheel=0.36.2=pyhd3eb1b0_0
xz=5.2.5=h7b6447c_0
zipp=3.4.1=pyhd3eb1b0_0
zlib=1.2.11=h7b6447c_3
settings.py
import os
import environ
from pathlib import Path
env = environ.Env(
# set casting, default value
DEBUG=(bool, False)
)
# reading .env file
environ.Env.read_env()
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = env('SECRET_KEY')
# False if not in os.environ
DEBUG = env('DEBUG')
im not adding the rest of settings. i dont think its important. if need please mention I.ll update.
i placed .env file in root of the project where manage.py and db.sqlite3 are placed
.env
#env file
DEBUG=on
#copied the entire line from settings.py
SECRET_KEY ='xxxx django secret key here xxxx'
while running "python manage.py runserver", i got this error.
im not sure what im missing. i got some kind of error, when i tried each method and errors are not same. sorry that i cannot explain every method and error here.
there are several questions asked in this form. but most are not answered and some are not accurately explains my situation. please mention if anything else is needed or for more clarification.
First check that you have installed django-environ and maybe you have a typing mistake in your requirements.txt it should be django-environ=0.4.5 instead of django-environ=0.4.5=py_1
you can pass the path of your .env inside read_env(env_file="relative_path_of_your_env_file")
it read a .env file into os.environ.
If not given a path to a dotenv path, does filthy magic stack backtracking
to find manage.py and then find the dotenv.
go through this code https://github.com/joke2k/django-environ/blob/master/environ/environ.py#L614
From the structure of the file tree, its clear that .env file is placed in the root folder of the project. When checking the error message, its visible that whoever is searching for .env file is checking at the same place as settings.py.
So, the short answer is if you are using django-environ to keep secret-key outside, place .env file together with settings.py in the same directory.
For a bit more elaborated content, you can refer to this link. I felt it is suitable for newbies.
I am writing a file using Python, and I want it to be placed in a specific path. How can I safely make sure that the path exists?
That is: how can I check whether the folder exists, along with its parents? If there are missing folders along the path, how can I create them?
On Python ≥ 3.5, use pathlib.Path.mkdir:
from pathlib import Path
Path("/my/directory").mkdir(parents=True, exist_ok=True)
For older versions of Python, I see two answers with good qualities, each with a small flaw, so I will give my take on it:
Try os.path.exists, and consider os.makedirs for the creation.
import os
if not os.path.exists(directory):
os.makedirs(directory)
As noted in comments and elsewhere, there's a race condition – if the directory is created between the os.path.exists and the os.makedirs calls, the os.makedirs will fail with an OSError. Unfortunately, blanket-catching OSError and continuing is not foolproof, as it will ignore a failure to create the directory due to other factors, such as insufficient permissions, full disk, etc.
One option would be to trap the OSError and examine the embedded error code (see Is there a cross-platform way of getting information from Python’s OSError):
import os, errno
try:
os.makedirs(directory)
except OSError as e:
if e.errno != errno.EEXIST:
raise
Alternatively, there could be a second os.path.exists, but suppose another created the directory after the first check, then removed it before the second one – we could still be fooled.
Depending on the application, the danger of concurrent operations may be more or less than the danger posed by other factors such as file permissions. The developer would have to know more about the particular application being developed and its expected environment before choosing an implementation.
Modern versions of Python improve this code quite a bit, both by exposing FileExistsError (in 3.3+)...
try:
os.makedirs("path/to/directory")
except FileExistsError:
# directory already exists
pass
...and by allowing a keyword argument to os.makedirs called exist_ok (in 3.2+).
os.makedirs("path/to/directory", exist_ok=True) # succeeds even if directory exists.
Python 3.5+:
import pathlib
pathlib.Path('/my/directory').mkdir(parents=True, exist_ok=True)
pathlib.Path.mkdir as used above recursively creates the directory and does not raise an exception if the directory already exists. If you don't need or want the parents to be created, skip the parents argument.
Python 3.2+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
If using Python 3.4, even though it comes with pathlib, it is missing the useful exist_ok option. The backport is intended to offer a newer and superior implementation of mkdir which includes this missing option.
Using os:
import os
os.makedirs(path, exist_ok=True)
os.makedirs as used above recursively creates the directory and does not raise an exception if the directory already exists. It has the optional exist_ok argument only if using Python 3.2+, with a default value of False. This argument does not exist in Python 2.x up to 2.7. As such, there is no need for manual exception handling as with Python 2.7.
Python 2.7+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
Using os:
import os
try:
os.makedirs(path)
except OSError:
if not os.path.isdir(path):
raise
While a naive solution may first use os.path.isdir followed by os.makedirs, the solution above reverses the order of the two operations. In doing so, it prevents a common race condition having to do with a duplicated attempt at creating the directory, and also disambiguates files from directories.
Note that capturing the exception and using errno is of limited usefulness because OSError: [Errno 17] File exists, i.e. errno.EEXIST, is raised for both files and directories. It is more reliable simply to check if the directory exists.
Alternative:
mkpath creates the nested directory, and does nothing if the directory already exists. This works in both Python 2 and 3. Note however that distutils has been deprecated, and is scheduled for removal in Python 3.12.
import distutils.dir_util
distutils.dir_util.mkpath(path)
Per Bug 10948, a severe limitation of this alternative is that it works only once per python process for a given path. In other words, if you use it to create a directory, then delete the directory from inside or outside Python, then use mkpath again to recreate the same directory, mkpath will simply silently use its invalid cached info of having previously created the directory, and will not actually make the directory again. In contrast, os.makedirs doesn't rely on any such cache. This limitation may be okay for some applications.
With regard to the directory's mode, please refer to the documentation if you care about it.
Using try except and the right error code from errno module gets rid of the race condition and is cross-platform:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
In other words, we try to create the directories, but if they already exist we ignore the error. On the other hand, any other error gets reported. For example, if you create dir 'a' beforehand and remove all permissions from it, you will get an OSError raised with errno.EACCES (Permission denied, error 13).
Starting from Python 3.5, pathlib.Path.mkdir has an exist_ok flag:
from pathlib import Path
path = Path('/my/directory/filename.txt')
path.parent.mkdir(parents=True, exist_ok=True)
# path.parent ~ os.path.dirname(path)
This recursively creates the directory and does not raise an exception if the directory already exists.
(just as os.makedirs got an exist_ok flag starting from python 3.2 e.g os.makedirs(path, exist_ok=True))
Note: when i posted this answer none of the other answers mentioned exist_ok...
I would personally recommend that you use os.path.isdir() to test instead of os.path.exists().
>>> os.path.exists('/tmp/dirname')
True
>>> os.path.exists('/tmp/dirname/filename.etc')
True
>>> os.path.isdir('/tmp/dirname/filename.etc')
False
>>> os.path.isdir('/tmp/fakedirname')
False
If you have:
>>> directory = raw_input(":: ")
And a foolish user input:
:: /tmp/dirname/filename.etc
... You're going to end up with a directory named filename.etc when you pass that argument to os.makedirs() if you test with os.path.exists().
Check os.makedirs: (It makes sure the complete path exists.)
To handle the fact the directory might exist, catch OSError.
(If exist_ok is False (the default), an OSError is raised if the target directory already exists.)
import os
try:
os.makedirs('./path/to/somewhere')
except OSError:
pass
Try the os.path.exists function
if not os.path.exists(dir):
os.mkdir(dir)
Insights on the specifics of this situation
You give a particular file at a certain path and you pull the directory from the file path. Then after making sure you have the directory, you attempt to open a file for reading. To comment on this code:
filename = "/my/directory/filename.txt"
dir = os.path.dirname(filename)
We want to avoid overwriting the builtin function, dir. Also, filepath or perhaps fullfilepath is probably a better semantic name than filename so this would be better written:
import os
filepath = '/my/directory/filename.txt'
directory = os.path.dirname(filepath)
Your end goal is to open this file, you initially state, for writing, but you're essentially approaching this goal (based on your code) like this, which opens the file for reading:
if not os.path.exists(directory):
os.makedirs(directory)
f = file(filename)
Assuming opening for reading
Why would you make a directory for a file that you expect to be there and be able to read?
Just attempt to open the file.
with open(filepath) as my_file:
do_stuff(my_file)
If the directory or file isn't there, you'll get an IOError with an associated error number: errno.ENOENT will point to the correct error number regardless of your platform. You can catch it if you want, for example:
import errno
try:
with open(filepath) as my_file:
do_stuff(my_file)
except IOError as error:
if error.errno == errno.ENOENT:
print 'ignoring error because directory or file is not there'
else:
raise
Assuming we're opening for writing
This is probably what you're wanting.
In this case, we probably aren't facing any race conditions. So just do as you were, but note that for writing, you need to open with the w mode (or a to append). It's also a Python best practice to use the context manager for opening files.
import os
if not os.path.exists(directory):
os.makedirs(directory)
with open(filepath, 'w') as my_file:
do_stuff(my_file)
However, say we have several Python processes that attempt to put all their data into the same directory. Then we may have contention over creation of the directory. In that case it's best to wrap the makedirs call in a try-except block.
import os
import errno
if not os.path.exists(directory):
try:
os.makedirs(directory)
except OSError as error:
if error.errno != errno.EEXIST:
raise
with open(filepath, 'w') as my_file:
do_stuff(my_file)
I have put the following down. It's not totally foolproof though.
import os
dirname = 'create/me'
try:
os.makedirs(dirname)
except OSError:
if os.path.exists(dirname):
# We are nearly safe
pass
else:
# There was an error on creation, so make sure we know about it
raise
Now as I say, this is not really foolproof, because we have the possiblity of failing to create the directory, and another process creating it during that period.
Check if a directory exists and create it if necessary?
The direct answer to this is, assuming a simple situation where you don't expect other users or processes to be messing with your directory:
if not os.path.exists(d):
os.makedirs(d)
or if making the directory is subject to race conditions (i.e. if after checking the path exists, something else may have already made it) do this:
import errno
try:
os.makedirs(d)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
But perhaps an even better approach is to sidestep the resource contention issue, by using temporary directories via tempfile:
import tempfile
d = tempfile.mkdtemp()
Here's the essentials from the online doc:
mkdtemp(suffix='', prefix='tmp', dir=None)
User-callable function to create and return a unique temporary
directory. The return value is the pathname of the directory.
The directory is readable, writable, and searchable only by the
creating user.
Caller is responsible for deleting the directory when done with it.
New in Python 3.5: pathlib.Path with exist_ok
There's a new Path object (as of 3.4) with lots of methods one would want to use with paths - one of which is mkdir.
(For context, I'm tracking my weekly rep with a script. Here's the relevant parts of code from the script that allow me to avoid hitting Stack Overflow more than once a day for the same data.)
First the relevant imports:
from pathlib import Path
import tempfile
We don't have to deal with os.path.join now - just join path parts with a /:
directory = Path(tempfile.gettempdir()) / 'sodata'
Then I idempotently ensure the directory exists - the exist_ok argument shows up in Python 3.5:
directory.mkdir(exist_ok=True)
Here's the relevant part of the documentation:
If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file.
Here's a little more of the script - in my case, I'm not subject to a race condition, I only have one process that expects the directory (or contained files) to be there, and I don't have anything trying to remove the directory.
todays_file = directory / str(datetime.datetime.utcnow().date())
if todays_file.exists():
logger.info("todays_file exists: " + str(todays_file))
df = pd.read_json(str(todays_file))
Path objects have to be coerced to str before other APIs that expect str paths can use them.
Perhaps Pandas should be updated to accept instances of the abstract base class, os.PathLike.
fastest safest way to do it is:
it will create if not exists and skip if exists:
from pathlib import Path
Path("path/with/childs/.../").mkdir(parents=True, exist_ok=True)
Best way to do this in python
#Devil
import os
directory = "./out_dir/subdir1/subdir2"
if not os.path.exists(directory):
os.makedirs(directory)
In Python 3.4 you can also use the brand new pathlib module:
from pathlib import Path
path = Path("/my/directory/filename.txt")
try:
if not path.parent.exists():
path.parent.mkdir(parents=True)
except OSError:
# handle error; you can also catch specific errors like
# FileExistsError and so on.
For a one-liner solution, you can use IPython.utils.path.ensure_dir_exists():
from IPython.utils.path import ensure_dir_exists
ensure_dir_exists(dir)
From the documentation: Ensure that a directory exists. If it doesn’t exist, try to create it and protect against a race condition if another process is doing the same.
IPython is an extension package, not part of the standard library.
In Python3, os.makedirs supports setting exist_ok. The default setting is False, which means an OSError will be raised if the target directory already exists. By setting exist_ok to True, OSError (directory exists) will be ignored and the directory will not be created.
os.makedirs(path,exist_ok=True)
In Python2, os.makedirs doesn't support setting exist_ok. You can use the approach in heikki-toivonen's answer:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
The relevant Python documentation suggests the use of the EAFP coding style (Easier to Ask for Forgiveness than Permission). This means that the code
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
else:
print "\nBE CAREFUL! Directory %s already exists." % path
is better than the alternative
if not os.path.exists(path):
os.makedirs(path)
else:
print "\nBE CAREFUL! Directory %s already exists." % path
The documentation suggests this exactly because of the race condition discussed in this question. In addition, as others mention here, there is a performance advantage in querying once instead of twice the OS. Finally, the argument placed forward, potentially, in favour of the second code in some cases --when the developer knows the environment the application is running-- can only be advocated in the special case that the program has set up a private environment for itself (and other instances of the same program).
Even in that case, this is a bad practice and can lead to long useless debugging. For example, the fact we set the permissions for a directory should not leave us with the impression permissions are set appropriately for our purposes. A parent directory could be mounted with other permissions. In general, a program should always work correctly and the programmer should not expect one specific environment.
I found this Q/A after I was puzzled by some of the failures and errors I was getting while working with directories in Python. I am working in Python 3 (v.3.5 in an Anaconda virtual environment on an Arch Linux x86_64 system).
Consider this directory structure:
└── output/ ## dir
├── corpus ## file
├── corpus2/ ## dir
└── subdir/ ## dir
Here are my experiments/notes, which provides clarification:
# ----------------------------------------------------------------------------
# [1] https://stackoverflow.com/questions/273192/how-can-i-create-a-directory-if-it-does-not-exist
import pathlib
""" Notes:
1. Include a trailing slash at the end of the directory path
("Method 1," below).
2. If a subdirectory in your intended path matches an existing file
with same name, you will get the following error:
"NotADirectoryError: [Errno 20] Not a directory:" ...
"""
# Uncomment and try each of these "out_dir" paths, singly:
# ----------------------------------------------------------------------------
# METHOD 1:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but no file created (os.makedirs creates dir, not files! ;-)
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# [2] https://docs.python.org/3/library/os.html#os.makedirs
# Uncomment these to run "Method 1":
#directory = os.path.dirname(out_dir)
#os.makedirs(directory, mode=0o777, exist_ok=True)
# ----------------------------------------------------------------------------
# METHOD 2:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## works
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## works
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but creates a .../doc.txt./ dir
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# Uncomment these to run "Method 2":
#import os, errno
#try:
# os.makedirs(out_dir)
#except OSError as e:
# if e.errno != errno.EEXIST:
# raise
# ----------------------------------------------------------------------------
Conclusion: in my opinion, "Method 2" is more robust.
[1] How can I safely create a nested directory?
[2] https://docs.python.org/3/library/os.html#os.makedirs
You can use mkpath
# Create a directory and any missing ancestor directories.
# If the directory already exists, do nothing.
from distutils.dir_util import mkpath
mkpath("test")
Note that it will create the ancestor directories as well.
It works for Python 2 and 3.
In case you're writing a file to a variable path, you can use this on the file's path to make sure that the parent directories are created.
from pathlib import Path
path_to_file = Path("zero/or/more/directories/file.ext")
parent_directory_of_file = path_to_file.parent
parent_directory_of_file.mkdir(parents=True, exist_ok=True)
Works even if path_to_file is file.ext (zero directories deep).
See pathlib.PurePath.parent and pathlib.Path.mkdir.
Why not use subprocess module if running on a machine that supports command
mkdir with -p option ?
Works on python 2.7 and python 3.6
from subprocess import call
call(['mkdir', '-p', 'path1/path2/path3'])
Should do the trick on most systems.
In situations where portability doesn't matter (ex, using docker) the solution is a clean 2 lines. You also don't have to add logic to check if directories exist or not. Finally, it is safe to re-run without any side effects
If you need error handling:
from subprocess import check_call
try:
check_call(['mkdir', '-p', 'path1/path2/path3'])
except:
handle...
You have to set the full path before creating the directory:
import os,sys,inspect
import pathlib
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
your_folder = currentdir + "/" + "your_folder"
if not os.path.exists(your_folder):
pathlib.Path(your_folder).mkdir(parents=True, exist_ok=True)
This works for me and hopefully, it will works for you as well
I saw Heikki Toivonen and A-B-B's answers and thought of this variation.
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST or not os.path.isdir(path):
raise
I use os.path.exists(), here is a Python 3 script that can be used to check if a directory exists, create one if it does not exist, and delete it if it does exist (if desired).
It prompts users for input of the directory and can be easily modified.
Use this command check and create dir
if not os.path.isdir(test_img_dir):
os.mkdir(test_img_dir)
Call the function create_dir() at the entry point of your program/project.
import os
def create_dir(directory):
if not os.path.exists(directory):
print('Creating Directory '+directory)
os.makedirs(directory)
create_dir('Project directory')
If you consider the following:
os.path.isdir('/tmp/dirname')
means a directory (path) exists AND is a directory. So for me this way does what I need. So I can make sure it is folder (not a file) and exists.
You can use os.listdir for this:
import os
if 'dirName' in os.listdir('parentFolderPath')
print('Directory Exists')
This may not exactly answer the question. But I guess your real intention is to create a file and its parent directories, given its content all in 1 command.
You can do that with fastcore extension to pathlib: path.mk_write(data)
from fastcore.utils import Path
Path('/dir/to/file.txt').mk_write('Hello World')
See more in fastcore documentation
I have a file called test.py with the following code:
import pytest
#pytest.mark.webtest
def test_http_request():
pass
class TestClass:
def test_method(self):
pass
pytest -s test.py passed but gave the following warnings:
pytest -s test.py
=============================== test session starts ============================
platform linux -- Python 3.7.3, pytest-5.2.4, py-1.8.0, pluggy-0.13.1
rootdir: /home/user
collected 2 items
test.py ..
=============================== warnings summary ===============================
anaconda3/lib/python3.7/site-packages/_pytest/mark/structures.py:325
~/anaconda3/lib/python3.7/site-packages/_pytest/mark/structures.py:325:
PytestUnknownMarkWarning: Unknown pytest.mark.webtest - is this a typo? You can register
custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
PytestUnknownMarkWarning,
-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============================== 2 passed, 1 warnings in 0.03s ===================
Environment: Python 3.7.3, pytest 5.2.4, anaconda3
What is the best way to get rid of the warning message?
To properly handle this you need to register the custom marker. Create a pytest.ini file and place the following inside of it.
[pytest]
markers =
webtest: mark a test as a webtest.
Next time you run the tests, the warning about the unregistered marker will not be there.
without updating pytest.ini, we can ignore warning using --disable-warnings
We can also use --disable-pytest-warnings
Example using your case:
pytest -s test.py -m webtest --disable-warnings
#gold_cy's answer works. If you have too many custom markers need to register in pytest.ini, an alternative way is to use the following configuration in pytest.ini:
[pytest]
filterwarnings =
ignore::UserWarning
or in general, use the following:
[pytest]
filterwarnings =
error
ignore::UserWarning
the configuration above will ignore all user warnings, but will transform all other warnings into errors. See more at Warnings Capture
test.py (updated with two custom markers)
import pytest
#pytest.mark.webtest
def test_http_request():
print("webtest::test_http_request() called")
class TestClass:
#pytest.mark.test1
def test_method(self):
print("test1::test_method() called")
Use the following commands to run desired tests:
pytest -s test.py -m webtest
pytest -s test.py -m test1
The best way to get rid of the message is to register the custom marker as per #gold_cy's answer.
However if you just wish to suppress the warning as per Jonathon's answer, rather than ignoring UserWarning (which will suppress all instances of the warning regardless of their source) you can specify the particular warning you want to suppress like so (in pytest.ini):
ignore::_pytest.warning_types.PytestUnknownMarkWarning
Note: For third party libraries/modules the full path to the warning is required to avoid an _OptionError exception
If you don't have pytest.ini and don't wanna create one just for this then you can also register it programmatically in conftest.py as described here:
def pytest_configure(config):
# register an additional marker
config.addinivalue_line(
"markers", "env(name): mark test to run only on named environment"
)
To add to the existing answers - custom markers can also be registered in pyproject.toml:
# pyproject.toml
[tool.pytest.ini_options]
markers = [
"webtest: mark a test as a webtest.",
]
Related docs.
Is there anything that offers replay-type functionality, by pointing at a predefined prompt-answer file?
What works and what I'd like to achieve.
Let's take an example, using a cookiecutter to prep a Python package for pypi
cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git
You've downloaded /Users/jluc/.cookiecutters/cookiecutter-pypackage before. Is it okay to delete and re-download it? [yes]:
full_name [Audrey Roy Greenfeld]: Spartacus 👈 constant for me/my organization
email [audreyr#example.com]: spartacus#example.com 👈 constant for me/my organization
...
project_name [Python Boilerplate]: GladiatorRevolt 👈 this will vary.
project_slug [q]: gladiator-revolt 👈 this too
...
OK, done.
Now, I can easily redo this, for this project, via:
cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git --replay
This is great!
What I want:
Say I create another project, UnleashHell.
I want to prep a file somehow that has my developer-info and project level info for Unleash. And I want to be able to run it multiple times against this template, without having to deal with prompts. This particular pypi template gets regular updates, for example python 2.7 support has been dropped.
The problem:
A --replay will just inject the last run for this cookiecutter template. If it was run against a different pypi project, too bad.
I'm good with my developer-level info, but I need to vary all the project level info.
I tried copying the replay file via:
cp ~/.cookiecutter_replay/cookiecutter-pypackage.json unleash.json
Edit unleash.json to reflect necessary changes.
Then specify it via --config-file flag
cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git --config-file unleash.json
I get an ugly error, it wants YAML, apparently.
cookiecutter.exceptions.InvalidConfiguration: Unable to parse YAML file .../000.packaging/unleash.json. Error: None of the known patterns match for {
"cookiecutter": {
"full_name": "Spartacus",
No problem, json2yaml to the rescue.
That doesn't work either.
cookiecutter.exceptions.InvalidConfiguration: Unable to parse YAML file ./cookie.yaml. Error: Unable to determine type for "
full_name: "Spartacus"
I also tried a < stdin redirect:
cookiecutter.prompts.txt:
yes
Spartacus
...
It doesn't seem to use it and just aborts.
cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git < ./cookiecutter.prompts.txt
You've downloaded ~/.cookiecutters/cookiecutter-pypackage before. Is it okay to delete and re-download it? [yes]
: full_name [Audrey Roy Greenfeld]
: email [audreyr#example.com]
: Aborted
I suspect I am missing something obvious, not sure what. To start with, what is the intent and format expected for the --config file?
Debrief - how I got it working from accepted answer.
Took accepted answer, but adjusted it for ~/.cookiecutterrc usage. It works but the format is not super clear. Especially not on the rc which has to be yaml, though that's not always/often the case with rc files.
This ended up working:
file ~/.cookiecutterrc:
without nesting under default_context... tons of unhelpful yaml parse errors (on a valid yaml doc).
default_context:
#... cut out for privacy
add_pyup_badge: y
command_line_interface: "Click"
create_author_file: "y"
open_source_license: "MIT license"
# the names to use here are:
# full_name:
# email:
# github_username:
# project_name:
# project_slug:
# project_short_description:
# pypi_username:
# version:
# use_pytest:
# use_pypi_deployment_with_travis:
# add_pyup_badge:
# command_line_interface:
# create_author_file:
# open_source_license:
I still could not get a combination of ~/.cookiecutterrc and a project-specific config.yaml to work. Too bad that expected configuration format is so lightly documented.
So I will use the .rc but enter the project name, slug and description each time. Oh well, good enough for now.
You are near.
Try this cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git --no-input --config-file config.yaml
The --no-input parameter will suppress the terminal user input, it is optional of course.
The config.yaml file could look like this:
default_context:
full_name: "Audrey Roy"
email: "audreyr#example.com"
github_username: "audreyr"
cookiecutters_dir: "/home/audreyr/my-custom-cookiecutters-dir/"
replay_dir: "/home/audreyr/my-custom-replay-dir/"
abbreviations:
pp: https://github.com/audreyr/cookiecutter-pypackage.git
gh: https://github.com/{0}.git
bb: https://bitbucket.org/{0}
Reference to this example file: https://cookiecutter.readthedocs.io/en/1.7.0/advanced/user_config.html
You probably just need the default_context block since that is where the user input goes.
I am writing a file using Python, and I want it to be placed in a specific path. How can I safely make sure that the path exists?
That is: how can I check whether the folder exists, along with its parents? If there are missing folders along the path, how can I create them?
On Python ≥ 3.5, use pathlib.Path.mkdir:
from pathlib import Path
Path("/my/directory").mkdir(parents=True, exist_ok=True)
For older versions of Python, I see two answers with good qualities, each with a small flaw, so I will give my take on it:
Try os.path.exists, and consider os.makedirs for the creation.
import os
if not os.path.exists(directory):
os.makedirs(directory)
As noted in comments and elsewhere, there's a race condition – if the directory is created between the os.path.exists and the os.makedirs calls, the os.makedirs will fail with an OSError. Unfortunately, blanket-catching OSError and continuing is not foolproof, as it will ignore a failure to create the directory due to other factors, such as insufficient permissions, full disk, etc.
One option would be to trap the OSError and examine the embedded error code (see Is there a cross-platform way of getting information from Python’s OSError):
import os, errno
try:
os.makedirs(directory)
except OSError as e:
if e.errno != errno.EEXIST:
raise
Alternatively, there could be a second os.path.exists, but suppose another created the directory after the first check, then removed it before the second one – we could still be fooled.
Depending on the application, the danger of concurrent operations may be more or less than the danger posed by other factors such as file permissions. The developer would have to know more about the particular application being developed and its expected environment before choosing an implementation.
Modern versions of Python improve this code quite a bit, both by exposing FileExistsError (in 3.3+)...
try:
os.makedirs("path/to/directory")
except FileExistsError:
# directory already exists
pass
...and by allowing a keyword argument to os.makedirs called exist_ok (in 3.2+).
os.makedirs("path/to/directory", exist_ok=True) # succeeds even if directory exists.
Python 3.5+:
import pathlib
pathlib.Path('/my/directory').mkdir(parents=True, exist_ok=True)
pathlib.Path.mkdir as used above recursively creates the directory and does not raise an exception if the directory already exists. If you don't need or want the parents to be created, skip the parents argument.
Python 3.2+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
If using Python 3.4, even though it comes with pathlib, it is missing the useful exist_ok option. The backport is intended to offer a newer and superior implementation of mkdir which includes this missing option.
Using os:
import os
os.makedirs(path, exist_ok=True)
os.makedirs as used above recursively creates the directory and does not raise an exception if the directory already exists. It has the optional exist_ok argument only if using Python 3.2+, with a default value of False. This argument does not exist in Python 2.x up to 2.7. As such, there is no need for manual exception handling as with Python 2.7.
Python 2.7+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
Using os:
import os
try:
os.makedirs(path)
except OSError:
if not os.path.isdir(path):
raise
While a naive solution may first use os.path.isdir followed by os.makedirs, the solution above reverses the order of the two operations. In doing so, it prevents a common race condition having to do with a duplicated attempt at creating the directory, and also disambiguates files from directories.
Note that capturing the exception and using errno is of limited usefulness because OSError: [Errno 17] File exists, i.e. errno.EEXIST, is raised for both files and directories. It is more reliable simply to check if the directory exists.
Alternative:
mkpath creates the nested directory, and does nothing if the directory already exists. This works in both Python 2 and 3. Note however that distutils has been deprecated, and is scheduled for removal in Python 3.12.
import distutils.dir_util
distutils.dir_util.mkpath(path)
Per Bug 10948, a severe limitation of this alternative is that it works only once per python process for a given path. In other words, if you use it to create a directory, then delete the directory from inside or outside Python, then use mkpath again to recreate the same directory, mkpath will simply silently use its invalid cached info of having previously created the directory, and will not actually make the directory again. In contrast, os.makedirs doesn't rely on any such cache. This limitation may be okay for some applications.
With regard to the directory's mode, please refer to the documentation if you care about it.
Using try except and the right error code from errno module gets rid of the race condition and is cross-platform:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
In other words, we try to create the directories, but if they already exist we ignore the error. On the other hand, any other error gets reported. For example, if you create dir 'a' beforehand and remove all permissions from it, you will get an OSError raised with errno.EACCES (Permission denied, error 13).
Starting from Python 3.5, pathlib.Path.mkdir has an exist_ok flag:
from pathlib import Path
path = Path('/my/directory/filename.txt')
path.parent.mkdir(parents=True, exist_ok=True)
# path.parent ~ os.path.dirname(path)
This recursively creates the directory and does not raise an exception if the directory already exists.
(just as os.makedirs got an exist_ok flag starting from python 3.2 e.g os.makedirs(path, exist_ok=True))
Note: when i posted this answer none of the other answers mentioned exist_ok...
I would personally recommend that you use os.path.isdir() to test instead of os.path.exists().
>>> os.path.exists('/tmp/dirname')
True
>>> os.path.exists('/tmp/dirname/filename.etc')
True
>>> os.path.isdir('/tmp/dirname/filename.etc')
False
>>> os.path.isdir('/tmp/fakedirname')
False
If you have:
>>> directory = raw_input(":: ")
And a foolish user input:
:: /tmp/dirname/filename.etc
... You're going to end up with a directory named filename.etc when you pass that argument to os.makedirs() if you test with os.path.exists().
Check os.makedirs: (It makes sure the complete path exists.)
To handle the fact the directory might exist, catch OSError.
(If exist_ok is False (the default), an OSError is raised if the target directory already exists.)
import os
try:
os.makedirs('./path/to/somewhere')
except OSError:
pass
Try the os.path.exists function
if not os.path.exists(dir):
os.mkdir(dir)
Insights on the specifics of this situation
You give a particular file at a certain path and you pull the directory from the file path. Then after making sure you have the directory, you attempt to open a file for reading. To comment on this code:
filename = "/my/directory/filename.txt"
dir = os.path.dirname(filename)
We want to avoid overwriting the builtin function, dir. Also, filepath or perhaps fullfilepath is probably a better semantic name than filename so this would be better written:
import os
filepath = '/my/directory/filename.txt'
directory = os.path.dirname(filepath)
Your end goal is to open this file, you initially state, for writing, but you're essentially approaching this goal (based on your code) like this, which opens the file for reading:
if not os.path.exists(directory):
os.makedirs(directory)
f = file(filename)
Assuming opening for reading
Why would you make a directory for a file that you expect to be there and be able to read?
Just attempt to open the file.
with open(filepath) as my_file:
do_stuff(my_file)
If the directory or file isn't there, you'll get an IOError with an associated error number: errno.ENOENT will point to the correct error number regardless of your platform. You can catch it if you want, for example:
import errno
try:
with open(filepath) as my_file:
do_stuff(my_file)
except IOError as error:
if error.errno == errno.ENOENT:
print 'ignoring error because directory or file is not there'
else:
raise
Assuming we're opening for writing
This is probably what you're wanting.
In this case, we probably aren't facing any race conditions. So just do as you were, but note that for writing, you need to open with the w mode (or a to append). It's also a Python best practice to use the context manager for opening files.
import os
if not os.path.exists(directory):
os.makedirs(directory)
with open(filepath, 'w') as my_file:
do_stuff(my_file)
However, say we have several Python processes that attempt to put all their data into the same directory. Then we may have contention over creation of the directory. In that case it's best to wrap the makedirs call in a try-except block.
import os
import errno
if not os.path.exists(directory):
try:
os.makedirs(directory)
except OSError as error:
if error.errno != errno.EEXIST:
raise
with open(filepath, 'w') as my_file:
do_stuff(my_file)
I have put the following down. It's not totally foolproof though.
import os
dirname = 'create/me'
try:
os.makedirs(dirname)
except OSError:
if os.path.exists(dirname):
# We are nearly safe
pass
else:
# There was an error on creation, so make sure we know about it
raise
Now as I say, this is not really foolproof, because we have the possiblity of failing to create the directory, and another process creating it during that period.
Check if a directory exists and create it if necessary?
The direct answer to this is, assuming a simple situation where you don't expect other users or processes to be messing with your directory:
if not os.path.exists(d):
os.makedirs(d)
or if making the directory is subject to race conditions (i.e. if after checking the path exists, something else may have already made it) do this:
import errno
try:
os.makedirs(d)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
But perhaps an even better approach is to sidestep the resource contention issue, by using temporary directories via tempfile:
import tempfile
d = tempfile.mkdtemp()
Here's the essentials from the online doc:
mkdtemp(suffix='', prefix='tmp', dir=None)
User-callable function to create and return a unique temporary
directory. The return value is the pathname of the directory.
The directory is readable, writable, and searchable only by the
creating user.
Caller is responsible for deleting the directory when done with it.
New in Python 3.5: pathlib.Path with exist_ok
There's a new Path object (as of 3.4) with lots of methods one would want to use with paths - one of which is mkdir.
(For context, I'm tracking my weekly rep with a script. Here's the relevant parts of code from the script that allow me to avoid hitting Stack Overflow more than once a day for the same data.)
First the relevant imports:
from pathlib import Path
import tempfile
We don't have to deal with os.path.join now - just join path parts with a /:
directory = Path(tempfile.gettempdir()) / 'sodata'
Then I idempotently ensure the directory exists - the exist_ok argument shows up in Python 3.5:
directory.mkdir(exist_ok=True)
Here's the relevant part of the documentation:
If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file.
Here's a little more of the script - in my case, I'm not subject to a race condition, I only have one process that expects the directory (or contained files) to be there, and I don't have anything trying to remove the directory.
todays_file = directory / str(datetime.datetime.utcnow().date())
if todays_file.exists():
logger.info("todays_file exists: " + str(todays_file))
df = pd.read_json(str(todays_file))
Path objects have to be coerced to str before other APIs that expect str paths can use them.
Perhaps Pandas should be updated to accept instances of the abstract base class, os.PathLike.
fastest safest way to do it is:
it will create if not exists and skip if exists:
from pathlib import Path
Path("path/with/childs/.../").mkdir(parents=True, exist_ok=True)
Best way to do this in python
#Devil
import os
directory = "./out_dir/subdir1/subdir2"
if not os.path.exists(directory):
os.makedirs(directory)
In Python 3.4 you can also use the brand new pathlib module:
from pathlib import Path
path = Path("/my/directory/filename.txt")
try:
if not path.parent.exists():
path.parent.mkdir(parents=True)
except OSError:
# handle error; you can also catch specific errors like
# FileExistsError and so on.
For a one-liner solution, you can use IPython.utils.path.ensure_dir_exists():
from IPython.utils.path import ensure_dir_exists
ensure_dir_exists(dir)
From the documentation: Ensure that a directory exists. If it doesn’t exist, try to create it and protect against a race condition if another process is doing the same.
IPython is an extension package, not part of the standard library.
In Python3, os.makedirs supports setting exist_ok. The default setting is False, which means an OSError will be raised if the target directory already exists. By setting exist_ok to True, OSError (directory exists) will be ignored and the directory will not be created.
os.makedirs(path,exist_ok=True)
In Python2, os.makedirs doesn't support setting exist_ok. You can use the approach in heikki-toivonen's answer:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
The relevant Python documentation suggests the use of the EAFP coding style (Easier to Ask for Forgiveness than Permission). This means that the code
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
else:
print "\nBE CAREFUL! Directory %s already exists." % path
is better than the alternative
if not os.path.exists(path):
os.makedirs(path)
else:
print "\nBE CAREFUL! Directory %s already exists." % path
The documentation suggests this exactly because of the race condition discussed in this question. In addition, as others mention here, there is a performance advantage in querying once instead of twice the OS. Finally, the argument placed forward, potentially, in favour of the second code in some cases --when the developer knows the environment the application is running-- can only be advocated in the special case that the program has set up a private environment for itself (and other instances of the same program).
Even in that case, this is a bad practice and can lead to long useless debugging. For example, the fact we set the permissions for a directory should not leave us with the impression permissions are set appropriately for our purposes. A parent directory could be mounted with other permissions. In general, a program should always work correctly and the programmer should not expect one specific environment.
I found this Q/A after I was puzzled by some of the failures and errors I was getting while working with directories in Python. I am working in Python 3 (v.3.5 in an Anaconda virtual environment on an Arch Linux x86_64 system).
Consider this directory structure:
└── output/ ## dir
├── corpus ## file
├── corpus2/ ## dir
└── subdir/ ## dir
Here are my experiments/notes, which provides clarification:
# ----------------------------------------------------------------------------
# [1] https://stackoverflow.com/questions/273192/how-can-i-create-a-directory-if-it-does-not-exist
import pathlib
""" Notes:
1. Include a trailing slash at the end of the directory path
("Method 1," below).
2. If a subdirectory in your intended path matches an existing file
with same name, you will get the following error:
"NotADirectoryError: [Errno 20] Not a directory:" ...
"""
# Uncomment and try each of these "out_dir" paths, singly:
# ----------------------------------------------------------------------------
# METHOD 1:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but no file created (os.makedirs creates dir, not files! ;-)
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# [2] https://docs.python.org/3/library/os.html#os.makedirs
# Uncomment these to run "Method 1":
#directory = os.path.dirname(out_dir)
#os.makedirs(directory, mode=0o777, exist_ok=True)
# ----------------------------------------------------------------------------
# METHOD 2:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## works
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## works
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but creates a .../doc.txt./ dir
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# Uncomment these to run "Method 2":
#import os, errno
#try:
# os.makedirs(out_dir)
#except OSError as e:
# if e.errno != errno.EEXIST:
# raise
# ----------------------------------------------------------------------------
Conclusion: in my opinion, "Method 2" is more robust.
[1] How can I safely create a nested directory?
[2] https://docs.python.org/3/library/os.html#os.makedirs
You can use mkpath
# Create a directory and any missing ancestor directories.
# If the directory already exists, do nothing.
from distutils.dir_util import mkpath
mkpath("test")
Note that it will create the ancestor directories as well.
It works for Python 2 and 3.
In case you're writing a file to a variable path, you can use this on the file's path to make sure that the parent directories are created.
from pathlib import Path
path_to_file = Path("zero/or/more/directories/file.ext")
parent_directory_of_file = path_to_file.parent
parent_directory_of_file.mkdir(parents=True, exist_ok=True)
Works even if path_to_file is file.ext (zero directories deep).
See pathlib.PurePath.parent and pathlib.Path.mkdir.
Why not use subprocess module if running on a machine that supports command
mkdir with -p option ?
Works on python 2.7 and python 3.6
from subprocess import call
call(['mkdir', '-p', 'path1/path2/path3'])
Should do the trick on most systems.
In situations where portability doesn't matter (ex, using docker) the solution is a clean 2 lines. You also don't have to add logic to check if directories exist or not. Finally, it is safe to re-run without any side effects
If you need error handling:
from subprocess import check_call
try:
check_call(['mkdir', '-p', 'path1/path2/path3'])
except:
handle...
You have to set the full path before creating the directory:
import os,sys,inspect
import pathlib
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
your_folder = currentdir + "/" + "your_folder"
if not os.path.exists(your_folder):
pathlib.Path(your_folder).mkdir(parents=True, exist_ok=True)
This works for me and hopefully, it will works for you as well
I saw Heikki Toivonen and A-B-B's answers and thought of this variation.
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST or not os.path.isdir(path):
raise
I use os.path.exists(), here is a Python 3 script that can be used to check if a directory exists, create one if it does not exist, and delete it if it does exist (if desired).
It prompts users for input of the directory and can be easily modified.
Use this command check and create dir
if not os.path.isdir(test_img_dir):
os.mkdir(test_img_dir)
Call the function create_dir() at the entry point of your program/project.
import os
def create_dir(directory):
if not os.path.exists(directory):
print('Creating Directory '+directory)
os.makedirs(directory)
create_dir('Project directory')
If you consider the following:
os.path.isdir('/tmp/dirname')
means a directory (path) exists AND is a directory. So for me this way does what I need. So I can make sure it is folder (not a file) and exists.
You can use os.listdir for this:
import os
if 'dirName' in os.listdir('parentFolderPath')
print('Directory Exists')
This may not exactly answer the question. But I guess your real intention is to create a file and its parent directories, given its content all in 1 command.
You can do that with fastcore extension to pathlib: path.mk_write(data)
from fastcore.utils import Path
Path('/dir/to/file.txt').mk_write('Hello World')
See more in fastcore documentation