Unable to start Celery in Flask application - python-3.x

I am getting the following error when trying to start a Celery worker (Windows).
I'm using Celery 5.0.5.
celery.exe -A api.app -> ModuleNotFoundError: No module named 'api'
main.py
import os
from api.app import create_app
app = create_app(os.getenv("FLASK_ENV"))
if __name__ == '__main__':
app.run(threaded=True, host='0.0.0.0')
api\app.py
from celery import Celery
from flask import Flask
from flask_redis import FlaskRedis
from flask_restful import Api
from api.config import env_config
redis_client = FlaskRedis()
celery = Celery(__name__, broker="redis://...")
def create_app(config_name):
import resources
app = Flask(__name__)
app.config.from_object(env_config[config_name])
redis_client.init_app(app)
app.config.update(
CELERY_BROKER_URL="redis://...",
CELERY_RESULT_BACKEND="redis://..."
)
celery.conf.update(app.config)
with app.app_context():
from .routes import ccl_routes
from .routes import scan_routes
return app
Folder structure is like so:
helheim
| |api
| |app.py
| |__init.py__
|main.py
What am I doing wrong? It's late here so probably something obvious but can't see it :)
Thanks!

edit: i think u can try celery -A api.app.celery worker before doing any major changes.
not sure how your entire file structure looks like but for example
project
| |_api
| |_app
_...
you should start it with celery -A project.api.app.celery worker. replace .celery to your celery instance name. or you can also refer to this repo https://github.com/mushcatshiro/flask-template.
also note, its a bit challenging to work with celery 4.X on windows from my experience, use celery 5.0 instead.

Related

Implementing celery messed up my INSTALLED_APPS (ModuleNotFoundError: No module named myapp)

I think there are may be similar questions to this one, but not quite the same so this is why I'm posting this. Basically my Djang Rest Framework was working fine, then I tried implementing celery and now it's messed up my imports, and seems to be throwing an error message after initiating a celery worker. I'm on Windows 10.
Project structure
- MyProj
-> MyProj (same name)
-> __init__.py
-> celery_tasks.py
-> settings.py
- MyApp
-> __init__.py
-> load_of_other_files
init.py
#from __future__ import absolute_import, unicode_literals
from MyProj.celery_tasks import app as celery_app
__all__ = ('celery_app',)
celery_tasks.py
#from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
import time
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
app = Celery("tasks", broker="amqp://localhost//", backend="rpc://")
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
settings.py
import os
..
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'MyApp',
]
Then I cd into my MyProj directory and trigger a worker like so:
celery -A celery_tasks worker --loglevel=INFO --concurrency=10 -n worker1.%h --pool=solo
The traceback I receive is:
ModuleNotFoundError: No module named 'MyApp'
I've tried with and without the from __future__ import absolute_import, unicode_literals but doesn't seem to work. I understand that the absolute_import is necessary to avoid the celery.py conflict with the celery library, which in my case is not the case, because my celery.py is called celery_tasks.py (to avoid confusion). How can I bypass this error message?
I had a similar error win10, Django and celery 5.0.5. I overcame this by modifying the app.autodiscover_tasks() within the celery.py file. In my case, the
app = Celery('project',
broker='amqp://',
backend='rpc://',
# include=['project.tasks'] ### project.tasks not found
)
did not resolve, but when I specified the Django modules which contain the tasks.py files i.e. app.autodiscover_tasks(['parm','demoapp']), the ModuleNotFoundError: was resolved.

Unable to run celery task directly but still possible via Python console

I'd like to run a simple test (run a task) first via RabbitMQ and once this is setup correctly, then encapsulate in Docker and run from there.
My structure looks like so:
-rabbitmq_docker
- test_celery
- __init__.py
- celeryapp.py
- celeryconfig.py
- runtasks.py
- tasks.py
- docker-compose.yml
- dockerfile
- requirements.txt
celeryconfig.py
## List of modules to import when celery starts
CELERY_IMPORTS = ['test_celery.tasks',] # Required to import module containing tasks
## Message Broker (RabbitMQ) settings
CELERY_BROKER_URL = "amqp://guest#localhost//"
CELERY_BROKER_PORT = 5672
CELERY_RESULT_BACKEND = 'rpc://'
celeryapp.py
from celery import Celery
app = Celery('test_celery')
app.config_from_object('test_celery.celeryconfig', namespace='CELERY')
__init__.py
from .celeryapp import app as celery_app
run_tasks.py
from tasks import reverse
from celery.utils.log import get_task_logger
LOGGER = get_task_logger(__name__)
if __name__ == '__main__':
async_result = reverse.delay("rabbitmq")
LOGGER.info(async_result.get())
tasks.py
from test_celery.celeryapp import app
#app.task(name='tasks.reverse')
def reverse(string):
return string[::-1]
I run celery -A test_celery worker --loglevel=info from the rabbitmq_docker directory. Then in a separate window I trigger reverse.delay("rabbitmq") in the Python console, after importing the required module. This works. Now when I try to trigger the reverse function via the run_tasks.py i.e. python test_celery/run_tasks.py I get:
Traceback (most recent call last):
File "test_celery/run_tasks.py", line 1, in <module>
from tasks import reverse
File "/Users/my_mbp/Software/rabbitmq_docker/test_celery/tasks.py", line 1, in <module>
from test_celery.celeryapp import app
ModuleNotFoundError: No module named 'test_celery'
What I don't get is why this Traceback doesn't get thrown when called directly from the Python console. Could anyone help me out here? I'd eventually like to startup docker, and just run the tests automatically (without going into the Python console).
The problem is simply because your module is not in the Python path.
These should help:
Specify the PYTHONPATH to point to the directory where your test_celery package.
Always run your Python code in the directory where your test_celery package is located.
Or alternatively reorganise your imports...

dask worker cannot import module

I am running a dask cluster and a worker w. 16 cores using the CLI utilities.
In general it seems to work very well.
However, for some reason it will not import modules in the cwd.
I try to run the following from my notebook instance:
def tstimp():
import os
return os.listdir()
c.run(tstimp)
And i get the following output:
{'tcp://192.168.1.90:35885': ['class_positions.csv',
'.gitignore',
'README.md',
'fullrun.ipynb',
'.git',
'rf.py',
'__pycache__',
'dask-worker-space',
'utils.py',
'.ipynb_checkpoints']}
Note that the module rf.py is listed here.
Thus it should be possible to import it in the worker, but when i run the following code:
def tstimp():
import rf
return 42
c.run(tstimp)
I get this error: ModuleNotFoundError: No module named 'rf'
Why am I getting this error?
It seems like the current directory is not added to the python path of the workers.
You should be able to fix this by adding it to the path.
def tstimp():
import sys
sys.path.append('.')
import rf
return 42
c.run(tstimp)

Deploying Python Flask app on web using pythonanywhere

I want to deploy my flask app publicly using pythonanywhere. I have followed all steps exactly. Tried implementing it with virtualenv and without virtualenv but none works.
I can get the simple flask page 'Hello to flask app" working but my code is not working.
Path is /home/anwaraliafshan/bella and file is afshan.py
This is my WSGI.py and I tried replacing flask with bella and afshan but nothing worked.
Also getting import imutil error in error.log though install imutil successfully on python3
Please help me finding the cause. Thanks in advance
# This file contains the WSGI configuration required to serve up your
# web application at http://<your-username>.pythonanywhere.com/
# It works by setting the variable 'application' to a WSGI handler of some
# description.
#
# The below has been auto-generated for your Flask project
import sys
# add your project directory to the sys.path
project_home = '/home/anwaraliafshan/bella/'
if project_home not in sys.path:
sys.path = [project_home] + sys.path
# import flask app but need to call it "application" for WSGI to work
from flask import app as application # noqa
As quoted from their website:
import sys
path = '/home/yourusername/mysite'
if path not in sys.path:
sys.path.insert(0, path)
from flask_app import app as application

ModuleNotFoundError when running imported Flask app

I have a python module with the following layout:
foo
| __init__.py
| __main__.py
| bar.py
__init__.py is empty.
Content of foo/bar.py:
from flask import Flask
app = Flask(__name__)
def baz(): pass
When running python3 -m foo i get confusing results.
Contents of foo/__main__.py
# Results in a ModuleNotFoundError: No module named 'foo'
from foo.bar import app
app.run()
# Raises no error and correctly prints the type
from foo.bar import app
print(type(app))
# Also runs without an error
from foo.bar import baz
baz()
Why is it possible to import and execute a function from this module, but when trying to do the same with a flask app it results in a ModuleNotFoundError?
I just can't see any way this makes any sense.
Edit:
The error is persistent even with this code:
from foo.bar import app
print(type(app))
app.run()
Output:
<class 'flask.app.Flask'>
* Serving Flask app "foo.bar" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
Traceback (most recent call last):
File "/home/user/projects/ftest/foo/__main__.py", line 1, in <module>
from foo.bar import app
ModuleNotFoundError: No module named 'foo'
So, obviously the module can be imported, because type(app) works just fine and flask does start. It seems like flask does a reload and is messing around with imports somehow.
Edit 2:
I turned debug mode off and it works just fine.
This error only occurs if you set export FLASK_DEBUG=True or explicitly enable debug via app.config["DEBUG"] = True
It turns out it's a bug in werkzeug.
The code works as expected if werkzeug's reloader is disabled.
How to reproduce the behaviour
Directory structure:
foo
| __init__.py
| __main__.py
Content of __init__.py:
from flask import Flask
app = Flask(__name__)
app.config["DEBUG"] = True
Content of __main__.py:
from foo import app
app.run()
If we run it:
$python3 -m foo
* Serving Flask app "foo" (lazy loading)
* Environment: development
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
Traceback (most recent call last):
File "/home/user/projects/ftest/foo/__main__.py", line 1, in <module>
from foo import app
ModuleNotFoundError: No module named 'foo'
If we change __main__.py:
from foo import app
app.run(use_reloader=False)
Everything works just fine.
What's going on
The problem is in werkzeug._reloader.ReloaderLoop.restart_with_reloader. It calls a subprocess with the arguments provided by werkzeug._reloader._get_args_for_reloading but this function does not behave as expected when executing a package via the -m switch.
def _get_args_for_reloading():
"""Returns the executable. This contains a workaround for windows
if the executable is incorrectly reported to not have the .exe
extension which can cause bugs on reloading.
"""
rv = [sys.executable]
py_script = sys.argv[0]
if os.name == 'nt' and not os.path.exists(py_script) and \
os.path.exists(py_script + '.exe'):
py_script += '.exe'
if os.path.splitext(rv[0])[1] == '.exe' and os.path.splitext(py_script)[1] == '.exe':
rv.pop(0)
rv.append(py_script)
rv.extend(sys.argv[1:])
return rv
In our case it returns ['/usr/local/bin/python3.7', '/home/user/projects/ftest/foo/__main__.py']. This is because sys.argv[0] is set to the full path of the module file but it should return ['/usr/local/bin/python3.7', '-m', 'foo']` (At least from my understanding it should and it works this way).
I have no good idea on how to fix this behaviour, or if it is something that need to be fixed. It just seems weird to me that I'm the only one that has encountered this problem, since it doesn't seem too much of a corner case to me.
Adding the following line before app.run() works around the werkzeug reloader bug:
os.environ['PYTHONPATH'] = os.getcwd()
Thanks to #bootc for the tip! https://github.com/pallets/flask/issues/1246
Have you tried from foo import app in your main.py file?

Resources