Related
I have created flask app that can run jupyter notebook server.
When we click on app from the UI, a new subprocess is created using subprocess.Popen() to construct a Jupyter notebook server.
I created the windows service for the flask webapp using win32serviceutil.
There are three scenarios.
Scenario 1: Jupyter notebooks work properly when a flask app run using python.exe .
Scenario 2: When the Flask app service is executed in debug mode , it works fine.
command :-
python service.py debug
Scenario 3 :- When flask app is executed as windows service jupyter notebook kernel dies.
Error :-
Traceback (most recent call last):
File "d:\v307\lib\site-packages\tornado\web.py", line 1704, in _execute
result = await result
File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 328, in __wakeup
future.result()
File "d:\v307\lib\site-packages\tornado\gen.py", line 769, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "d:\v307\lib\site-packages\notebook\services\sessions\handlers.py", line 74, in post
model = yield maybe_future(
File "d:\v307\lib\site-packages\tornado\gen.py", line 762, in run
value = future.result()
File "d:\v307\lib\site-packages\tornado\gen.py", line 769, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "d:\v307\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 98, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
File "d:\v307\lib\site-packages\tornado\gen.py", line 762, in run
value = future.result()
File "d:\v307\lib\site-packages\tornado\gen.py", line 769, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "d:\v307\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 110, in start_kernel_for_session
kernel_id = yield maybe_future(
File "d:\v307\lib\site-packages\tornado\gen.py", line 762, in run
value = future.result()
File "C:\Program Files\Python39\lib\asyncio\futures.py", line 201, in result
raise self._exception
File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 256, in __step
result = coro.send(None)
File "d:\v307\lib\site-packages\notebook\services\kernels\kernelmanager.py", line 176, in start_kernel
kernel_id = await maybe_future(self.pinned_superclass.start_kernel(self, **kwargs))
File "d:\v307\lib\site-packages\jupyter_client\utils.py", line 30, in wrapped
raise e
File "d:\v307\lib\site-packages\jupyter_client\utils.py", line 27, in wrapped
return loop.run_until_complete(future)
File "d:\v307\lib\site-packages\nest_asyncio.py", line 89, in run_until_complete
return f.result()
File "C:\Program Files\Python39\lib\asyncio\futures.py", line 201, in result
raise self._exception
File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 256, in __step
result = coro.send(None)
File "d:\v307\lib\site-packages\jupyter_client\multikernelmanager.py", line 212, in _async_start_kernel
starter = ensure_async(km.start_kernel(**kwargs))
File "d:\v307\lib\site-packages\jupyter_client\utils.py", line 30, in wrapped
raise e
File "d:\v307\lib\site-packages\jupyter_client\utils.py", line 27, in wrapped
return loop.run_until_complete(future)
File "d:\v307\lib\site-packages\nest_asyncio.py", line 89, in run_until_complete
return f.result()
File "C:\Program Files\Python39\lib\asyncio\futures.py", line 201, in result
raise self._exception
File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 256, in __step
result = coro.send(None)
File "d:\v307\lib\site-packages\jupyter_client\manager.py", line 83, in wrapper
raise e
File "d:\v307\lib\site-packages\jupyter_client\manager.py", line 75, in wrapper
out = await method(self, *args, **kwargs)
File "d:\v307\lib\site-packages\jupyter_client\manager.py", line 383, in _async_start_kernel
kernel_cmd, kw = await ensure_async(self.pre_start_kernel(**kw))
File "d:\v307\lib\site-packages\jupyter_client\utils.py", line 30, in wrapped
raise e
File "d:\v307\lib\site-packages\jupyter_client\utils.py", line 27, in wrapped
return loop.run_until_complete(future)
File "d:\v307\lib\site-packages\nest_asyncio.py", line 89, in run_until_complete
return f.result()
File "C:\Program Files\Python39\lib\asyncio\futures.py", line 201, in result
raise self._exception
File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 256, in __step
result = coro.send(None)
File "d:\v307\lib\site-packages\jupyter_client\manager.py", line 349, in _async_pre_start_kernel
kw = await self.provisioner.pre_launch(**kw)
File "d:\v307\lib\site-packages\jupyter_client\provisioning\local_provisioner.py", line 190, in pre_launch
km.write_connection_file()
File "d:\v307\lib\site-packages\jupyter_client\connect.py", line 503, in write_connection_file
self.connection_file, cfg = write_connection_file(
File "d:\v307\lib\site-packages\jupyter_client\connect.py", line 164, in write_connection_file
with secure_write(fname) as f:
File "C:\Program Files\Python39\lib\contextlib.py", line 117, in __enter__
return next(self.gen)
File "d:\v307\lib\site-packages\jupyter_core\paths.py", line 893, in secure_write
win32_restrict_file_to_user(fname)
File "d:\v307\lib\site-packages\jupyter_core\paths.py", line 442, in win32_restrict_file_to_user
user, _domain, _type = win32security.LookupAccountName(
pywintypes.error: (1789, 'LookupAccountName', 'The trust relationship between this workstation and the primary domain failed.')
I have used waitress as a web server and win32serviceutil for creating service of flask app
Python version 3.9
I have tried using NSSM and Winsw for adding flask app as a service but faced same error.
Added creationflags=CREATE_NEW_CONSOLE argument while creating subprocess using Popen but no progress.
Expection
Way to solve issue when we run flask webapp as a service.
When trying to create a fabric2.Connection, Paramiko tries to invoke a local /bin/bash command:
$ fab2 db-shell
Traceback (most recent call last):
File "/nix/store/m2iyj18cifr4a1rvpfgphg7kfgsf2pj2-python3.9-fabric2-2.7.1/bin/.fab2-wrapped", line 9, in <module>
sys.exit(program.run())
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/program.py", line 384, in run
self.execute()
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/program.py", line 566, in execute
executor.execute(*self.tasks)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/executor.py", line 129, in execute
result = call.task(*args, **call.kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/tasks.py", line 127, in __call__
result = self.body(*args, **kwargs)
File "/home/username/project/fabfile.py", line 645, in db_shell
bastion_connection().run(
File "/home/username/project/fabfile_utils.py", line 141, in bastion_connection
conn = Connection(
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/fabric2/connection.py", line 403, in __init__
self.ssh_config = self.config.base_ssh_config.lookup(host)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 223, in lookup
options = self._lookup(hostname=hostname)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 250, in _lookup
or self._does_match(
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 389, in _does_match
passed = invoke.run(exec_cmd, hide="stdout", warn=True).ok
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/__init__.py", line 48, in run
return Context().run(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/context.py", line 95, in run
return self._run(runner, command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/context.py", line 102, in _run
return runner.run(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 380, in run
return self._run_body(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 431, in _run_body
self.start(command, self.opts["shell"], self.env)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 1291, in start
self.process = Popen(
File "/nix/store/0zzvjh5gnz0ny7ckilzyn9hmg5lypszf-python3-3.9.13/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/nix/store/0zzvjh5gnz0ny7ckilzyn9hmg5lypszf-python3-3.9.13/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/bin/bash'
I've not been able to find anything in the docs about this yet.
I'm running fabric2==2.7.1, invoke==1.6.0, and paramiko==2.8.0 on NixOS stable 22.11.
Invoke currently defaults to /bin/bash, which is a known issue, since operating systems like Alpine and NixOS don't have that path. Fortunately we can specify a command in $PATH rather than manually looking it up, so adding the following to the Connection call does the trick: config=invoke.Config(overrides={"shell": "bash"}).
I am trying to make an executable of a python 3.10 file using PyInstaller in the command prompt:
python -m PyInstaller --onefile 1.0.py
with "1.0" being my file name. I have done this many times before on the same PC and with the same python version, but for this file it is throwing a "file specified not found" error:
while run_next_command(read_fh, write_fh):
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\__main__.py", line 188, in <module>
run()
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\__main__.py", line 178, in run
run_build(pyi_config, spec_file, **vars(args))
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\__main__.py", line 59, in run_build
PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\building\build_main.py", line 934, in main
build(specfile, distpath, workpath, clean_build)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\building\build_main.py", line 856, in build
exec(code, spec_namespace)
File "E:\Python Scripts\High Spen\Combined\1.0.spec", line 7, in <module>
a = Analysis(
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\building\build_main.py", line 381, in __init__
self.__postinit__()
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\building\datastruct.py", line 173, in __postinit__
self.assemble()
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\building\build_main.py", line 660, in assemble
isolated.call(find_binary_dependencies, list(self.binaries), self.binding_redirects, collected_packages)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\isolated\_parent.py", line 238, in call
return isolated.call(function, *args, **kwargs)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\isolated\_parent.py", line 176, in call
raise RuntimeError(f"Child process call to {function.__name__}() failed with:\n" + output)
RuntimeError: Child process call to find_binary_dependencies() failed with:
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\win32ctypes\pywin32\pywintypes.py", line 35, in pywin32error
yield
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\win32ctypes\pywin32\win32api.py", line 43, in LoadLibraryEx
return _dll._LoadLibraryEx(fileName, 0, flags)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\win32ctypes\core\cffi\_dll.py", line 23, in _LoadLibraryEx
result = check_null(
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\win32ctypes\core\cffi\_util.py", line 81, in __call__
self._raise_error(function_name)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\win32ctypes\core\cffi\_util.py", line 92, in _raise_error
raise exception
OSError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\isolated\_child.py", line 63, in run_next_command
output = function(*args, **kwargs)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\building\build_main.py", line 162, in find_binary_dependencies
return bindepend.Dependencies(binaries, redirects=binding_redirects, xtrapath=extra_libdirs)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\depend\bindepend.py", line 205, in Dependencies
for ftocnm, fn in getAssemblyFiles(pth, manifest, redirects):
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\depend\bindepend.py", line 372, in getAssemblyFiles
for assembly in getAssemblies(pth):
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\depend\bindepend.py", line 326, in getAssemblies
res = winmanifest.GetManifestResources(pth)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\utils\win32\winmanifest.py", line 979, in GetManifestResources
return winresource.GetResources(filename, [RT_MANIFEST], names, languages)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\PyInstaller\utils\win32\winresource.py", line 155, in GetResources
hsrc = win32api.LoadLibraryEx(filename, 0, LOAD_LIBRARY_AS_DATAFILE)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\win32ctypes\pywin32\win32api.py", line 42, in LoadLibraryEx
with _pywin32error():
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "C:\Users\jackp\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\win32ctypes\pywin32\pywintypes.py", line 37, in pywin32error
raise error(exception.winerror, exception.function, exception.strerror)
win32ctypes.pywin32.pywintypes.error: (2, 'LoadLibraryEx', 'The system cannot find the file specified')
Please could someone let me know how I would solve this or otherwise another method of turning this into an .exe. Thanks
Update:
I have tried uninstalling my python 3.10 and downloading 3.7 instead, as I saw somewhere that PyInstaller is incompatible with 3.8 and above, but this still didn't work for another reason. So I reinstalled 3.10 (effectively at this point I have uninstalled and reinstalled my python and IDE) and I am still getting the same error.
I am trying to host django project on EC2 usnig apache2 which is using mongodb atlas as database.
When I am running this locally on EC2 after openning some port like 8000, it runs properly but on apache it's giving error.
Below is some TLSFeature error, I am facing
mod_wsgi (pid=67994, process='tracky', application='ip-IP_ADDRESS|'): Loading Python script file '/home/tracky/tracky-api/tracky/wsgi.py'.
mongodb+srv://username:password#host/dbname?retryWrites=true&w=majority
mod_wsgi (pid=67994): Exception occurred processing WSGI script '/home/tracky/tracky-api/tracky/wsgi.py'.
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pool.py", line 1278, in _get_socket
sock_info = self.sockets.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site- File "/home/tracky/tracky-api/trackyapi/urls.py", line 2, in <module>
from .views import Tracky
File "/home/tracky/tracky-api/trackyapi/views.py", line 16, in <module>
from .serializers import GoalsSerializer, GoalSerializer
File "/home/tracky/tracky-api/trackyapi/serializers.py", line 9, in <module>
class GoalSerializer(DocumentSerializer):
File "/home/tracky/tracky-api/trackyapi/serializers.py", line 11, in GoalSerializer
uid = ReferenceField(User, write_only=True)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/rest_framework_mongoengine/fields.py", line 217, in __init__
self.queryset = model.objects
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/mongoengine/queryset/manager.py", line 38, in __get__
queryset = queryset_class(owner, owner._get_collection())
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/mongoengine/document.py", line 215, in _get_collection
db = cls._get_db()
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/mongoengine/document.py", line 193, in _get_db
return get_db(cls._meta.get("db_alias", DEFAULT_CONNECTION_NAME))
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/mongoengine/connection.py", line 363, in get_db
db.authenticate(
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/database.py", line 1492, in authenticate
self.client._cache_credentials(
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/mongo_client.py", line 780, in _cache_credentials
with server.get_socket(all_credentials) as sock_info:
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pool.py", line 1231, in get_socket
sock_info = self._get_socket(all_credentials)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pool.py", line 1281, in _get_socket
sock_info = self.connect(all_credentials)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pool.py", line 1180, in connect
sock = _configured_socket(self.address, self.opts)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pool.py", line 1002, in _configured_socket
sock = ssl_context.wrap_socket(sock, server_hostname=host)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pyopenssl_context.py", line 313, in wrap_socket
ssl_conn.do_handshake()
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pyopenssl_context.py", line 116, in do_handshake
return self._call(super(_sslConn, self).do_handshake, *args, **kwargs)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/pyopenssl_context.py", line 107, in _call
return call(*args, **kwargs)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1618, in _raise_ssl_error
self._context._ocsp_helper.raise_if_problem()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 294, in raise_if_problem
raise self._problems.pop(0)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 590, in wrapper
valid = callback(conn, ocsp_data, data)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/ocsp_support.py", line 292, in _ocsp_callback
ext = _get_extension(cert, _TLSFeature)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/pymongo/ocsp_support.py", line 119, in _get_extension
return cert.extensions.get_extension_for_class(klass)
File "/home/ubuntu/anaconda3/envs/tracky/lib/python3.6/site-packages/cryptography/x509/extensions.py", line 134, in get_extension_for_class
raise ExtensionNotFound(
cryptography.x509.extensions.ExtensionNotFound: No <class 'cryptography.x509.extensions.TLSFeature'> extension was found
Any suggestion will be helpful.
Thanks.
I have set up Airflow on an AWS EC2 server with Ubuntu 18.04, Python 3.6.9. The DB backend is a 5.7.26 MySQL, I am using a LocalExecutor. The setup is nothing more than:
apt-get install python3 python3-pip python3-venv libmysqlclient-dev
I install Airflow with pip3 install apache-airflow[mysql]==1.10.9, init DB connection, start the webserver and it works both normally and with -D. The scheduler, however, works only when run in the foreground. Trying to run it as a daemon fails with the following trace:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 37, in <module> args.func(args)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/cli.py", line 75, in wrapper return f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/bin/cli.py", line 1032, in schedulerjob.run()
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/base_job.py", line 215, in run session.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 1036, in commit self.transaction.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 503, in commitself._prepare_impl()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 482, in _prepare_impl self.session.flush()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2496, in flushself._flush(objects)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2637, in _flush transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__exc_value, with_traceback=exc_tb,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 178, in raise_raise exception
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2597, in _flush flush_context.execute()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute rec.execute(self)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute uow,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py", line 213, in save_obj) in _organize_states_for_save(base_mapper, states, uowtransaction):
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py", line 374, in _organize_states_for_save base_mapper, uowtransaction, states
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states connection = uowtransaction.transaction.connection(base_mapper)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 313, in connection return self._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 420, in _connection_for_bind conn = self._parent._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 432, in _connection_for_bind conn = bind._contextual_connect()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 2251, in _contextual_connect self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect return fn()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 363, in connect return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 804, in _checkout result = pool._dialect.do_ping(fairy.connection)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 138, in do_pingdbapi_connection.ping(False)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 546, in ping self._execute_command(COMMAND.COM_PING, "")
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 771, in _execute_command self._write_bytes(packet)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 711, in _write_bytesself._sock.settimeout(self._write_timeout)
OSError: [Errno 9] Bad file descriptor
Exception ignored in: <function _ConnectionRecord.checkout.<locals>.<lambda> at 0x7f2eb025e268>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 503, in <lambda>
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 702, in _finalize_fairy
File "/usr/lib/python3.6/logging/__init__.py", line 1337, in error
File "/usr/lib/python3.6/logging/__init__.py", line 1444, in _log
File "/usr/lib/python3.6/logging/__init__.py", line 1454, in handle
File "/usr/lib/python3.6/logging/__init__.py", line 1516, in callHandlers
File "/usr/lib/python3.6/logging/__init__.py", line 865, in handle
File "/usr/lib/python3.6/logging/__init__.py", line 1071, in emit
File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open
NameError: name 'open' is not defined
I am trying to set up a server for semi-production purposes, so this is actually a blocker for me. I would be grateful for any advice.
EDIT
I tried using mysql-connector-python==8.0.18 and the scheduler did run as a daemon, but an attempt to open a dag failed with the following trace:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/dist-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/flask_login/utils.py", line 258, in decorated_view
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/www/utils.py", line 384, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/www/utils.py", line 290, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/www/views.py", line 1559, in tree
start_date=min_date, end_date=base_date, session=session)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/models/dag.py", line 837, in get_task_instances
tis = tis.order_by(TaskInstance.execution_date).all()
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3233, in all
return list(self)
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3389, in __iter__
return self._execute_and_instances(context)
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3414, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 982, in execute
return meth(self, multiparams, params)
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
distilled_params,
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
e, statement, parameters, cursor, context
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1478, in _handle_dbapi_exception
util.reraise(*exc_info)
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
raise value
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "/home/ubuntu/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
File "/home/ubuntu/.local/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 272, in execute
self._handle_result(result)
File "/home/ubuntu/.local/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 163, in _handle_result
self._handle_resultset()
File "/home/ubuntu/.local/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 651, in _handle_resultset
self._rows = self._cnx.get_rows()[0]
File "/home/ubuntu/.local/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 301, in get_rows
else self._cmysql.fetch_row()
SystemError: <built-in method fetch_row of _mysql_connector.MySQL object at 0x3cdaa10> returned a result with an error set
EDIT 2
What finally did work was using mysqlclient by specifying the SQLAlchemy connection string to begin with mysql+mysqldb. I am not giving this as an answer, because the initial problem persists.