I cannot run boto3 on Ubuntu 18.04 (AWS), initially I thought the code was broken but it might actually be the installation itself?
ubuntu#ip-172-30-10-199:~$ sudo -H pip3 install boto3
Requirement already satisfied: boto3 in /usr/lib/python3/dist-packages
ubuntu#ip-172-30-10-199:~$ sudo -H pip3 install botocore
Requirement already satisfied: botocore in /usr/lib/python3/dist-packages
ubuntu#ip-172-30-10-199:~$ python3 --version
**Python 3.6.9**
ubuntu#ip-172-30-10-199:~$ sudo apt-get install python3-boto3
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-boto3 is already the newest version (1.4.2-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ubuntu#ip-172-30-10-199:~$
s3_list.py file:
import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print(bucket.name)
Running the code.
ubuntu#ip-172-30-10-199:~$ python3 s3_list.py
Traceback (most recent call last):
File "s3_list.py", line 4, in <module>
s3 = boto3.resource('s3')
File "/home/ubuntu/boto3/__init__.py", line 100, in resource
return _get_default_session().resource(*args, **kwargs)
File "/home/ubuntu/boto3/session.py", line 389, in resource
aws_session_token=aws_session_token, config=config)
File "/home/ubuntu/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/home/ubuntu/botocore/session.py", line 835, in create_client
client_config=config, api_version=api_version)
File "/home/ubuntu/botocore/client.py", line 79, in create_client
cls = self._create_client_class(service_name, service_model)
File "/home/ubuntu/botocore/client.py", line 109, in _create_client_class
base_classes=bases)
File "/home/ubuntu/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/home/ubuntu/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/home/ubuntu/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/home/ubuntu/boto3/utils.py", line 61, in _handler
module = import_module(module)
File "/home/ubuntu/boto3/utils.py", line 52, in import_module
__import__(name)
File "/home/ubuntu/boto3/s3/inject.py", line 15, in <module>
from boto3.s3.transfer import create_transfer_manager
File "/home/ubuntu/boto3/s3/transfer.py", line 127, in <module>
from s3transfer.exceptions import RetriesExceededError as \
File "/home/ubuntu/s3transfer/__init__.py", line 134, in <module>
import concurrent.futures
File "/home/ubuntu/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/home/ubuntu/concurrent/futures/_base.py", line 414
raise exception_type, self._exception, self._traceback
^
SyntaxError: invalid syntax
ubuntu#ip-172-30-10-199:~$
I do have a file ~/.aws/credentials containing the respective keys. I do use these same keys to access the bucket from my laptop (S3 Browser).
I manage to find the answer.
Somehow it was related to S3 permissions, instead of using Key/secret e manage to give S3 Fullcontrol role to my EC2 and it worked like a charm.
Mea culpa, I think the error message could be also more informative. Please learn from my mistake :-)
Related
I understand that huggingface prefers to load models from the internet. However, I have a machine that doesn't have access to the internet. So, I am trying to load a model from its cached files, which I have moved to a custom directory /tmp/models/huggingface_cache. I have a volume for the container to access these files, and I have verified the container has access.
What I don't understand is why huggingface wants to create a directory to manage cache when I specified a custom directory first. The volume access should remain as read only. Therefore, I'm curious why I'm having these issues, and whether there is a better route to workaround huggingface's functionality of loading cached models.
I'm running an interactive container built from Ubuntu with the following command:
docker run -it --rm -v /tmp:/tmp:rw generic-container /bin/bash
I then run the following commands in python3:
from transformers import pipeline
from os import environ
environ["TRANSFORMERS_OFFLINE"] = "1"
environ["TRANSFORMERS_CACHE"] = "/tmp/models/huggingface_cache"
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli", cache_dir="/tmp/models/huggingface_cache")
And I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/transformers/pipelines/__init__.py", line 645, in pipeline
config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/configuration_auto.py", line 809, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 559, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 614, in _get_config_dict
resolved_config_file = cached_file(
File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 409, in cached_file
resolved_file = hf_hub_download(
File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_validators.py", line 124, in _inner_fn
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/file_download.py", line 1038, in hf_hub_download
os.makedirs(storage_folder, exist_ok=True)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.cache'
Environment(s)
Ubuntu 20.04 & Debian 10 with Python 3.8 or 3.7, respectively.
Postgresql versions 11, 12, and 14 have been tried.
Psycopg2-binary 2.8.6
Overview
I'm attempting to install a Django project, and I'm getting this error:
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/citext.control": No such file or directory
The psycopg devs informed me this is likely an issue with the postgresql-contrib libraries. Similarly, others have been able to fix this error by installing postgresql-contrib, however, this does not work for me. I've also tried installing postgresql-12.
I can see that citext.control is available in /usr/share/postgresql/12/extension/citext.control, so I tried ln -s /usr/share/postgresql/12 /usr/share/pgsql with no effect.
I also ran CREATE EXTENSION citext; in Postgres, also without effect.
Any support with this would be greatly appreciated, as I was hoping to have this project live already!
Thanks so much.
Trace
Running migrations:
Applying core.0043_install_ci_extension_pg...Traceback (most recent call last):
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/pgsql/extension/citext.control": No such file or directory
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 16, in <module>
execute_from_command_line(sys.argv)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/home/user/janeway/src/utils/management/commands/install_janeway.py", line 58, in handle
call_command('migrate')
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/core/management/__init__.py", line 131, in call_command
return command.execute(*args, **defaults)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 202, in handle
post_migrate_state = executor.migrate(
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/contrib/postgres/operations.py", line 17, in database_forwards
schema_editor.execute("CREATE EXTENSION IF NOT EXISTS %s" % schema_editor.quote_name(self.name))
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 136, in execute
cursor.execute(sql, params)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/user/venvs/janeway/lib/python3.8/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.OperationalError: could not open extension control file "/usr/share/pgsql/extension/citext.control": No such file or directory
Have you tried installing python3-psycopg2 from the packaging system instead?
My Ubuntu 20 setup has this installed, and postgresql connections work with django.
ii python3-psycopg2 2.8.4-2 amd64 Python 3 module for PostgreSQL
I had installed postgresql-contrib on the local django server rather than the remote DB server. Installing on the same server as Postgresql resolved the issue.
My local Rethink database disappeared after upgrading to MacOS 11 (“Big Sur”). But that's alright, as I have a backup. But I can't, for the life of me, restore it.
rethinkdb restore [file] and rethinkdb import [file] didn't work, but after some googling I found that I had to install the python tools (pip3 install rethinkdb). I did that, but now I get this error:
% rethinkdb restore rethinkdb_dump_2020-11-21T20:35:20.tar.gz > bug.log
Traceback (most recent call last):
File "/usr/local/bin/rethinkdb-restore", line 10, in <module>
sys.exit(main())
File "/Users/$USER/Library/Python/3.8/lib/python/site-packages/rethinkdb/_restore.py", line 339, in main
do_restore(options)
File "/Users/$USER/Library/Python/3.8/lib/python/site-packages/rethinkdb/_restore.py", line 315, in do_restore
_import.import_tables(options, sources)
File "/Users/$USER/Library/Python/3.8/lib/python/site-packages/rethinkdb/_import.py", line 1359, in import_tables
progress_bar.start()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread._local' object
I've tried to uninstall RethinkDB and the Python tools and reinstall both. Same outcome.
I was able to downgrade to python 3.7, reinstall the rethinkdb python driver and made it further in the rethinkdb restore process
I am new to Python and trying to install Airflow in my Mac, by following this tutorial
While these two commands work fine:
$ airflow initdb
$ airflow webserver -p 8080
The scheduler command (airflow scheduler) throws the following error:
[2020-02-18 13:18:09,012] {scheduler_job.py:1382} ERROR - Exception when executing execute_helper Traceback (most recent call last):
File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1380, in _execute
self._execute_helper()
File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1413, in _execute_helper
self.processor_agent.start()
File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 554, in start
self._process.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'SchedulerJob._execute.<locals>.processor_factory'
[2020-02-18 13:18:09,035] {helpers.py:322} INFO - Sending Signals.SIGTERM to GPID None
Traceback (most recent call last): File "/Users/mac/Workspace/airflow/airflow_venv/bin/airflow", line 37, in <module>
args.func(args) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 75, in wrapper
return f(*args, **kwargs) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/bin/cli.py", line 1040, in scheduler
job.run() File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 221, in run
self._execute() File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1384, in _execute
self.processor_agent.end() File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/dag_processing.py", line 707, in end
reap_process_group(self._process.pid, log=self.log) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/helpers.py", line 324, in reap_process_group
signal_procs(sig) File "/Users/mac/Workspace/airflow/airflow_venv/lib/python3.8/site-packages/airflow/utils/helpers.py", line 293, in signal_procs
os.killpg(pgid, sig)
TypeError: an integer is required (got type NoneType)
EDIT: Python 3.8 is supported now https://github.com/apache/airflow#requirements. So this answer might not be relevant now.
This due to the Python version you are using. Airflow doesn't support Python 3.8 yet https://github.com/apache/airflow#stable-version-1109.
Downgrade your Python to 3.7 and check.
Maybe there are some compatibility problems?
Using Python 3.6.10 and airflow v1.10.4, I can get airflow running. Maybe you could try some other versions?
This worked for me!
1- Make sure you are using the correct celery version that supports your other packages like RabbitMQ ( as V5 doesn't support AMQP in its usual format), my advice is to use V4.6.X
2-THIS HAS NOTHING TO DO WITH PYTHON VERSION IF YOU ARE USING AIRFLOW V2.0
3- simply make yourself happy with airflow db reset (command may differ if you are using airflow Version X<2.0 )
4- Avoid deleting any dag like you delete a file and use airflow dag ... commands to do so. (it makes up a mess in your environment that you wont like, trust me on this..)
Wish you luck bearing python stuff..
This question already has answers here:
Jupyter Notebook with Python 3.8 - NotImplementedError
(4 answers)
Closed 3 years ago.
Here is the detailed error when launching jupyter notebook with python version 3.8
File "c:\users\kokat\appdata\local\programs\python\python38\lib\runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\kokat\appdata\local\programs\python\python38\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\kokat\AppData\Local\Programs\Python\Python38\Scripts\jupyter-notebook.EXE\__main__.py", line 9, in <module>
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\jupyter_core\application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\traitlets\config\application.py", line 657, in launch_instance
app.initialize(argv)
File "<c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\decorator.py:decorator-gen-7>", line 2, in initialize
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\notebook\notebookapp.py", line 1628, in initialize
self.init_webapp()
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\notebook\notebookapp.py", line 1407, in init_webapp
self.http_server.listen(port, self.ip)
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\tornado\tcpserver.py", line 144, in listen
self.add_sockets(sockets)
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\tornado\tcpserver.py", line 157, in add_sockets
self._handlers[sock.fileno()] = add_accept_handler(
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\tornado\netutil.py", line 268, in add_accept_handler
io_loop.add_handler(sock, accept_handler, IOLoop.READ)
File "c:\users\kokat\appdata\local\programs\python\python38\lib\site-packages\tornado\platform\asyncio.py", line 79, in add_handler
self.asyncio_loop.add_reader(
File "c:\users\kokat\appdata\local\programs\python\python38\lib\asyncio\events.py", line 498, in add_reader
raise NotImplementedError
NotImplementedError
Any helps would be appreciated.
Following one of the comments in this thread, I've solved the problem by:
Locate and open the asyncio.py file. In my case it's in C:\Users\[USERNAME]\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\tornado\platform\
After the line import asyncio add the following:
from sys import platform
if platform == "win32":
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) # python-3.8.0a4
This will force using SelectorEventLoop on Windows.
Jupyter notebook runs as a tornado web server. Your browser connect to this tornado server via a socket.
The I/O of the socket are handled by tornado's asyncio, which relies on the add_reader implementation of the native asyncio module of python runtime. As pointed out in the documentation of asyncio, this method is only supported with Windows SelectorEventLoop so make sure you are using this kind of event loop in your python installation. To find out which eventloop implementation is in usage, you can run the following commands in python shell:
import asyncio
print(asyncio.get_event_loop().__class__)
# Output: <class 'asyncio.windows_events._WindowsSelectorEventLoop'>
There is an ongoing discussion about allowing user to change the EventLoopPolicy of asyncio in jupyter's configuration file.