New to ElasticSearch. I was following this guide to get things set up: https://john.soban.ski/boto3-ec2-to-amazon-elasticsearch.html
I ran the "connect_to_es.py" script there, and oddly it worked the first time, but in a subsequent runs, it started throwing this error:
Traceback (most recent call last):
File "../connect_to_es.py", line 21, in <module>
print(json.dumps(es.info(), indent=4, sort_keys=True))
File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 294, in info
return self.transport.perform_request(
File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 413, in perform_request
_ProductChecker.raise_error(self._verified_elasticsearch)
File "/home/ubuntu/projects/.venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 630, in raise_error
raise UnsupportedProductError(message)
elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch
The elasticsearch python library version I have is 7.14, and my elasticsearch on AWS is running 7.10. Any thoughts on what's going on here?
Copy of code:
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3
import json
host = '<url>.us-east-1.es.amazonaws.com'
region = 'us-east-1'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
es = Elasticsearch(
hosts = [{'host': host, 'port': 443}],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
print(json.dumps(es.info(), indent=4, sort_keys=True))
Seems like downgrading fixed it pip3 install 'elasticsearch<7.14.0'
New elasticsearch-js has an issue:
The new product version check rejects oss distributions?
Downgrading it to lower version (e.g. 7.13) should help.
As some of the other answers indicate, you can downgrade right now but opensearch-py is a better long term solution
It should be a drop-in replacement for elasticsearch-py and it will be updated and patched over time. It supports OSS Elasticsearch and OpenSearch.
this error occurs because of the version conflict. Version of elasticsearch python library and elasticsearch should be the same.
In my case, elasticsearch version was 7.10 on AWS and I was using elasticsearch python library version 7.15 with my Django project. I removed it and installed new python library with version 7.10 in Django project and it worked fine for me.
I fixed the error by making following changes in Gemfile -
I changed -
gem 'elasticsearch'
to -
gem 'elasticsearch', '~> 7.1'
Ideally, I downgraded from 7.18(current version as of today) to 7.1
Related
i am currently trying to get Pyflink running with the AWS-Kinesis-SQL-Connector.
A use the TableAPI and can read from Kinesis and also write back to another Kinesis Stream. As soon as i use a udtf decorated function i get the following exception:
File "/home/user/anaconda3/envs/flink-env/lib/python3.8/site-packages/pyflink/table/table_environment.py", line 828, in execute_sql
return TableResult(self._j_tenv.executeSql(stmt))
File "/home/user/anaconda3/envs/flink-env/lib/python3.8/site-packages/py4j/java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "/home/user/anaconda3/envs/flink-env/lib/python3.8/site-packages/pyflink/util/exceptions.py", line 158, in deco
raise java_exception
pyflink.util.exceptions.TableException: org.apache.flink.table.api.TableException: Unsupported Python SqlFunction CAST.
I try to sum up the core snippets of the script:
#udtf(result_types=[DataTypes.STRING(), DataTypes.INT()])
def flatten_row(row: Row) -> Row:
for s in row["members"]:
yield Row(str(s["id"]), s["name"])
result_table = input_table.flat_map(flatten_row).alias("id", "name")
table_env.create_temporary_view("result_table", result_table)
As soon as i want to execute it on the Stream the exception get's raised.
table_result = table_env.execute_sql(f"INSERT INTO {output_table_name} SELECT * FROM result_table")
The output_table and input_table are connected to Kinesis Streams and without the udtf function it works.
Environment
Used apache-flink==1.16.0 and python3.8. Tried Conda and PIP environments
Thank you!
Already tried different versions of the apache-flink and the amazon-kinesis-sql-connector. Conda and PIP environments with Python3.8.
Finally i found out that the problem was the JDK version pre-installed on my MacOS. I downgraded from 15.0.2 until i reached 11.0.16, which was finally working without any error. So it seems that the Python apache-flink package needs older JDK versions.
This question just recently posted has some useful answers, but it's not the same as mine. I'm running urllib3 1.26.4 and Python 3.7 from an ArcGIS Pro Notebook. I also have Fiddler 4 open because I want to track web traffic while troubleshooting a script. I only get the following error when I have Fiddler open. If I close Fiddler I get <Response [200]>. Is it not possible to use the requests module with Fiddler open? I'm new to Fiddler.
Truncated script:
import requests
#url
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'
#headers
headers = {'user-agent': 'Mozilla/5.0'}
response = requests.get(idph_data, headers=headers, verify=True)
Error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
In [35]:
Line 4: response = requests.get(idph_data,verify=True)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py, in get:
Line 76: return request('get', url, params=params, **kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py, in request:
Line 61: return session.request(method=method, url=url, **kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py, in request:
Line 542: resp = self.send(prep, **send_kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py, in send:
Line 655: r = adapter.send(request, **kwargs)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\adapters.py, in send:
Line 449: timeout=timeout
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connectionpool.py, in urlopen:
Line 696: self._prepare_proxy(conn)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connectionpool.py, in _prepare_proxy:
Line 964: conn.connect()
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connection.py, in connect:
Line 359: conn = self._connect_tls_proxy(hostname, conn)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\connection.py, in _connect_tls_proxy:
Line 506: ssl_context=ssl_context,
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\util\ssl_.py, in ssl_wrap_socket:
Line 432: ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\urllib3\util\ssl_.py, in _ssl_wrap_socket_impl:
Line 474: return ssl_context.wrap_socket(sock)
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py, in wrap_socket:
Line 423: session=session
File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\ssl.py, in _create:
Line 827: raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname
---------------------------------------------------------------------------
I am running into this issue as well with the environment provided by the current version of ArcGIS Pro. Per a lower-rated answer in the question you linked, I ran pip install urllib3==1.25.11 in the desired environment (in my case a clone of the default), and the issue appears to be resolved.
This is apparently due to a new feature in the urllib3 version provided by ArcGIS Pro. The above command downgrades to a relatively recent, but working, version. This will not be resolved in newer versions of urllib3, but instead, there is currently a pull request pending to fix the underlying issue in Python itself.
By the way, while it's possible to configure pip to be able to run through the fiddler proxy, it's not too easy, so it is best to turn off Fiddler while running any pip commands.
The pertinent bug report is found here. The issue appears to be that there is a very old bug in how Windows system proxy settings are being parsed by CPython / built-in urllib, causing the proxy entry for use with https URLs to always receive a HTTPS prefix (instead of HTTP). Newer version of urllib3 actually support using proxies over HTTPS, which was not previously the case. So before, urllib3 would ignore the prefix, but now, it attempts to use HTTPS to communicate with a HTTP url.
I've updated to requests v. 2.7.0, the latest, and I'm no longer receiving the error. If it was a version-specific issue related to v. 2.25.1, which was what I was using, I'm not sure. I haven't came across any evidence of that.
In a Windows command prompt in the same directory as my Python executable:
python -m pip install requests==2.7.0
Now if I run my original script with Fiddler capturing, I get a HTTP status of 200 and my script no longer gives me the error.
EDIT: I believe this traceback stems from some sort of issue with the dependencies. Using pip to upgrade the packages didn't work but I created a new folder and installed from scratch and that worked out
I'm a Python novice so I'm struggling to debug an AWS Lambda I'm writing.
I've narrowed it down to this line of code s3_client = botoSession.resource('s3'), which is giving a long traceback with Syntax Error: invalid syntax. The botoSession variable is just for the credentials - botoSession = boto3.session.Session(aws_access_token, aws_secret_access_token).
I've also tried s3_client = boto3.client('s3'), s3_client = boto3.resource('s3'), s3_client = botoSession.resource('s3').
When I used botoSession.client('ses', region) I had no issues sending emails.
I found Error: client = boto3.client('s3') | AWS Elastic Beanstalk Worker Environment which appeared to be a similar issue, but it appeared to be fairly old and I wasn't able to figure out what the solution was. I tried adding
import sys
sys.path = [p for p in sys.path if not p.endswith('futures-3.0.3-py3.4.egg')]
to the top of my file which didn't seem to work.
The entire traceback is as follows:
Traceback (most recent call last):
File "smartsheetExporter.py", line 45, in <module>
s3_client = botoSession.resource('s3')
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/session.py", line 389, in resource
aws_session_token=aws_session_token, config=config)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/session.py", line 836, in create_client
client_config=config, api_version=api_version)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/client.py", line 65, in create_client
cls = self._create_client_class(service_name, service_model)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/client.py", line 90, in _create_client_class
base_classes=bases)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/utils.py", line 61, in _handler
module = import_module(module)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/utils.py", line 52, in import_module
__import__(name)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/s3/inject.py", line 15, in <module>
from boto3.s3.transfer import create_transfer_manager
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/s3/transfer.py", line 127, in <module>
from s3transfer.exceptions import RetriesExceededError as \
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/s3transfer/__init__.py", line 134, in <module>
import concurrent.futures
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/concurrent/futures/_base.py", line 381
raise exception_type, self._exception, self._traceback
^
SyntaxError: invalid syntax
Whenever strange things are happening, it's always a good idea to update things:
sudo pip install pip --upgrade
sudo pip install boto --upgrade
sudo pip install boto3 --upgrade
sudo pip install awscli --upgrade
If you're using Python 3, try pip3 instead of pip.
I just had this same issue with boto3 and ended up having to downgrade the Python version that my lambda was running from Python 3.6 to Python 2.7. If you're using Serverless Framework for this, your serverless.yml file looks like this.
provider:
name: aws
runtime: python3.6
memorySize: 3008
cool_function:
name: cool-function
description: This lambda goes and performs magic.
handler: cool_function.lambda_handler
runtime: python2.7
- schedule:
rate: rate(4 hours)
timeout: 180
I had the same problem with python 3.6 and AWS Lambda.
I found another answer that helped me here.
You should use futures==2.2.0
If you are running your code on an Amazon EC2 instance with a Role assigned to the instance, then you only need this:
import boto3
s3_client = boto3.client('s3')
s3_resource = boto3.resource('s3') # Pick whichever is wish to use
If you are not on an Amazon EC2 instance, this works:
import boto3
session = boto3.Session(aws_access_key_id='AKIAxxx',aws_secret_access_key='yyy')
s3_client = session.client('s3')
s3_resource = session.resource('s3')
Of course, you should never put your credentials in the code file. Instead, put them in a credentials file (easiest is via aws configure) or in Environment Variables. That way, they won't be copied into any code repository.
See: Boto3 credentials
After converting our app to Python 2.7, configuring for multithreading, and referencing mapreduce in app.yaml like this...
- url: /mapreduce(/.*)?
script: mapreduce.main.app
#script: google.appengine.ext.mapreduce.main.app
login: admin
and invoking mapreduce like this...
control.start_map(
"FNFR",
"fnfr.fnfrHandler",
"mapreduce.input_readers.BlobstoreLineInputReader",
{"blob_keys": blobKey},
shard_count=32,
mapreduce_parameters={'done_callback': '/fnfrdone','blobKey': blobKey, 'userID':thisUserID})
we get the following stack trace...
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 189, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 241, in _LoadHandler
raise ImportError('%s has no attribute %s' % (handler, name))
ImportError: <module 'mapreduce.main' from '/base/data/home/apps/s~xxxxxxonline/2.361692533819432574/mapreduce/main.pyc'> has no attribute app
I found one SO reference ( How to migrate my app.yaml to 2.7? ) but as you can see from my yaml, I think I've tried all combinations to try to get it to resolve. Thanks.
This worked for me, but I'm still on a pretty old version of the SDK, I don't know if they fixed this:
- url: /mapreduce(/.*)?
script: mapreduce.main.APP
login: admin
so I installed neo4j on ArchLinux (AUR Link) and want to test it using python 3.2.
I am using python 3.2, Eclipse with Pydev.
I tried following code from the neo4j website, allthough I think it was still 2.7 python code and I tried to convert it to Python 3.2 code.
Here's the code:
import os
libpath = '/usr/share/java/neo4j'
os.environ['CLASSPATH'] = ';'.join( [ os.path.abspath(p) for p in
os.listdir(libpath)])
from neo4j import GraphDatabase
# Create a database
db = GraphDatabase('/home/USERNAME/.db/neo4j/HelloWorld')
# All write operations happen in a transaction
with db.transaction:
firstNode = db.node(name='Hello')
secondNode = db.node(name='world!')
# Create a relationship with type 'knows'
relationship = firstNode.knows(secondNode, name='graphy')
# Read operations can happen anywhere
message = ' '.join([firstNode['name'], relationship['name'], secondNode['name']])
print(message)
# Delete the data
with db.transaction:
firstNode.knows.single.delete()
firstNode.delete()
secondNode.delete()
# Always shut down your database when your application exits
db.shutdown()
But I get following error message:
Traceback (most recent call last):
File "/home/USERNAME/PATH/TO/src/neo4j-HelloWorld.py", line 12, in <module>
from neo4j import GraphDatabase
File "/usr/lib/python3.2/site-packages/neo4j_embedded-1.6-py3.2.egg/neo4j/__init__.py", line 29, in <module>
from neo4j.core import GraphDatabase, Direction, NotFoundException, BOTH, ANY, INCOMING, OUTGOING
File "/usr/lib/python3.2/site-packages/neo4j_embedded-1.6-py3.2.egg/neo4j/core.py", line 19, in <module>
from _backend import *
ImportError: No module named _backend
I just can't figure out what's wrong!
I tried to set the CLASSPATH as described here, but it doesn't change anything.
I would really appreciate any help!
Did you run the code through 2to3?
If not, I suggest you do.
I think the problem is that the relative import syntax changed in 3.x, see PEP328 for details.
e.g. the offending import in core.py should probably say from ._backend import *