autobahn.twisted.websocket opening handshake error 400 - python-3.x

Problem: when Bitfinex websocket API connection established application gets errors:
2018-02-07T18:51:52+0200 connecting once using transport type "websocket" over endpoint "tcp"
2018-02-07T18:51:52+0200 Starting factory <autobahn.twisted.websocket.WampWebSocketClientFactory object at 0x106708550>
2018-02-07T18:51:52+0200 failing WebSocket opening handshake ('WebSocket connection upgrade failed (400 - BadRequest)')
2018-02-07T18:51:52+0200 dropping connection to peer tcp4:104.16.173.181:443 with abort=True: WebSocket connection upgrade failed (400 - BadRequest)
2018-02-07T18:51:52+0200 component failed: ConnectionAborted: Connection was aborted locally, using.
2018-02-07T18:51:52+0200 Connection failed: ConnectionAborted: Connection was aborted locally, using.
2018-02-07T18:51:52+0200 Stopping factory <autobahn.twisted.websocket.WampWebSocketClientFactory object at 0x106708550>
Code below:
from autobahn.twisted.component import Component, run
import json
cmp_Bitfinex = Component(
transports=[
{
u'type': u'websocket',
u'url': u'wss://api.bitfinex.com/ws',
u'endpoint': {
u'type': u'tcp',
u'host': 'api.bitfinex.com',
u'port': 443,
},
u'options': {
u"open_handshake_timeout": 100,
}
}
],
realm=u"realm1",
)
#cmp_Bitfinex.on_join
def joined(self):
def onTicker(*args):
print("Ticker event received:", args)
try:
yield from self.subscribe(onTicker, 'ticker', options=json.dumps({
"event": "subscribe",
"channel": "ticker",
"pair": "BTCUSD",
"prec": "P0",
"freq": "F0"
}))
except Exception as e:
print("Could not subscribe to topic:", e)
#cmp_Bitfinex.on_connect
def connected(session, details):
print('Connected: {} \r\n {}'.format(session, details))
def main():
run([cmp_Bitfinex])
if __name__ == "__main__":
main()
I understand, that problem can be in data that application sends. But, I can't understand what exactly wrong in my code.
App uses python 3.6, latest autobahn and latest Twisted

AFAICT autobahn components are for programming with WAMP.
I think you should be using their websocket programming. Check this example

Related

RabbitMQ' pika handshaking fails when SSL is set

I am setting up the SSL layer on RabbitMQ on both server and clients. But the clients are failing when creating the connection to the server. At this point I am running the RabbitMQ server on a docker locally and the client locally using a conda environment.
Once the RabbitMQ server is up I see that the secure connection is accepting incoming connections:
test-rabbitmq-1 | 2023-01-20 08:22:01.692731+00:00 [info] <0.726.0> started TCP listener on [::]:5672
test-rabbitmq-1 | 2023-01-20 08:22:01.694836+00:00 [info] <0.746.0> started TLS (SSL) listener on [::]:7575
But the client refuses to connect with:
(rabbitmq-test) ➜ RabbitMQ-TSL ✗ python3 test.py
Enter PEM pass phrase: ********
INFO:pika.adapters.utils.connection_workflow:Pika version 1.3.1 connecting to ('127.0.0.1', 7575)
INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 43142), raddr=('127.0.0.1', 7575)>
ERROR:pika.adapters.utils.io_services_utils:SSL do_handshake failed: error=SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 43142), raddr=('127.0.0.1', 7575)>
Traceback (most recent call last):
File "/.../.local/lib/python3.10/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)
ERROR:pika.adapters.utils.connection_workflow:Attempt to create the streaming transport failed: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); 'localhost'/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 7575)); ssl=True
ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)')
ERROR:pika.adapters.utils.connection_workflow:AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); first exception - None.
ERROR:pika.adapters.utils.connection_workflow:AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); first exception - None
ERROR:pika.adapters.blocking_connection:Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); first exception - None
ERROR:pika.adapters.blocking_connection:Error in _create_connection().
Traceback (most recent call last):
File "/.../.local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "/.../.local/lib/python3.10/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)
Traceback (most recent call last):
File "/.../test.py", line 16, in <module>
with pika.BlockingConnection(conn_params) as conn:
File "/.../.local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "/.../.local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "/.../.local/lib/python3.10/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)
Any idea of what I am not setting properly on pika or at the RabbitMQ server?
On the server side I set the SSL layer at rabbitmq.conf as:
# Enable AMQPS
listeners.ssl.default = 7575
ssl_options.cacertfile = /etc/rabbitmq/cer/ca_certificate.pem
ssl_options.certfile = /etc/rabbitmq/cer/server_certificate.pem
ssl_options.keyfile = /etc/rabbitmq/cer/server_key.pem
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = true
# Enable HTTPS
management.listener.port = 15671
management.listener.ssl = true
management.listener.ssl_opts.cacertfile = /etc/rabbitmq/cer/ca_certificate.pem
management.listener.ssl_opts.certfile = /etc/rabbitmq/cer/server_certificate.pem
management.listener.ssl_opts.keyfile = /etc/rabbitmq/cer/server_key.pem
The docker compose file contains:
version: '3.8'
services:
rabbitmq:
image: rabbitmq:3-management
hostname: rabbitmq-server
volumes:
- ./rabbitmq-config/rabbitmq-cert:/etc/rabbitmq/cer
- ./rabbitmq-config/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
- ./rabbitmq/data:/var/lib/rabbitmq/mnesia/rabbit#my-rabbit
- ./rabbitmq/logs:/var/log/rabbitmq/log
ports:
- 5672:5672
- 7575:7575
- 15672:15672
Then, and to simply, I am using the demo client from RabbitMQ's guide as:
import logging
import pika
import ssl
from pika.credentials import ExternalCredentials
logging.basicConfig(level=logging.INFO)
context = ssl.create_default_context(
cafile = '/rabbitmq-config/rabbitmq-cert/ca_certificate.pem'
)
context.load_cert_chain(
'/rabbitmq-config/rabbitmq-cert/client_certificate.pem',
'/rabbitmq-config/rabbitmq-cert/client_key.pem'
)
ssl_options = pika.SSLOptions(context, "localhost")
conn_params = pika.ConnectionParameters(
port = 7575,
ssl_options = ssl_options,
credentials = ExternalCredentials()
)
with pika.BlockingConnection(conn_params) as conn:
ch = conn.channel()
ch.queue_declare("foobar")
ch.basic_publish("", "foobar", "Hello, world!")
print(ch.basic_get("foobar"))
Your Python code is set up to do X509 certificate authentication (you're not using username/password but are using ExternalCredentials). However, you have not configured RabbitMQ to accept X509 certificates for authentication (docs).
The Pika docs need to be updated, so I opened this issue - https://github.com/pika/pika/issues/1413
You would probably get a hint of this if you look at the RabbitMQ log file at the time your Python client tries to connect.
In order to enable X509 certificate authentication, do the following:
rabbitmq-plugins enable rabbitmq_auth_mechanism_ssl (docs)
Edit your rabbitmq.conf file and add the following section:
auth_mechanisms.1 = PLAIN
auth_mechanisms.1 = AMQPLAIN
auth_mechanisms.1 = EXTERNAL
Restart RabbitMQ
Add a password-less user that exactly matches the CN= value in your client certificate: rabbitmqctl add_user 'foobar, O=baz'. You can also get this value by attempting to connect after making the above changes. The failed auth attempt will be logged.
If you continue to have problems, I suggest asking on the mailing list as this is not a good forum for detailed analysis.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

urllib3 proxy exception handling

Errors
Urllib3 Exception Given 8 Errors. I need To Handle This Errors.
Error No 1
it given OSError. why oserror given?
Error No 2
it given ssl certificate error, but i pass the ssl certificate to proxymanager(below my code)?
Error No 3
New connection stablish error? but my internet conectiomn working fine?why stablish error given?
More Errors Below
1 ('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 VirusFound'))HTTPSConnectionPool(host='api.ipify.org', port=443)
2 (SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)HTTPSConnectionPool(host='api.ipify.org', port=443)/
3 (NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1aa0334cd0>: Failed to establish a new connection: [Errno 110] Connection timed out'))HTTPSConnectionPool(host='api.ipify.org', port=443)
4 ('Cannot connect to proxy.', OSError('Tunnel connection failed: 307 Temporary Redirect'))HTTPSConnectionPool(host='api.ipify.org', port=443)
5 (EOF occurred in violation of protocol (_ssl.c:1131)HTTPSConnectionPool(host='api.ipify.org', port=443)
6 ('Cannot connect to proxy.', ConnectionResetError(104, 'Connection reset by peer'))HTTPSConnectionPool(host='api.ipify.org', port=443)
7 ('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1aa20a5430>: Failed to establish a new connection: [Errno 110] Connection timed out'))HTTPSConnectionPool(host='api.ipify.org', port=443)
8 ('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)')))HTTPSConnectionPool(host='api.ipify.org', port=443)
i have a many proxies. i send url api.ipify.org to request after some proxies given above 8 errors? how to handle this 8 errors?i need to except it. how to do it?
My Code is Below
global gcontext
gcontext = ssl.SSLContext()
def check_url_thread(in_proxy,urls):
global checked_url
headers = {
'User-Agent': random.choice(var.ua),
'Accept': 'text/html',
'Accept-Encoding': 'gzip;q=0,deflate;q=0',
'Connection': 'Keep-Alive',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Upgrade-Insecure-Requests': '1',
}
proxy = urllib3.ProxyManager('http://'+in_proxy,proxy_ssl_context = gcontext,headers =headers)
sys.stdout.write('| Cheking : '+urls+' | '+in_proxy+' | ')
try:
res = proxy.request('GET', url)
#checked_url = False
sys.stdout.write('\n| > '+'Success My proxy : '+str(in_proxy)+'| response : '+str(res.data.decode('utf-8'))+ ' | '+str(res.status))
except urllib3.exceptions.NewConnectionError as e:
sys.stdout.write('\n| > '+'Error New Connection Error : '+str(e))
except urllib3.exceptions.ProxySchemeUnknown as e:
sys.stdout.write('\n| > '+'Error proxy scheme unknown : '+str(e.scheme))
except urllib3.exceptions.ProxyError as e:
sys.stdout.write('\n| > '+'Error proxy error : '+str(e))
except urllib3.exceptions.MaxRetryError as e:
sys.stdout.write('\n| > '+'Error max retry error : '+str(e.reason)+str(e.pool)+str(e.url))
except urllib3.exceptions.TimeoutError as e:
sys.stdout.write('\n| > '+'Error Timeout Error : '+str(e))
except Exception as e:
sys.stdout.write('\n| > '+str(in_proxy)+str(e))
proxy.clear()
in_file = open('proxy.txt',"r")
for i in in_file:
lista_proxy_data.append(i.split("/n")[0])
in_file.close()
url = 'https://api.ipify.org'
for in_proxy in lista_proxy_data: // lista_proxy_data is proxy list array
try:
th = threading.Thread(target=check_url_thread,args=(in_proxy,url))
th.setDaemon(True)
th.start()
time.sleep(1)
except:
pass

How to setup a distributed eventbus in a vertx Hazelcast Cluster?

Here is the sender verticle
I have set multicast enabled and set the public host to my machines ip address
VertxOptions options = new VertxOptions()
.setClusterManager(ClusterManagerConfig.getClusterManager());
EventBusOptions eventBusOptions = new EventBusOptions()
.setClustered(true)
.setClusterPublicHost("10.10.1.160");
options.setEventBusOptions(eventBusOptions);
Vertx.clusteredVertx(options, res -> {
if (res.succeeded()) {
Vertx vertx = res.result();
vertx.deployVerticle(new requestHandler());
vertx.deployVerticle(new requestSender());
EventBus eventBus = vertx.eventBus();
eventBus.send("some.address","hello",reply -> {
System.out.println(reply.toString());
});
} else {
LOGGER.info("Failed: " + res.cause());
}
});
}
here's the reciever verticle
VertxOptions options = new VertxOptions().setClusterManager(mgr);
options.setEventBusOptions(new EventBusOptions()
.setClustered(true)
.setClusterPublicHost("10.10.1.174") );
Vertx.clusteredVertx(options, res -> {
if (res.succeeded()) {
Vertx vertx1 = res.result();
System.out.println("Success");
EventBus eb = vertx1.eventBus();
System.out.println("ready");
eb.consumer("some.address", message -> {
message.reply("hello hello");
});
} else {
System.out.println("Failed");
}
});
I get this result when i run both main verticles , so the verticles are detected by hazelcast and a connection is established
INFO: [10.10.1.160]:33001 [dev] [3.10.5] Established socket connection between /10.10.1.160:33001 and /10.10.1.174:35725
Jan 11, 2021 11:45:10 AM com.hazelcast.internal.cluster.ClusterService
INFO: [10.10.1.160]:33001 [dev] [3.10.5]
Members {size:2, ver:2} [
Member [10.10.1.160]:33001 - 51b8c249-6b3c-4ca8-a238-c651845629d8 this
Member [10.10.1.174]:33001 - 1cba1680-025e-469f-bad6-884111313672
]
Jan 11, 2021 11:45:10 AM com.hazelcast.internal.partition.impl.MigrationManager
INFO: [10.10.1.160]:33001 [dev] [3.10.5] Re-partitioning cluster data... Migration queue size: 271
Jan 11, 2021 11:45:11 AM com.hazelcast.nio.tcp.TcpIpAcceptor
But when the event-bus tries to send a message to given address i encounter this error is this a problem with event-bus configuration?
Jan 11, 2021 11:59:57 AM io.vertx.core.eventbus.impl.clustered.ConnectionHolder
WARNING: Connecting to server 10.10.1.174:39561 failed
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.10.1.174:39561
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:665)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
... 11 more
In Vert.x 3, the cluster host and cluster public host default to localhost.
If you only change the cluster public host in VertxOptions, Vert.x will bind EventBus transport servers to localhost while telling other nodes to connect to the public host.
This kind of configuration is needed when running Vert.x on some cloud providers, but in most cases you only need to set the cluster host (and then the public host will default to its value):
EventBusOptions eventBusOptions = new EventBusOptions()
.setClustered(true)
.setHost("10.10.1.160");

How can I connect a Wordpress service to a MariaDB service through Consul Connect?

I'm having a serious problem with connecting to MariaDB through Consul Connect.
I'm using Nomad to create the services with the proxies, with the following job definition:
job "wordpress" {
type = "service"
datacenters = ["dc1"]
group "server" {
network {
mode = "bridge"
port "http" {
static = 8080
to = 80
}
}
task "server" {
driver = "docker"
config {
image = "wordpress"
}
env {
WORDPRESS_DB_HOST = "${NOMAD_UPSTREAM_ADDR_database}"
WORDPRESS_DB_USER = "exampleuser"
WORDPRESS_DB_PASSWORD = "examplepass"
WORDPRESS_DB_NAME = "exampledb"
}
resources {
cpu = 100
memory = 64
network {
mbits = 10
}
}
}
service {
name = "wordpress"
tags = ["production", "wordpress"]
port = "http"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "database"
local_bind_port = 3306
}
}
}
}
}
}
group "database" {
network {
mode = "bridge"
port "db" {
to = 3306
}
}
task "database" {
driver = "docker"
config {
image = "mariadb"
}
env {
MYSQL_RANDOM_ROOT_PASSWORD = "yes"
MYSQL_INITDB_SKIP_TZINFO = "yes"
MYSQL_DATABASE = "exampledb"
MYSQL_USER = "exampleuser"
MYSQL_PASSWORD = "examplepass"
}
resources {
cpu = 100
memory = 128
network {
mbits = 10
}
}
}
service {
name = "database"
tags = ["production", "mariadb"]
port = "db"
connect {
sidecar_service {}
}
}
}
}
However, it seems that the server can't reach the database.
MySQL Connection Error: (2006) MySQL server has gone away
[25-Aug-2020 10:46:53 UTC] PHP Warning: mysqli::__construct(): Error while reading greeting packet. PID=187 in Standard input code on line 22
[25-Aug-2020 10:46:53 UTC] PHP Warning: mysqli::__construct(): (HY000/2006): MySQL server has gone away in Standard input code on line 22
MySQL Connection Error: (2006) MySQL server has gone away
WARNING: unable to establish a database connection to '127.0.0.1:3306'
continuing anyways (which might have unexpected results)
And the logs of the server and database proxies shows that some sort of TLS issue is happening, but I've got no clue how to solve this problem.
Server Proxy Logs
[2020-08-25 12:20:35.841][18][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:344] [C1229] Creating connection to cluster database.default.dc1.internal.0198bec5-d0b4-332c-973e-372808379192.consul
[2020-08-25 12:20:35.841][18][debug][pool] [source/common/tcp/conn_pool.cc:82] creating a new connection
[2020-08-25 12:20:35.841][18][debug][pool] [source/common/tcp/conn_pool.cc:362] [C1230] connecting
[2020-08-25 12:20:35.841][18][debug][connection] [source/common/network/connection_impl.cc:704] [C1230] connecting to 172.29.168.233:29307
[2020-08-25 12:20:35.841][18][debug][connection] [source/common/network/connection_impl.cc:713] [C1230] connection in progress
[2020-08-25 12:20:35.841][18][debug][pool] [source/common/tcp/conn_pool.cc:108] queueing request due to no available connections
[2020-08-25 12:20:35.841][18][debug][main] [source/server/connection_handler_impl.cc:280] [C1229] new connection
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:458] [C1229] socket event: 2
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:543] [C1229] write ready
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:458] [C1230] socket event: 2
[2020-08-25 12:20:35.841][18][trace][connection] [source/common/network/connection_impl.cc:543] [C1230] write ready
[2020-08-25 12:20:35.841][18][debug][connection] [source/common/network/connection_impl.cc:552] [C1230] connected
[2020-08-25 12:20:35.841][18][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C1230] handshake error: 2
[2020-08-25 12:20:35.842][18][trace][connection] [source/common/network/connection_impl.cc:458] [C1230] socket event: 3
[2020-08-25 12:20:35.842][18][trace][connection] [source/common/network/connection_impl.cc:543] [C1230] write ready
[2020-08-25 12:20:35.842][18][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C1230] handshake error: 5
[2020-08-25 12:20:35.842][18][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:201] [C1230]
[2020-08-25 12:20:35.842][18][debug][connection] [source/common/network/connection_impl.cc:190] [C1230] closing socket: 0
[2020-08-25 12:20:35.842][18][debug][pool] [source/common/tcp/conn_pool.cc:123] [C1230] client disconnected
Database Proxy Logs
[2020-08-25 12:26:07.093][15][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:201] [C927] new tcp proxy session
[2020-08-25 12:26:07.093][15][trace][connection] [source/common/network/connection_impl.cc:290] [C927] readDisable: enabled=true disable=true
[2020-08-25 12:26:07.093][15][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:344] [C927] Creating connection to cluster local_app
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:82] creating a new connection
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:362] [C928] connecting
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:704] [C928] connecting to 127.0.0.1:26344
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:713] [C928] connection in progress
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:108] queueing request due to no available connections
[2020-08-25 12:26:07.093][15][debug][main] [source/server/connection_handler_impl.cc:280] [C927] new connection
[2020-08-25 12:26:07.093][15][trace][connection] [source/common/network/connection_impl.cc:458] [C927] socket event: 2
[2020-08-25 12:26:07.093][15][trace][connection] [source/common/network/connection_impl.cc:543] [C927] write ready
[2020-08-25 12:26:07.093][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C927] handshake error: 5
[2020-08-25 12:26:07.093][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:201] [C927]
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:190] [C927] closing socket: 0
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:204] canceling pending request
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:212] canceling pending connection
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:101] [C928] closing data_to_write=0 type=1
[2020-08-25 12:26:07.093][15][debug][connection] [source/common/network/connection_impl.cc:190] [C928] closing socket: 1
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:123] [C928] client disconnected
[2020-08-25 12:26:07.093][15][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=1)
[2020-08-25 12:26:07.093][15][debug][main] [source/server/connection_handler_impl.cc:80] [C927] adding to cleanup list
[2020-08-25 12:26:07.093][15][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=2)
[2020-08-25 12:26:07.093][15][trace][main] [source/common/event/dispatcher_impl.cc:76] clearing deferred deletion list (size=2)
[2020-08-25 12:26:07.093][15][debug][pool] [source/common/tcp/conn_pool.cc:236] [C928] connection destroyed

Configure Pyspark AWS credentials within docker container

I'm using Docker to develop local AWS glue jobs with pyspark. The song_data.py file contains the AWS glue job. I configured the spark session with my AWS credentials although the errors below suggest otherwise. Within the file, I set up 4 different try statements using glue context methods to create a dynamic frame. Here's the glue job file (song_data.py):
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark import SQLContext
from pyspark.sql import SparkSession
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from configparser import ConfigParser
from pyspark import SparkConf
config = ConfigParser()
config.read_file(open('/app/config/aws.cfg'))
conf = (
SparkConf()
.set('spark.hadoop.fs.s3a.access.key', config.get('AWS', 'KEY'))
.set('spark.hadoop.fs.s3a.secret.key', config.get('AWS', 'SECRET'))
.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
)
sc = SparkContext(conf=conf)
spark = SparkSession(sc)
glueContext = GlueContext(spark)
conf_dict = spark.sparkContext.getConf().getAll()
print(conf_dict)
try:
print('Attempt 1: spark.read.json')
url = 's3a://sparkify-dend-analytics/song_data/A/A/A/TRAAAAW128F429D538.json'
spark.read.json(url).show(1)
except Exception as e:
print(e)
try:
print('Attempt 2: create_dynamic_frame.from_options')
song_df = glueContext.create_dynamic_frame.from_options(
connection_type='s3',
connection_options={"paths": [ "s3a://sparkify-dend-analytics/song_data/"]},
format='json')
print ('Count: ', song_df.count())
print('Schema: ')
song_df.printSchema()
except Exception as e:
print(e)
try:
print('Attempt 3: create_dynamic_frame.from_catalog')
song_df = glueContext.create_dynamic_frame.from_catalog(
database='sparkify',
table_name='song_data')
print ('Count: ', song_df.count())
print('Schema: ')
song_df.printSchema()
except Exception as e:
print(e)
try:
print('Attempt 4: create_dynamic_frame_from_catalog')
song_df = glueContext.create_dynamic_frame_from_catalog(
database='sparkify',
table_name='song_data')
print ('Count: ', song_df.count())
print('Schema: ')
song_df.printSchema()
except Exception as e:
print(e)
The command I use to run the glue job is: gluesparksubmit glue_etl_scripts/song_data.py --JOB-NAME test. Here are the short versions of the error outputs for each try statement:
Attempt 1: spark.read.json()
WARN FileStreamSink: Error while looking for metadata directory.
An error occurred while calling o87.json.
: org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on sparkify-dend-analytics:
com.amazonaws.AmazonClientException: No AWS Credentials provided by
DefaultAWSCredentialsProviderChain : com.amazonaws.SdkClientException: Unable to load AWS
credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to
load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and
AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load
AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey),
WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName,
com.amazonaws.auth.profile.ProfileCredentialsProvider#401a5902: profile file cannot be null,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper#2b6e2cf9: Failed to connect to service
endpoint: ]: No AWS Credentials provided by DefaultAWSCredentialsProviderChain :
com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain:
[EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment
variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)),
SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties
(aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a
value for roleArn and roleSessionName,
com.amazonaws.auth.profile.ProfileCredentialsProvider#401a5902: profile file cannot be null,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper#2b6e2cf9: Failed to connect to service
endpoint: ]
Attempt 2: create_dynamic_frame.from_options()
WARN InstanceMetadataServiceResourceFetcher: Fail to retrieve token
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
....
Caused by: java.net.ConnectException: Connection refused (Connection refused)
....
An error occurred while calling o125.getDynamicFrame.
: org.apache.hadoop.fs.s3a.AWSClientIOException: (same AWSClientIOException as above)
.....
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain:
Attempt 3: create_dynamic_frame.from_catalog()
WARN InstanceMetadataServiceResourceFetcher: Fail to retrieve token
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
.....
Caused by: java.net.ConnectException: Connection refused (Connection refused)
Attempt 4: create_dynamic_frame_from_catalog()
Same as attempt 3
When I printed out the configuration dict for the spark session, the aws access and secret key were valid.
Here's the spark configuration dict printed from running spark.sparkContext.getConf().getAll():
[('spark.app.name', 'song_data.py'), ('spark.driver.host', '73d3647fdf5b'),
('spark.hadoop.fs.s3a.secret.key', 'xxxxxxx'), ('spark.submit.pyFiles', '/glue/aws-glue-
libs/PyGlue.zip'), ('spark.executor.id', 'driver'), ('spark.driver.extraClassPath', '/glue/aws-
glue-libs/jarsv1/*'), ('spark.app.id', 'local-1593063861647'), ('spark.driver.port', '40655'),
('spark.executor.extraClassPath', '/glue/aws-glue-libs/jarsv1/*'), ('spark.rdd.compress',
'True'), ('spark.hadoop.fs.s3a.access.key', 'xxxxxxx'), ('spark.files', 'file:///glue/aws-glue-
libs/PyGlue.zip'), ('spark.serializer.objectStreamReset', '100'), ('spark.master', 'local[*]'),
('spark.submit.deployMode', 'client'), ('fs.s3.impl', 'org.apache.hadoop.fs.s3a.S3AFileSystem')]
Let me know if the Dockerfile is needed or any other code.

Resources