How to close the client streams after they are disconnected? - multithreading

there's a simple chat program using the asyncio library of Python. The clients can communicate without ever being blocked by any operation. Also the server prints out an information to the console window whenever a client is connected/disconnected to the server. Besides, all activity of the clients are written into a log file by the server.
There's no apparent problem when I look at the console windows. One second later after a client is disconnected, the information about that client is written to the console window of server. However when I look at the log file, it is seen that, the server is trying to send empty messages to all connected clients. But these messages are not displayed in clients' windows.
So it seems, the server is not properly closing the disconnected clients. The reader and writer streams of the disconnected client seems active even the client is disconnected.
How can this problem be solved according to the codes I've shared? By the way I am sharing the log file too. Here are the codes:
server.py
#!/usr/bin/env python3.7
# -*- coding: utf-8 -*-
import asyncio
import logging
logging.basicConfig(
filename="server.log",
format="- %(levelname)s - %(asctime)s - %(message)s",
level=logging.DEBUG,
datefmt="%d.%m.%Y %H:%M:%S"
)
class Server:
def __init__(self):
self.clients = []
def run(self):
asyncio.run(self.main())
async def client_connected(self, reader, writer):
client = f"{writer.get_extra_info('peername')}"
print(f"{client} is connected.")
logging.info(f"{client} is connected.")
self.clients.append((writer, reader))
while True:
try:
data = await reader.readline()
except (BrokenPipeError, ConnectionResetError):
data = "".encode()
await asyncio.sleep(1)
for i in self.clients:
msg = f"{client}: {data.decode()}"
try:
i[0].write(msg.encode())
logging.debug(msg[:-1])
await i[0].drain()
except (BrokenPipeError, ConnectionResetError):
i[0].close()
self.clients.remove(i)
print(f"{client} is disconnected.")
logging.info(f"{client} is disconnected.")
await asyncio.sleep(2)
break
async def main(self):
server = await asyncio.start_server(
client_connected_cb=self.client_connected,
host="127.0.0.1",
port=12345
)
print(f"Server started on {server.sockets[0].getsockname()}")
logging.info(f"Server started on {server.sockets[0].getsockname()}")
async with server:
await server.wait_closed()
if __name__ == "__main__":
Server().run()
client.py
#!/usr/bin/env python3.7
# -*- coding: utf-8 -*-
import sys
import asyncio
import threading
class Client:
def __init__(self):
self.nick = input("/nick ")
while not self.nick:
self.nick = input("/nick ")
def run(self):
asyncio.run(self.main())
async def read(self, reader, writer):
data = await reader.readline()
socket = writer.get_extra_info('socket').getsockname()
if str(socket) not in data.decode() or \
self.nick not in data.decode():
if data.decode().count("('") >= 1:
data = data.decode().split(": ")[-2:]
print(": ".join(data)[:-1])
async def write(self, writer):
t = threading.Thread(
target=lambda: writer.write(
f"{self.nick}: {sys.stdin.readline()}".encode()
)
)
t.daemon = True
t.start()
t.join(0.1)
await writer.drain()
async def main(self):
reader, writer = await asyncio.open_connection("127.0.0.1", 12345)
print(f"Connected to {writer.get_extra_info('peername')}")
while reader and writer:
await asyncio.gather(
self.read(reader, writer),
self.write(writer)
)
if __name__ == "__main__":
Client().run()
server.log
- DEBUG - 30.09.2019 15:17:50 - Using selector: EpollSelector
- INFO - 30.09.2019 15:17:50 - Server started on ('127.0.0.1', 12345)
- INFO - 30.09.2019 15:17:56 - ('127.0.0.1', 45562) is connected.
- INFO - 30.09.2019 15:18:05 - ('127.0.0.1', 45564) is connected.
- INFO - 30.09.2019 15:18:21 - ('127.0.0.1', 45566) is connected.
- DEBUG - 30.09.2019 15:18:48 - ('127.0.0.1', 45566): Client3: Hello Clients!
- DEBUG - 30.09.2019 15:18:48 - ('127.0.0.1', 45566): Client3: Hello Clients!
- DEBUG - 30.09.2019 15:18:48 - ('127.0.0.1', 45566): Client3: Hello Clients!
- DEBUG - 30.09.2019 15:18:51 - ('127.0.0.1', 45566): Client3: How are you?
- DEBUG - 30.09.2019 15:18:51 - ('127.0.0.1', 45566): Client3: How are you?
- DEBUG - 30.09.2019 15:18:51 - ('127.0.0.1', 45566): Client3: How are you?
- DEBUG - 30.09.2019 15:18:53 - ('127.0.0.1', 45566):
- DEBUG - 30.09.2019 15:18:53 - ('127.0.0.1', 45566):
- DEBUG - 30.09.2019 15:18:53 - ('127.0.0.1', 45566):
- DEBUG - 30.09.2019 15:18:53 - ('127.0.0.1', 45566):
- DEBUG - 30.09.2019 15:18:53 - ('127.0.0.1', 45566):
- DEBUG - 30.09.2019 15:18:53 - ('127.0.0.1', 45566):
- INFO - 30.09.2019 15:18:53 - ('127.0.0.1', 45566) is disconnected.
- DEBUG - 30.09.2019 15:18:55 - ('127.0.0.1', 45564):
- DEBUG - 30.09.2019 15:18:55 - ('127.0.0.1', 45564):
- DEBUG - 30.09.2019 15:18:55 - ('127.0.0.1', 45564):
- DEBUG - 30.09.2019 15:18:55 - ('127.0.0.1', 45564):
- INFO - 30.09.2019 15:18:55 - ('127.0.0.1', 45564) is disconnected.
- DEBUG - 30.09.2019 15:18:56 - ('127.0.0.1', 45566):
- DEBUG - 30.09.2019 15:18:57 - ('127.0.0.1', 45562):
- DEBUG - 30.09.2019 15:18:57 - ('127.0.0.1', 45562):
- INFO - 30.09.2019 15:18:57 - ('127.0.0.1', 45562) is disconnected.

According to the documentation of StreamWriter.close, it turns out you need to call wait_closed too:
stream.close()
await stream.wait_closed()
In your server code:
i[0].close()
await i[0].wait_closed()

Related

RabbitMQ' pika handshaking fails when SSL is set

I am setting up the SSL layer on RabbitMQ on both server and clients. But the clients are failing when creating the connection to the server. At this point I am running the RabbitMQ server on a docker locally and the client locally using a conda environment.
Once the RabbitMQ server is up I see that the secure connection is accepting incoming connections:
test-rabbitmq-1 | 2023-01-20 08:22:01.692731+00:00 [info] <0.726.0> started TCP listener on [::]:5672
test-rabbitmq-1 | 2023-01-20 08:22:01.694836+00:00 [info] <0.746.0> started TLS (SSL) listener on [::]:7575
But the client refuses to connect with:
(rabbitmq-test) ➜ RabbitMQ-TSL ✗ python3 test.py
Enter PEM pass phrase: ********
INFO:pika.adapters.utils.connection_workflow:Pika version 1.3.1 connecting to ('127.0.0.1', 7575)
INFO:pika.adapters.utils.io_services_utils:Socket connected: <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 43142), raddr=('127.0.0.1', 7575)>
ERROR:pika.adapters.utils.io_services_utils:SSL do_handshake failed: error=SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 43142), raddr=('127.0.0.1', 7575)>
Traceback (most recent call last):
File "/.../.local/lib/python3.10/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)
ERROR:pika.adapters.utils.connection_workflow:Attempt to create the streaming transport failed: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); 'localhost'/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 7575)); ssl=True
ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)')
ERROR:pika.adapters.utils.connection_workflow:AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); first exception - None.
ERROR:pika.adapters.utils.connection_workflow:AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); first exception - None
ERROR:pika.adapters.blocking_connection:Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorTransportSetupError: SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:997)'); first exception - None
ERROR:pika.adapters.blocking_connection:Error in _create_connection().
Traceback (most recent call last):
File "/.../.local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "/.../.local/lib/python3.10/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)
Traceback (most recent call last):
File "/.../test.py", line 16, in <module>
with pika.BlockingConnection(conn_params) as conn:
File "/.../.local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "/.../.local/lib/python3.10/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "/.../.local/lib/python3.10/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)
Any idea of what I am not setting properly on pika or at the RabbitMQ server?
On the server side I set the SSL layer at rabbitmq.conf as:
# Enable AMQPS
listeners.ssl.default = 7575
ssl_options.cacertfile = /etc/rabbitmq/cer/ca_certificate.pem
ssl_options.certfile = /etc/rabbitmq/cer/server_certificate.pem
ssl_options.keyfile = /etc/rabbitmq/cer/server_key.pem
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = true
# Enable HTTPS
management.listener.port = 15671
management.listener.ssl = true
management.listener.ssl_opts.cacertfile = /etc/rabbitmq/cer/ca_certificate.pem
management.listener.ssl_opts.certfile = /etc/rabbitmq/cer/server_certificate.pem
management.listener.ssl_opts.keyfile = /etc/rabbitmq/cer/server_key.pem
The docker compose file contains:
version: '3.8'
services:
rabbitmq:
image: rabbitmq:3-management
hostname: rabbitmq-server
volumes:
- ./rabbitmq-config/rabbitmq-cert:/etc/rabbitmq/cer
- ./rabbitmq-config/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
- ./rabbitmq/data:/var/lib/rabbitmq/mnesia/rabbit#my-rabbit
- ./rabbitmq/logs:/var/log/rabbitmq/log
ports:
- 5672:5672
- 7575:7575
- 15672:15672
Then, and to simply, I am using the demo client from RabbitMQ's guide as:
import logging
import pika
import ssl
from pika.credentials import ExternalCredentials
logging.basicConfig(level=logging.INFO)
context = ssl.create_default_context(
cafile = '/rabbitmq-config/rabbitmq-cert/ca_certificate.pem'
)
context.load_cert_chain(
'/rabbitmq-config/rabbitmq-cert/client_certificate.pem',
'/rabbitmq-config/rabbitmq-cert/client_key.pem'
)
ssl_options = pika.SSLOptions(context, "localhost")
conn_params = pika.ConnectionParameters(
port = 7575,
ssl_options = ssl_options,
credentials = ExternalCredentials()
)
with pika.BlockingConnection(conn_params) as conn:
ch = conn.channel()
ch.queue_declare("foobar")
ch.basic_publish("", "foobar", "Hello, world!")
print(ch.basic_get("foobar"))
Your Python code is set up to do X509 certificate authentication (you're not using username/password but are using ExternalCredentials). However, you have not configured RabbitMQ to accept X509 certificates for authentication (docs).
The Pika docs need to be updated, so I opened this issue - https://github.com/pika/pika/issues/1413
You would probably get a hint of this if you look at the RabbitMQ log file at the time your Python client tries to connect.
In order to enable X509 certificate authentication, do the following:
rabbitmq-plugins enable rabbitmq_auth_mechanism_ssl (docs)
Edit your rabbitmq.conf file and add the following section:
auth_mechanisms.1 = PLAIN
auth_mechanisms.1 = AMQPLAIN
auth_mechanisms.1 = EXTERNAL
Restart RabbitMQ
Add a password-less user that exactly matches the CN= value in your client certificate: rabbitmqctl add_user 'foobar, O=baz'. You can also get this value by attempting to connect after making the above changes. The failed auth attempt will be logged.
If you continue to have problems, I suggest asking on the mailing list as this is not a good forum for detailed analysis.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

Error when try to Deploy flask API using Gunicorn3 and NGINX on a linux server

The code has a good execution when i've started
(MeasuresApi) vitor.marques#sbiapi-lnx01:~/SVLoad/MeasuresApi$ gunicorn3 --workers=3 app_newVersion:app
[2022-08-08 16:58:46 +0000] [22151] [INFO] Starting gunicorn 20.1.0
[2022-08-08 16:58:46 +0000] [22151] [INFO] Listening at: http://127.0.0.1:8000 (22151)
[2022-08-08 16:58:46 +0000] [22151] [INFO] Using worker: sync
[2022-08-08 16:58:46 +0000] [22152] [INFO] Booting worker with pid: 22152
[2022-08-08 16:58:46 +0000] [22153] [INFO] Booting worker with pid: 22153
[2022-08-08 16:58:46 +0000] [22154] [INFO] Booting worker with pid: 22154
But when i did some request in the web, i have this return
[2022-08-08 16:59:08 +0000] [22153] [ERROR] Error handling request /vehicles/consumption/totalconsumption?companyAliasCode=115&startDate=20220101&endDate=20220527&vehicleCode
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/gunicorn/workers/sync.py", line 136, in handle
self.handle_request(listener, req, client, addr)
File "/usr/lib/python3/dist-packages/gunicorn/workers/sync.py", line 179, in handle_request
respiter = self.wsgi(environ, resp.start_response)
TypeError: Application.call() takes 1 positional argument but 3 were given
[2022-08-08 16:59:08 +0000] [22154] [ERROR] Error handling request /favicon.ico
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/gunicorn/workers/sync.py", line 136, in handle
self.handle_request(listener, req, client, addr)
File "/usr/lib/python3/dist-packages/gunicorn/workers/sync.py", line 179, in handle_request
respiter = self.wsgi(environ, resp.start_response)
TypeError: Application.__call__() takes 1 positional argument but 3 were given

python aiosmtpd server with basic logging

I've been made aware that aiosmtpd logs to the syslog. I'm using a Red Hat Linux distribution and can't find anything related to my SMTP server in the messages or maillog file. I'm trying to debug an issue with a device that can't connect to my SMTP server with basic authentication as I can't find any reason why the device is being rejected by my server. The only way I've been able to debug so far is by using the EHLO and MAIL handlers and printing a message when that stage of the connection is reached. Ideally, I'd like as much as possible to be logged out, like with smtplib that enables you to see each message between the client and server. Is it possible to do this or some basic logging at least and how do I do it if so? The code I'm using is:
import email
from email.header import decode_header
from email import message_from_bytes
from email.policy import default
from aiosmtpd.controller import Controller
from aiosmtpd.smtp import LoginPassword, AuthResult
import os
import json
import re
import sys
import time
import signal
import logging
from datetime import datetime
import configparser
##setting timezone
os.environ['TZ'] = "Europe/London"
time.tzset()
#wildlifeCameraHome = os.getenv('WILDLIFE_CAMERA_HOME')
wildlifeCameraHome = "/home/matthew_gale/smtp-server"
startupConfigURL = "{}/Config/Config.ini".format(wildlifeCameraHome)
validCameraList = "{}/Config/ValidCameraIDs.txt".format(wildlifeCameraHome)
ouboxBaseURL = "{}/outbox".format(wildlifeCameraHome)
spacer = "*"*100
# Get command line parameters
if len( sys.argv ) > 1 and str( sys.argv[1] ) == "DEBUG":
debugMode = True
else:
debugMode = False
if not debugMode:
logFileURL = "{}/Logging/EmailExtractorLog.out".format(wildlifeCameraHome)
sys.stdout = open(logFileURL, 'a', 1)
sys.stderr = sys.stdout
if os.environ.get('VA_LOG_LEVEL') is None:
envlevel = 3
else:
envlevel = int(os.environ.get('VA_LOG_LEVEL'))
def Lprint(logstring, loglevel):
detailedtimeStamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
if loglevel <= envlevel or debugMode:
print(detailedtimeStamp + ":" + logstring)
return True
else:
return None
def onExit( sig, func=None):
Lprint("*************Stopping program*****************",3)
controller.stop()
exit()
signal.signal(signal.SIGTERM, onExit)
# removes the spaces and replaces with _ so they're valid folder names
def clean(text):
return "".join(c if c.isalnum() else "_" for c in text)
#get the configs from the config file
config = configparser.ConfigParser()
config.read(startupConfigURL)
gmailConfig = config['EmailExtractor']
validEmail = gmailConfig['validSender']
# check at the end if there's any validation regarding who sends the email
with open(validCameraList, 'r', encoding='utf-8') as f:
validCameraIDs = f.readlines()
for rowNumber, content in enumerate(validCameraIDs):
validCameraIDs[rowNumber] = content.replace("\n","")
Lprint("Valid cameraIDs are",3)
print (validCameraIDs)
auth_db = {
b"TestCamera1#gmail.com": b"password1",
b"user2": b"password2",
b"TestCamera1": b"password1",
}
def authenticator_func(server, session, envelope, mechanism, auth_data):
# Simple auth - is only being used because of the reolink cam
assert isinstance(auth_data, LoginPassword)
username = auth_data.login
password = auth_data.password
if auth_db.get(username) == password:
return AuthResult(success=True)
else:
return AuthResult(success=False, handled=False)
def configure_logging():
file_handler = logging.FileHandler("aiosmtpd.log", "a")
stderr_handler = logging.StreamHandler(sys.stderr)
logger = logging.getLogger("mail.log")
fmt = "[%(asctime)s %(levelname)s] %(message)s"
datefmt = None
formatter = logging.Formatter(fmt, datefmt, "%")
stderr_handler.setFormatter(stderr_handler)
logger.addHandler(stderr_handler)
file_handler.setFormatter(file_handler)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG)
class CustomHandler:
def handle_exception(self, error):
Lprint("exception occured",3)
print(error)
return '542 Internal Server Error'
async def handle_DATA(self, server, session, envelope):
peer = session.peer
data = envelope.content # type: bytes
msg = message_from_bytes(envelope.content, policy=default)
# decode the email subject
Lprint("Msg:{}".format(msg),3)
Lprint("Data:{}".format(data),3)
Lprint("All of the relevant data has been extracted from the email",3)
Lprint(spacer,3)
return '250 OK'
if __name__ == '__main__':
configure_logging()
handler = CustomHandler()
controller = Controller(handler, hostname='0.0.0.0', port=587, authenticator=authenticator_func, auth_required=True,auth_require_tls=False)
# Run the event loop in a separate thread.
controller.start()
#Confirmed that this is needed to keep the SMTP server running constantly
while True:
time.sleep(10)
If you search the aiosmtpd codebase for "logging.getLogger", you can find a few places where logging is being configured with Python's standard logging module.
In order to actually see these log messages, you need to configure the log level and add a log handler. Try calling the following "configure_logging" function early in your program. It will set up basic logging to stderr and to a file named "aiosmtpd.log". Complete example:
import logging
import sys
def configure_logging():
file_handler = logging.FileHandler("aiosmtpd.log", "a")
stderr_handler = logging.StreamHandler(sys.stderr)
logger = logging.getLogger("mail.log")
fmt = "[%(asctime)s %(levelname)s] %(message)s"
datefmt = None
formatter = logging.Formatter(fmt, datefmt, "%")
stderr_handler.setFormatter(formatter)
logger.addHandler(stderr_handler)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG)
configure_logging()
# aiosmtpd example from https://stackoverflow.com/a/43904837/1570972
import aiosmtpd.controller
class CustomSMTPHandler:
async def handle_DATA(self, server, session, envelope):
print(len(envelope.content), repr(envelope.content[:50]))
return '250 OK'
handler = CustomSMTPHandler()
server = aiosmtpd.controller.Controller(handler, hostname="127.0.0.1")
server.start()
input("Server started. Press Return to quit.\n")
server.stop()
Running the above script in one terminal and then running swaks --server '127.0.0.1:8025' --to foo#example.com in another (the Swiss Army Knife for SMTP) to send a test email gives the following output on the terminal:
[2021-12-07 19:37:57,124 INFO] Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin)
[2021-12-07 19:37:57,124 INFO] Peer: ('127.0.0.1', 44126)
[2021-12-07 19:37:57,125 INFO] ('127.0.0.1', 44126) handling connection
[2021-12-07 19:37:57,125 DEBUG] ('127.0.0.1', 44126) << b'220 alcyone.localdomain Python SMTP 1.4.2'
Server started. Press Return to quit.
[2021-12-07 19:37:57,126 INFO] ('127.0.0.1', 44126) EOF received
[2021-12-07 19:37:57,126 INFO] ('127.0.0.1', 44126) Connection lost during _handle_client()
[2021-12-07 19:37:57,126 INFO] ('127.0.0.1', 44126) connection lost
[2021-12-07 19:38:02,012 INFO] Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin)
[2021-12-07 19:38:02,012 INFO] Peer: ('127.0.0.1', 44128)
[2021-12-07 19:38:02,013 INFO] ('127.0.0.1', 44128) handling connection
[2021-12-07 19:38:02,013 DEBUG] ('127.0.0.1', 44128) << b'220 alcyone.localdomain Python SMTP 1.4.2'
[2021-12-07 19:38:02,013 DEBUG] _handle_client readline: b'EHLO alcyone.localdomain\r\n'
[2021-12-07 19:38:02,013 INFO] ('127.0.0.1', 44128) >> b'EHLO alcyone.localdomain'
[2021-12-07 19:38:02,013 DEBUG] ('127.0.0.1', 44128) << b'250-alcyone.localdomain'
[2021-12-07 19:38:02,013 DEBUG] ('127.0.0.1', 44128) << b'250-SIZE 33554432'
[2021-12-07 19:38:02,013 DEBUG] ('127.0.0.1', 44128) << b'250-8BITMIME'
[2021-12-07 19:38:02,013 DEBUG] ('127.0.0.1', 44128) << b'250-SMTPUTF8'
[2021-12-07 19:38:02,013 DEBUG] ('127.0.0.1', 44128) << b'250 HELP'
[2021-12-07 19:38:02,014 DEBUG] _handle_client readline: b'MAIL FROM:<rav#alcyone.localdomain>\r\n'
[2021-12-07 19:38:02,014 INFO] ('127.0.0.1', 44128) >> b'MAIL FROM:<rav#alcyone.localdomain>'
[2021-12-07 19:38:02,014 INFO] ('127.0.0.1', 44128) sender: rav#alcyone.localdomain
[2021-12-07 19:38:02,014 DEBUG] ('127.0.0.1', 44128) << b'250 OK'
[2021-12-07 19:38:02,014 DEBUG] _handle_client readline: b'RCPT TO:<foo#example.com>\r\n'
[2021-12-07 19:38:02,014 INFO] ('127.0.0.1', 44128) >> b'RCPT TO:<foo#example.com>'
[2021-12-07 19:38:02,014 INFO] ('127.0.0.1', 44128) recip: foo#example.com
[2021-12-07 19:38:02,014 DEBUG] ('127.0.0.1', 44128) << b'250 OK'
[2021-12-07 19:38:02,014 DEBUG] _handle_client readline: b'DATA\r\n'
[2021-12-07 19:38:02,014 INFO] ('127.0.0.1', 44128) >> b'DATA'
[2021-12-07 19:38:02,015 DEBUG] ('127.0.0.1', 44128) << b'354 End data with <CR><LF>.<CR><LF>'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'Date: Tue, 07 Dec 2021 19:38:02 +0100\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'To: foo#example.com\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'From: rav#alcyone.localdomain\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'Subject: test Tue, 07 Dec 2021 19:38:02 +0100\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'Message-Id: <20211207193802.024948#alcyone.localdomain>\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'X-Mailer: swaks vDEVRELEASE jetmore.org/john/code/swaks/\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'This is a test mailing\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'\r\n'
[2021-12-07 19:38:02,015 DEBUG] DATA readline: b'.\r\n'
283 b'Date: Tue, 07 Dec 2021 19:38:02 +0100\r\nTo: foo#exa'
[2021-12-07 19:38:02,015 DEBUG] ('127.0.0.1', 44128) << b'250 OK'
[2021-12-07 19:38:02,015 DEBUG] _handle_client readline: b'QUIT\r\n'
[2021-12-07 19:38:02,015 INFO] ('127.0.0.1', 44128) >> b'QUIT'
[2021-12-07 19:38:02,015 DEBUG] ('127.0.0.1', 44128) << b'221 Bye'
[2021-12-07 19:38:02,016 INFO] ('127.0.0.1', 44128) connection lost
[2021-12-07 19:38:02,016 INFO] ('127.0.0.1', 44128) Connection lost during _handle_client()

BitTorrent client: trouble downloading last few blocks from peers

The BitTorrent client I'm working on is almost working except it couldnt get the last few blocks from peers even though the requests have been sent. I get no data from peers except for keep-alive messages and peers would close their connection after sending a few keep-alive.
I know there is a end game mode, where there's a tendency for the last few blocks to trickle in slowly. But in my case, all peers would stop sending any data when there are a few blocks left and close their connection one by one.
What could be the problem that is causing this?
2016-07-19 15:05:27,131 - main.torrent_client - INFO - we have piece 69
2016-07-19 15:05:27,131 - main.torrent_client - INFO - downloaded: 71, total: 72
2016-07-19 15:05:27,132 - main.torrent_client - INFO - peer queue {70, 71}
2016-07-19 15:05:27,132 - main.torrent_client - INFO - 2 blocks left to request from 198.251.56.71
2016-07-19 15:05:27,132 - main.torrent_client - INFO - pop a block from piece 70
2016-07-19 15:05:27,132 - main.torrent_client - INFO - peer queue {71}
2016-07-19 15:05:27,132 - main.torrent_client - INFO - 1 blocks left to request from 198.251.56.71
2016-07-19 15:05:27,133 - main.torrent_client - INFO - pop a block from piece 71
2016-07-19 15:05:30,138 - main.torrent_client - INFO - requested {'index': 71, 'begin_offset': 0, 'request_length': 16384} from 198.251.56.71
2016-07-19 15:05:27,133 - main.pieces - INFO - Done requesting all the pieces!!!!!!!!!
2016-07-19 15:06:18,066 - main.torrent_client - INFO - just sent keep alive message to {'port': 57430, 'host': '198.251.56.71'}
2016-07-19 15:06:18,066 - main.torrent_client - INFO - just sent keep alive message to {'port': 29063, 'host': '38.76.93.8'}
2016-07-19 15:07:15,856 - main.torrent_client - DEBUG - Peer {'port': 29063, 'host': '38.76.93.8'} sent KEEP ALIVE message
2016-07-19 15:07:48,065 - main.torrent_client - INFO - just sent keep alive message to {'port': 57430, 'host': '198.251.56.71'}
2016-07-19 15:07:48,065 - main.torrent_client - INFO - just sent keep alive message to {'port': 29063, 'host': '38.76.93.8'}
2016-07-19 15:09:16,796 - main.torrent_client - DEBUG - Peer {'port': 29063, 'host': '38.76.93.8'} sent KEEP ALIVE message
2016-07-19 15:09:18,066 - main.torrent_client - INFO - just sent keep alive message to {'port': 57430, 'host': '198.251.56.71'}
2016-07-19 15:10:48,070 - main.torrent_client - INFO - just sent keep alive message to {'port': 29063, 'host': '38.76.93.8'}
2016-07-19 15:11:17,785 - main.torrent_client - DEBUG - Peer {'port': 29063, 'host': '38.76.93.8'} sent KEEP ALIVE message
2016-07-19 15:13:48,073 - main.torrent_client - DEBUG - connection closed by {'port': 57430, 'host': '198.251.56.71'}

Selenium Webdriver requires restarts to function consistently

My testing stack consists of the latest version of Selenium Server (2.33.0, aka selenium-server-standalone-2.33.0.jar), Mocha, Node.js, and PhantomJS.
My question regards the following code:
var webdriver = require('../../../lib/selenium/node_modules/selenium-webdriver/'),
driver = new webdriver.Builder().
withCapabilities({'browserName': 'phantomjs'}).
build();
driver.manage().timeouts().implicitlyWait(15000);
describe('Wordpress', function() {
it('should be able to log in', function(done) {
driver.get('http://#### REDACTED ####/wp-login.php');
driver.findElement(webdriver.By.css('#user_login')).sendKeys('#### REDACTED ####');
driver.findElement(webdriver.By.css('#user_pass')).sendKeys('#### REDACTED ####');
driver.findElement(webdriver.By.css('#wp-submit')).click();
// #wpwrap is an element on the Wordpress dashboard that is displayed once
// the user is logged in. By testing for its presence, we can determine
// if the login attempt succeeded.
driver.findElement(webdriver.By.css('#wpwrap')).then(function(v) {
done();
});
});
});
On my local system, OS X, the test runs fine consistently. However, once the test is uploaded to our CentOS server (where we hope to do Continuous Integration testing), the test behaves extremely strangely.
After Selenium Server is started, the test runs successfully once. From that point on, the test only succeeds every one out of ten times or so. Restarting Selenium Server guarantees that the test will run successfully. In fact, if Selenium Server is restarted every time the test is run, the test will succeed every time.
How can I get this test to succeed without restarting Selenium Server every time?
Thank you so much for your help! :)
UPDATE: In addition to the error log below, I'm also occasionally getting the following error:
Exception in thread "Thread-21" java.lang.OutOfMemoryError: unable to create new native thread
Details on the error messages follow below:
A successful test yields the following output from Mocha:
[s5rich#host ~]$ mocha test/selenium/acceptance/simple.js
Wordpress
✓ should be able to log in (2604ms)
1 passing (3 seconds)
A successful test also yields the following output from Selenium Server:
23:21:50.517 INFO - Executing: [new session: {browserName=phantomjs}] at URL: /session)
23:21:50.527 INFO - Creating a new session for Capabilities [{browserName=phantomjs}]
23:21:50.547 INFO - executable: /usr/local/bin/phantomjs
23:21:50.547 INFO - port: 26515
23:21:50.547 INFO - arguments: [--webdriver=26515, --webdriver-logfile=/home/s5rich/phantomjsdriver.log]
23:21:50.547 INFO - environment: {}
PhantomJS is launching GhostDriver...
[INFO - 2013-07-24T05:21:50.923Z] GhostDriver - Main - running on port 26515
[INFO - 2013-07-24T05:21:51.435Z] Session [f235d040-f420-11e2-8d90-f50327bc3449] - CONSTRUCTOR - Desired Capabilities: {"browserName":"phantomjs"}
[INFO - 2013-07-24T05:21:51.435Z] Session [f235d040-f420-11e2-8d90-f50327bc3449] - CONSTRUCTOR - Negotiated Capabilities: {"browserName":"phantomjs","version":"1.9.1","driverName":"ghostdriver","driverVersion":"1.0.3","platform":"linux-unknown-32bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
[INFO - 2013-07-24T05:21:51.435Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: f235d040-f420-11e2-8d90-f50327bc3449
23:21:51.495 INFO - Done: /session
23:21:51.504 INFO - Executing: org.openqa.selenium.remote.server.handler.GetSessionCapabilities#46b78d at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2)
23:21:51.505 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2
23:21:51.520 INFO - Executing: [get: http://#### REDACTED ####/wp-login.php] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/url)
23:21:51.821 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/url
23:21:51.827 INFO - Executing: [find element: By.selector: #user_login] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:51.874 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
23:21:51.883 INFO - Executing: [send keys: 0 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#20788bf8, [#### REDACTED ####]] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/0/value)
23:21:51.939 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/0/value
23:21:51.948 INFO - Executing: [find element: By.selector: #user_pass] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:51.965 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
23:21:52.001 INFO - Executing: [send keys: 1 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#20788bf9, [#### REDACTED ####]] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/1/value)
23:21:52.065 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/1/value
23:21:52.074 INFO - Executing: [find element: By.selector: #wp-submit] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:52.099 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
23:21:52.106 INFO - Executing: [click: 2 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#20788bfa] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/2/click)
23:21:52.842 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/2/click
23:21:52.850 INFO - Executing: [find element: By.selector: #wpwrap] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:52.871 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
A failed test yields the following output from Mocha:
[s5rich#host ~]$ mocha test/selenium/acceptance/simple.js
Wordpress
1) should be able to log in
0 passing (2 seconds)
1 failing
1) Wordpress should be able to log in:
Uncaught UnknownError: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: unknown
at new bot.Error (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/atoms/error.js:108:18)
at Object.bot.response.checkResponse (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/atoms/response.js:106:9)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/webdriver.js:262:20
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/goog/base.js:1112:15
at webdriver.promise.ControlFlow.runInNewFrame_ (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1431:20)
at notify (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:315:12)
at notifyAll (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:284:7)
at fulfill (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:389:7)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1298:10
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/goog/base.js:1112:15
at webdriver.promise.ControlFlow.runInNewFrame_ (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1431:20)
at notify (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:315:12)
at notifyAll (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:284:7)
at fulfill (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:389:7)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/goog/base.js:1112:15
at webdriver.promise.ControlFlow.runInNewFrame_ (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1431:20)
at notify (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:315:12)
at notifyAll (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:284:7)
at fulfill (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:389:7)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:600:51
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/http/http.js:96:5
at IncomingMessage.<anonymous> (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/http/index.js:113:7)
at IncomingMessage.EventEmitter.emit (events.js:117:20)
at _stream_readable.js:910:16
at process._tickCallback (node.js:415:13)
==== async task ====
WebDriver.findElement(By.cssSelector("#wpwrap"))
at webdriver.WebDriver.schedule (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/webdriver.js:246:15)
at webdriver.WebDriver.findElement (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/webdriver.js:685:17)
at Context.<anonymous> (/home/s5rich/test/selenium/acceptance/simple.js:12:10)
at Test.Runnable.run (/usr/local/lib/node_modules/mocha/lib/runnable.js:194:15)
at Runner.runTest (/usr/local/lib/node_modules/mocha/lib/runner.js:355:10)
at /usr/local/lib/node_modules/mocha/lib/runner.js:401:12
at next (/usr/local/lib/node_modules/mocha/lib/runner.js:281:14)
at /usr/local/lib/node_modules/mocha/lib/runner.js:290:7
at next (/usr/local/lib/node_modules/mocha/lib/runner.js:234:23)
at Object._onImmediate (/usr/local/lib/node_modules/mocha/lib/runner.js:258:5)
at processImmediate [as _immediateCallback] (timers.js:330:15)
A failed test also yields the following output from Selenium Server:
23:25:34.742 INFO - Executing: [new session: {browserName=phantomjs}] at URL: /session)
23:25:34.743 INFO - Creating a new session for Capabilities [{browserName=phantomjs}]
23:25:34.744 INFO - executable: /usr/local/bin/phantomjs
23:25:34.744 INFO - port: 2897
23:25:34.744 INFO - arguments: [--webdriver=2897, --webdriver-logfile=/home/s5rich/phantomjsdriver.log]
23:25:34.744 INFO - environment: {}
PhantomJS is launching GhostDriver...
[INFO - 2013-07-24T05:25:34.879Z] GhostDriver - Main - running on port 2897
[INFO - 2013-07-24T05:25:35.270Z] Session [77a0a110-f421-11e2-a6fd-61cd002d7d02] - CONSTRUCTOR - Desired Capabilities: {"browserName":"phantomjs"}
[INFO - 2013-07-24T05:25:35.270Z] Session [77a0a110-f421-11e2-a6fd-61cd002d7d02] - CONSTRUCTOR - Negotiated Capabilities: {"browserName":"phantomjs","version":"1.9.1","driverName":"ghostdriver","driverVersion":"1.0.3","platform":"linux-unknown-32bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
[INFO - 2013-07-24T05:25:35.270Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: 77a0a110-f421-11e2-a6fd-61cd002d7d02
23:25:35.275 INFO - Done: /session
23:25:35.283 INFO - Executing: org.openqa.selenium.remote.server.handler.GetSessionCapabilities#13b4ce4 at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee)
23:25:35.284 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee
23:25:35.297 INFO - Executing: [get: http://#### REDACTED ####/wp-login.php] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/url)
23:25:35.592 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/url
23:25:35.597 INFO - Executing: [find element: By.selector: #user_login] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
23:25:35.619 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element
23:25:35.631 INFO - Executing: [send keys: 0 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#240035bc, [#### REDACTED ####]] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/0/value)
23:25:35.683 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/0/value
23:25:35.695 INFO - Executing: [find element: By.selector: #user_pass] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
23:25:35.712 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element
23:25:35.723 INFO - Executing: [send keys: 1 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#240035bd, [#### REDACTED ####]] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/1/value)
23:25:35.783 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/1/value
23:25:35.800 INFO - Executing: [find element: By.selector: #wp-submit] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
23:25:35.822 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element
23:25:35.832 INFO - Executing: [click: 2 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#240035be] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/2/click)
23:25:36.105 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/2/click
23:25:36.121 INFO - Executing: [find element: By.selector: #wpwrap] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
e = java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
23:25:36.285 WARN - Exception thrown
org.openqa.selenium.NoSuchElementException: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Command duration or timeout: 149 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Session ID: 77a0a110-f421-11e2-a6fd-61cd002d7d02
Driver info: org.openqa.selenium.phantomjs.PhantomJSDriver
Capabilities [{platform=LINUX, acceptSslCerts=false, javascriptEnabled=true, browserName=phantomjs, rotatable=false, driverVersion=1.0.3, locationContextEnabled=false, version=1.9.1, databaseEnabled=false, cssSelectorsEnabled=true, handlesAlerts=false, browserConnectionEnabled=false, proxy={proxyType=direct}, webStorageEnabled=false, nativeEvents=true, driverName=ghostdriver, applicationCacheEnabled=false, takesScreenshot=true}]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:191)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:554)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:307)
at org.openqa.selenium.remote.RemoteWebDriver.findElementByCssSelector(RemoteWebDriver.java:396)
at org.openqa.selenium.By$ByCssSelector.findElement(By.java:407)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.openqa.selenium.support.events.EventFiringWebDriver$2.invoke(EventFiringWebDriver.java:101)
at $Proxy1.findElement(Unknown Source)
at org.openqa.selenium.support.events.EventFiringWebDriver.findElement(EventFiringWebDriver.java:180)
at org.openqa.selenium.remote.server.handler.FindElement.call(FindElement.java:47)
at org.openqa.selenium.remote.server.handler.FindElement.call(FindElement.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at org.openqa.selenium.remote.server.DefaultSession$1.run(DefaultSession.java:169)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.openqa.selenium.remote.ScreenshotException: Screen shot has been taken
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: EventFiringWebDriver
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:125)
... 20 more
Caused by: org.openqa.selenium.remote.ErrorHandler$UnknownServerException: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: unknown
23:25:36.293 WARN - Exception: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: unknown
From what I see, it is a simple problem to solve. You have a unhandled exception that is being thrown when there is a failure: NoSuchElementException . I don't think you need to restart your grid node every time (assuming you are running your grid hub separate from your grid node instances). It may be that successive runs of the browser from your Grid Node use just enough more memory to cause a minor delay in the page, breaking your script. All you need to do is handle the NoSuchElementException gracefully by wrapping it in a retry try loop. You can also effectively do similar using FluentWait with the .ignoring method.

Resources