Cassandra retry_policy is not enforced in python client? - cassandra

Question

I have a single node Apache Cassandra cluster 3.11.6
I write 2750 rows by 10000 columns each first in apache airflow DAG task (this is successfully passed)
Then immediately after I try to connect to Cassandra to perform various reads inside another set of parallel tasks of the same airflow DAG and it fails with
ERROR - ('Unable to connect to any servers', {'10.0.1.135:9042': OperationTimedOut('errors=None, last_host=None')})
I have configured retries in execution_profiles, but it seems they're not enforced or I am misreading how the "retry" is supposed to work on the client side.
nodetool status shows UN, means UP/Normal.
I have multiple DAG tasks running in parallel pulling info from Cassandra. Some of them finish successfully (green), but some do fail because of the OperationTimedOut exception.
I don't get retries with failed connections, you can see this in the following apache airflow log:
it started at [2020-07-22 22:16:28,345]
and it errored at [2020-07-22 22:16:31,546]
which is just 3 seconds. However in the profile I set:
retry_policy=ConstantReconnectionPolicy(delay=10),
Log
[2020-07-22 22:16:28,345] {{taskinstance.py:880}} INFO - Starting attempt 1 of 1
[2020-07-22 22:16:28,345] {{taskinstance.py:881}} INFO -
--------------------------------------------------------------------------------
[2020-07-22 22:16:28,359] {{taskinstance.py:900}} INFO - Executing <Task(DjangoOperator): RespondentMediaValueMatrixImportStep> on 2020-07-22T22:02:20+00:00
[2020-07-22 22:16:28,363] {{standard_task_runner.py:53}} INFO - Started process 651 to run task
[2020-07-22 22:16:28,622] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: etl-run-dag.RespondentMediaValueMatrixImportStep 2020-07-22T22:02:20+00:00 [running]> 10.0.102.143
[2020-07-22 22:16:28,803] {{logging_mixin.py:112}} INFO - [2020-07-22 22:16:28,802] {{connection.py:101}} WARNING - Cluster.__init__ called with contact_points specified, but load-balancing policies are not specified in some ExecutionProfiles. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = ['cassandra-node0.dev.emotionaldna.host'], EPs without explicit LBPs = ('EXEC_PROFILE_DEFAULT',))
[2020-07-22 22:16:29,543] {{logging_mixin.py:112}} INFO - [2020-07-22 22:16:29,543] {{policies.py:292}} INFO - Using datacenter 'us-east-2' for DCAwareRoundRobinPolicy (via host '10.0.1.135:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
[2020-07-22 22:16:31,545] {{logging_mixin.py:112}} INFO - [2020-07-22 22:16:31,545] {{connection.py:103}} WARNING - [control connection] Error connecting to 10.0.1.135:9042:
Traceback (most recent call last):
File "cassandra/cluster.py", line 3522, in cassandra.cluster.ControlConnection._reconnect_internal
File "cassandra/cluster.py", line 3591, in cassandra.cluster.ControlConnection._try_connect
File "cassandra/cluster.py", line 3588, in cassandra.cluster.ControlConnection._try_connect
File "cassandra/cluster.py", line 3690, in cassandra.cluster.ControlConnection._refresh_schema
File "cassandra/metadata.py", line 142, in cassandra.metadata.Metadata.refresh
File "cassandra/metadata.py", line 165, in cassandra.metadata.Metadata._rebuild_all
File "cassandra/metadata.py", line 2522, in get_all_keyspaces
File "cassandra/metadata.py", line 2031, in get_all_keyspaces
File "cassandra/metadata.py", line 2719, in cassandra.metadata.SchemaParserV3._query_all
File "cassandra/connection.py", line 985, in cassandra.connection.Connection.wait_for_responses
File "cassandra/connection.py", line 983, in cassandra.connection.Connection.wait_for_responses
File "cassandra/connection.py", line 1435, in cassandra.connection.ResponseWaiter.deliver
cassandra.OperationTimedOut: errors=None, last_host=None
[2020-07-22 22:16:31,546] {{logging_mixin.py:112}} INFO - [2020-07-22 22:16:31,545] {{connection.py:103}} ERROR - Control connection failed to connect, shutting down Cluster:
Traceback (most recent call last):
File "cassandra/cluster.py", line 1690, in cassandra.cluster.Cluster.connect
File "cassandra/cluster.py", line 3488, in cassandra.cluster.ControlConnection.connect
File "cassandra/cluster.py", line 3533, in cassandra.cluster.ControlConnection._reconnect_internal
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.1.135:9042': OperationTimedOut('errors=None, last_host=None')})
[2020-07-22 22:16:31,546] {{logging_mixin.py:112}} INFO - [2020-07-22 22:16:31,546] {{connection.py:107}} WARNING - [Connection: default] connect failed, setting up for re-attempt on first use
[2020-07-22 22:16:31,546] {{taskinstance.py:1145}} ERROR - ('Unable to connect to any servers', {'10.0.1.135:9042': OperationTimedOut('errors=None, last_host=None')})
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 978, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/app/dags/etl/workflow.py", line 126, in run_import_step
keyspace=task_instance.get('cassandra_keyspace')
File "/app/etl_process/import_steps/mixins.py", line 469, in __init__
super().__init__(*args, **kwargs)
File "/app/etl_process/import_steps/mixins.py", line 268, in __init__
super().__init__(*args, **kwargs)
File "/app/etl_process/import_steps/abstract.py", line 177, in __init__
self._cas = get_session()
File "/app/etl_process/cassandra/client.py", line 60, in get_session
execution_profiles={EXEC_PROFILE_DEFAULT: profile},
File "/usr/local/lib/python3.7/site-packages/cassandra/cqlengine/connection.py", line 326, in setup
retry_connect=retry_connect, cluster_options=kwargs, default=True)
File "/usr/local/lib/python3.7/site-packages/cassandra/cqlengine/connection.py", line 195, in register_connection
conn.setup()
File "/usr/local/lib/python3.7/site-packages/cassandra/cqlengine/connection.py", line 103, in setup
self.session = self.cluster.connect()
File "cassandra/cluster.py", line 1667, in cassandra.cluster.Cluster.connect
File "cassandra/cluster.py", line 1703, in cassandra.cluster.Cluster.connect
File "cassandra/cluster.py", line 1690, in cassandra.cluster.Cluster.connect
File "cassandra/cluster.py", line 3488, in cassandra.cluster.ControlConnection.connect
File "cassandra/cluster.py", line 3533, in cassandra.cluster.ControlConnection._reconnect_internal
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.0.1.135:9042': OperationTimedOut('errors=None, last_host=None')})
settings.CASSANDRA is
CASSANDRA_REQUEST_TIMEOUT = 90000
CASSANDRA = {
'NAME': 'cassandra',
'USER': user,
'PASSWORD': password,
'TEST_NAME': 'test_db',
'HOST': host,
'OPTIONS': {
'replication': {
'strategy_class': 'SimpleStrategy',
'replication_factor': 1,
},
'connection': {
'consistency': CASSANDRA_CONSISTENCY_LEVEL,
'retry_connect': True,
},
'session': {
'default_timeout': CASSANDRA_REQUEST_TIMEOUT,
'default_fetch_size': 10000,
},
},
}
get_session()
from django.conf import settings
from cassandra.auth import PlainTextAuthProvider
from cassandra.cluster import EXEC_PROFILE_DEFAULT, ExecutionProfile
from cassandra.cqlengine import connection
from cassandra.policies import (
ConstantReconnectionPolicy, DowngradingConsistencyRetryPolicy
)
from cassandra.query import tuple_factory
__all__ = ['get_session']
def get_session(
keyspace: str = None,
consistency_level=settings.CASSANDRA_CONSISTENCY_LEVEL,
request_timeout=settings.CASSANDRA_REQUEST_TIMEOUT,
) -> connection:
"""Initiate connection with apache cassandra cluster.
Arguments:
:param str keyspace: default keyspace to connect to
:param int consistency_level: desired consistency level of the connection
:param int request_timeout: cassandra request timeout. If wait time exceeds
this number, then cassandra will send 1300 error code with 0 nodes
replied statement in the response.
"""
dbconf = settings.CASSANDRA
auth_provider = PlainTextAuthProvider(
username=dbconf['USER'],
password=dbconf['PASSWORD'],
)
host = dbconf['HOST']
# define execution profile for the cluster/session
profile = ExecutionProfile(
retry_policy=ConstantReconnectionPolicy(delay=10),
consistency_level=consistency_level,
request_timeout=request_timeout,
row_factory=tuple_factory
)
# the host should be always LIST passed in the connection
# setup
if isinstance(host, str):
host = [host]
# setup the connection
connection.setup(
host,
keyspace,
retry_connect=True,
protocol_version=4,
auth_provider=auth_provider,
consistency=consistency_level,
execution_profiles={EXEC_PROFILE_DEFAULT: profile},
)
return connection.session
cassandra.yaml
# How long the coordinator should wait for read operations to complete
read_request_timeout_in_ms: 600000
# How long the coordinator should wait for seq or index scans to complete
range_request_timeout_in_ms: 600000
# How long the coordinator should wait for writes to complete
write_request_timeout_in_ms: 600000
# How long the coordinator should wait for counter writes to complete
counter_write_request_timeout_in_ms: 100000
# How long a coordinator should continue to retry a CAS operation
# that contends with other proposals for the same row
cas_contention_timeout_in_ms: 100000
# How long the coordinator should wait for truncates to complete
# (This can be much longer, because unless auto_snapshot is disabled
# we need to flush first so we can snapshot before removing the data.)
truncate_request_timeout_in_ms: 600000
# The default timeout for other, miscellaneous operations
request_timeout_in_ms: 600000
# How long before a node logs slow queries. Select queries that take longer than
# this timeout to execute, will generate an aggregated log message, so that slow queries
# can be identified. Set this value to zero to disable slow query logging.
slow_query_log_timeout_in_ms: 3000
# Enable operation timeout information exchange between nodes to accurately
# measure request timeouts. If disabled, replicas will assume that requests
# were forwarded to them instantly by the coordinator, which means that
# under overload conditions we will waste that much extra time processing
# already-timed-out requests.
#
# Warning: before enabling this property make sure to ntp is installed
# and the times are synchronized between the nodes.
cross_node_timeout: false

Answer 1

You're using incorrect class as parameter for retry_policy. What you're specifying is reconnection policy that defines how to try to re-connect to the node that is marked as DOWN. Retry policy defines what to do with failed statements. And you can omit it, as by default it's set to the RetryPolicy class that may retry some statements, but only if they are marked as is_idempotent=True (see documentation)