I'm using threading to execute multiple SQL queries simultaneously. I first append the connections to a list called connections as such:
import pyodbc
connections = []
num_connnections = 100
for i in range(num_connections):
connections.append(pyodbc.connect('connection_string'))
This works and so does threading multiple queries. However, if I run this process many times, I get the error
('HY000', '[HY000] [Oracle][ODBC][Ora]ORA-12519: TNS:no appropriate service handler found
(12519) (SQLDriverConnect); [HY000] [Oracle][ODBC][Ora]ORA-12519: TNS:no appropriate service handler found (12519)')
I'm fairly sure this is because the number of ODBC connections becomes too high. When I try closing them with
pyodbc.pooling = False
for i in connections:
i.close()
del i
del connections
the list connections is deleted. However, it doesn't appear it closed any connections because the next time I run pyodbc.connect('connection_string') I immediately get the same error. Any ideas on what this could be?
Related
We have parse server using postgresql as database. Problem is it runs simple queries extremely slow compared to sql or running them from js via pg.
For example getting all users (~5k rows in table) takes couple of seconds. Getting sessions and users takes from 3 seconds up to in extreme cases 8-10. Example:
let Session = Parse.Object.extend("_Session");
let sessionQuery = new Parse.Query(Session);
sessionQuery.include("user");
sessionQuery.limit(100000);
let sessions = await sessionQuery.find({ useMasterKey: true });
This segment runs slow 2-3 seconds, sometimes up to 8, while running sql is 100ms or so. There are only ~5k users and 9k sessions. We tried setting direct access variable, we checked that its indeed properly set inside parseServer.js source. Currently we are moving all select type queries to SQL but it would be preferable to find solution.
I am currently using Cassandra 3.11 having 3 node cluster & consistency of one, along with NodeJS express cassandra client 2.3.0. I am using the method saveAsync for storing the model data. But I am getting an error : apollo.model.save.dberror
Error during save query on DB -> NoHostAvailableError: All host(s) tried for query failed. \
First host tried, <ip>:<port>: OperationTimedOutError: The host <ip>:<port> \
did not reply before timeout 12000 ms.
I am unable to verify what is causing this error. This happens for most of the records & only few of them are getting through. I am inserting this data reading from a kafka topic & pushing it to cassandra after few validations. The data rate is around 200 per second.
Tried googling & searching in stack-overflow but unable to get any details around it. Sample code.
SomeData.insert = function (data) {
data = transformData(data); // Basically cleans & normalizes the data to suit the model
let any_data = new ExpressCassandra.instance.data_store(data);
return any_data.saveAsync()
.catch(function(err){
return Promise.reject(err);
});
};
The Node.js driver returns NoHostAvailableError after it has tried all hosts and none of them were available.
The OperationTimedOutError means that driver attempted to contact a node but never got a reply. This is a different error to a read or write timeout which is a response returned by the coordinator node.
OperationTimedOutError indicates that the nodes are unresponsive, usually because they are overloaded. You will need to review the logs and if you're seeing lots of GC activity and/or lots of dropped mutations/reads/messages then the nodes cannot keep up with the load and you need to consider increasing the capacity of your cluster by adding more nodes. Cheers!
I need to insert a couple of dozen rows remotely once every 10 seconds. I am not sure whether I should use async with connection around each insert, closing the connection and re-opening every 10s, or just await conn = ... once and have a handle on an open connection forever.
Please explain at what point the decision would change, based on other numbers of rows and insert frequency.
How can i tell that i have reached the maximum number of queries that my cluster can handle at a time?
If the connection pool size is too small for the workload then we have to adjust the max_pooled_connections configuration variable, which controls the number of pooled connections between each pair of nodes.
However how can I tell how many pooled connections we have at a specific time ?
In memsql agregator status i can see following entries Aborted_connects is 11 - why do we abort those connection ? Also Max_used_connections is 41, while Connections is a number that increases constantly.
How can i tell that i have reached the maximum number of queries that
my cluster can handle at a time?
There isn't a hard limit on the number of queries you can send other than max_connections (100k), but at some point the cluster will not execute them all at once and will schedule/queue them up. Is your question about the former or the latter?
If the connection pool size is too small for the workload then we have
to adjust the max_pooled_connections configuration variable, which
controls the number of pooled connections between each pair of nodes.
However how can I tell how many pooled connections we have at a
specific time ?
show leaves will show how many connections are currently open from the current node to each leaf. So the current connection pool size is min(current open connections, max_pooled_connections). Note that this is per (node, node) pair.
In memsql agregator status i can see following entries
Aborted_connects is 11 - why do we abort those connection ? Also
Max_used_connections is 41, while Connections is a number that
increases constantly.
Aborted connects includes e.g. failed login authentications.
max_used_connections is max peak, connections is cumulative total.
Recently I have been trying to use Mongoengine and Flask with a Replica set. I can connect but, when the primary node changes, connection is lost and there is a break.
Here's a snippet where you can test the behavior. It is using the very useful http://flip-flop.mlab.com/ site to debug replica-set problems
from flask import Flask
from mongoengine import connect
from flask_mongoengine import MongoEngine
import os
db = MongoEngine()
app = Flask(__name__)
class TestDoc(db.Document):
texto = db.StringField()
class ProductionConfig:
def get_conn_data(self):
conn = {
'host':"mongodb://testdbuser:testdbpass#flip.mongolab.com:53117,flop.mongolab.com:54117/testdb?replicaSet=rs-flip-flop",
'replicaSet': 'rs-flip-flop'
}
return conn
import time
app.config['MONGODB_SETTINGS'] = ProductionConfig().get_conn_data()
db.init_app(app)
if __name__ == '__main__':
with app.test_client() as c:
while True:
time.sleep(1)
print(TestDoc.objects().count())
TestDoc(texto="1").save()
I get every time that the primary changes an error: pymongo.errors.AutoReconnect: connection closed
.
Many thanks! I have tried a couple of different pyMongo versions but without success. Any help will be really, really appreciated!
The issue here is that the election of a new primary is not instantaneous. From the docs:
It varies, but a replica set will generally select a new primary within a
minute.
For instance, it may take 10-30 seconds for the members of a replica
set to declare a primary inaccessible (see electionTimeoutMillis). One
of the remaining secondaries holds an election to elect itself as a
new primary. During the election, the cluster is unavailable for
writes.
The election itself may take another 10-30 seconds.
In the time between the primary going down and the replica being elected as the new primary then there is no connection that will accept writes (because they have to go to primaries).
However, there are some things you can do to your code to make it more resilient in these situations.
Firstly, you should set a read preference on the connection (more info here):
conn = {
'host':"mongodb://testdbuser:testdbpass#flip.mongolab.com:53117,flop.mongolab.com:54117/testdb",
'replicaSet': 'rs-flip-flop',
'read_preference': ReadPreference.SECONDARY_PREFERRED
}
This means that reads should be pretty robust during the election.
Unfortunately, short of wrapping all of your writes in try blocks your code will fall over if it is trying to write during the election.
This should be less of a problem than it seems in your question's example because (assuming you are doing your writes in a flask route) the webserver will throw a 500 error response. By the time you request the route again from flask the election should be complete and mongoengine will be writing to the new primary.