I have been using AWS for about 6 months now for some simple tasks like using EC2 to setup python scripts and cronjobs, updating tables on postgres. So far everything has been great but recently everything went nasty after the cronjobs that update tables increased upto 20, so there are multiple collisions of them simultaneously trying to connect to the database modify the tables(but no 2 scripts try to modify the same table at once). they started failing randomly and I am starting to see most of them fail, I need to reboot the aws rds instance for the cronjobs to work properly but again the same things repeat however. As you can see here, the db connections keep increasing and once I restart it, it's back to normal, same with the current activity sessions(wonder how it could be a float to be honest)
Error:
Exception: Unexpected exception type: ReadTimeout
I am wondering if the way I connect to the database is the problem or what? I can't even figure out if it's some measure from RDS to freeze any modifications to the tables because of many connection attempts. The thing is I am sure the script works, because when I stop all cronjobs, reboot the rds instance and run this script it works fine, but when I start all of them again, it's failing. Can someone please help me with some leads on how to figure this out?
Most of my scripts look pretty similar to the following, there are scripts that run once in a day and that runs once every 5 mins too
def get_daily_data(coin, start_date, time_period):
return df
def main():
coin1 = 'bitcoin'
time_period = '1d'
DATABASE_URI = f'postgresql+psycopg2://{USR}:{token}#{ENDPOINT}:{PORT}/{DBNAME}'
engine = create_engine(DATABASE_URI, echo=False, poolclass=NullPool)
connection = engine.connect()
if connection is not None:
old_btc_data = pd.read_sql("SELECT * FROM daily_data", connection)
start_date= str(old_btc_data['datetime'].iloc[-1].date() - datetime.timedelta(days=7))
latest_data_btc = get_daily_data(coin1, start_date, time_period)
if latest_data_btc is not None:
try:
latest_data_btc = latest_data_btc.reset_index()
latest_data_btc['datetime'] = latest_data_btc['datetime'].dt.tz_localize(None)
latest_data_btc = pd.concat([old_btc_data, latest_data_btc], ignore_index=True)
latest_data_btc = latest_data_btc.drop_duplicates(keep='last')
latest_data_btc = latest_data_btc.set_index('datetime')
latest_data_btc.to_sql(daily_data, if_exists='replace', con=connection)
print(f"Pushed BTC daily data to the database")
except:
print(f"BTC daily data to the database update FAILED!!!!!!")
connection.close()
if __name__ == '__main__':
main()
I learnt that the error is to related to creating a connection, somehow creating a connection using (connection = engine.connect()) is freezing any other modifications to the database table, it started working out for me by using the engine as the connection directly as follows:
connection = create_engine(DATABASE_URI, echo=False, poolclass=NullPool)
Edit: Much better to use sessions with psycopg2 cursors, exception handling would be easy.
Related
Base live update in Postgres
Hello guys.
I have a Postgres database where I create my dataframe.
So my graphics work, my calculations work, almost everything is beautiful.
The only problem is that if the database changes I need to stop the Flask service and restart, only then my dashboard is updated.
I tried a lot before coming back here with this answer.
I'm taking a beating with update database.
I can't get dash to update my charts and tables by itself.
I tried on other forums but was not successful.
I'll post my code here updated and minimized.
There will be two links to the drive, one with the postgres database and the other with the code.
the code is relatively small the part that doesn't work properly.
I will send you two links:
1 - Containing the complete code:
https://drive.google.com/file/d/1c0e8UelUgGVx_IrWLFncwc2JRPraUS79/view?usp=share_link
I have a few observations for this code to work:
a) Install the libraries highlighted in the teste.py file before the imports.
b) The database access configuration I made in a file called pg_config.json, just edit and inform your bank credentials.
2 - Containing the Postgres database:
https://drive.google.com/file/d/1Fwrc0xAMfVnv0_lUSHjNTO3HnQIkUvHN/view?usp=sharing
teste.py is a table that gets information from df_data_generals
I did something like below with mysql. Please refer it and check does it match your requirment.
dcc.Store(id='store-data', data=[], storage_type='memory'), # 'local' or 'session'
dcc.Interval(id='update', n_intervals = 0, interval=1000*30)
])
#app.callback(Output('store-data','data'),
[Input('update', 'n_intervals')])
def update_data(n):
global jobs_2
db = mysql.connector.connect(
host="localhost",
user="root",
password="",
database="indeed_data_dump")
cur = db.cursor()
cur.execute("SELECT * FROM jobs")
columns = [col[0] for col in mycursor.description]
data = mycursor.fetchall()
jobs_2 = pd.DataFrame(data, columns=columns)
db.close()
jobs_2.to_dict()
I want to decrease the number of idle PostgreSQL requests coming in from fast-api calls. I am not able to figure out what exactly is leaving this many idle connections as multiple people are using this DB and APIs associated with it.
Can someone suggest what I might have done wrong to leave this many idle connections or an efficient way to figure out what is causing this so that I can accordingly fix that portion and decrease it somehow.
Not sure if I have provided sufficient information to explain this but if anything else is required, I will be more than happy to provide that information.
postgresql idle connection screenshot
This is how I am creating a PostgreSQL object via fastapi
class postgres:
def __init__(self, config):
try:
SQLALCHEMY_DATABASE_URL = "postgresql://" + \
config['postgresql']['user']+":"+config['postgresql']['password']+"#" + \
config['postgresql']['host']+":5432" + \
"/"+config['postgresql']['database']
# print(SQLALCHEMY_DATABASE_URL)
engine = create_engine(SQLALCHEMY_DATABASE_URL, future=True)
self.SessionLocal = sessionmaker(
autocommit=False, autoflush=True, bind=engine)
except Exception as e:
raise
def get_db(self):
"""
Function to return session variable used by ORM
:return: SessionLocal
"""
try:
db = self.SessionLocal()
yield db
finally:
db.close()
I believe this is caused by the get_db(self), as it creates (yields) a new connection/session object every time you call SessionLocal().
There are two Python scripts. Script A keeps reading the database for records and then send out messages. Script B keeps finding data and inserting into the database. Both of them keeps running without being terminated.
If there are unhandled records in the database, when script A starts, script A can see these records, and handle them correctly. Script A is now idle. After some time, script B insert some new records. However, script A cannot see these newly added records.
In script B, after inserting the records, I have connection.commit(). The records do in the database.
script A does not work correctly when:
persist = PersistentDB(pymysql, host=DB_HOST, port=DB_PORT, user=DB_USERNAME, passwd=DB_PASSWORD, db=DB_DATABASE, charset='utf8mb4')
while True:
connection = persist.connection()
cursor = connection.cursor()
cursor.exec("SELECT...")
script A works correctly when:
while True:
persist = PersistentDB(pymysql, host=DB_HOST, port=DB_PORT, user=DB_USERNAME, passwd=DB_PASSWORD, db=DB_DATABASE, charset='utf8mb4')
connection = persist.connection()
cursor = connection.cursor()
cursor.exec("SELECT...")
Should I really use the PersistentDB in this way? It seems to override the persist connection pool everytime I declare it. Please tell me what is the correct way to do it.
P.S.
Connecting to MySQL 8 with Python 3, pymysql and DBUtils
I am connecting to snowflake datawarehouse from Python and I encounter a weird behavior. The Python program exits successfully if I retrieve fewer number of rows from SnowFlake but hangs in there in-definitely if I try to retrieve more than 200K rows. I am 100% sure that there are no issues with my machine because I am able to retrieve 5 to 10 million rows from other type of database systems such as Postgres.
My Python environment is Python 3.6 and I use the following version of the libraries -> SQLAlchemy 1.1.13, snowflake-connector-python 1.4.13, snowflake-sqlalchemy 1.0.7,
The following code prints the total number of rows and closes the connection.
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(URL(
account=xxxx,
user=xxxxx,
password=xxxxx,
database=xxxxx,
schema=xxxxxx,
warehouse=xxxxx))
query = """SELECT * FROM db_name.schema_name.table_name LIMIT 1000"""
results = engine.execute(query)
print (results.rowcount)
engine.dispose()
The following code prints the total number of rows but the connection doesn't close, it just hangs in there until I manually kill the Python process.
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(URL(
account=xxxx,
user=xxxxx,
password=xxxxx,
database=xxxxx,
schema=xxxxxx,
warehouse=xxxxx))
query = """SELECT * FROM db_name.schema_name.table_name LIMIT 500000"""
results = engine.execute(query)
print (results.rowcount)
engine.dispose()
I tried multiple different tables and I encounter the same issue with SnowFlake. Did anyone encounter similar issues?
Can you check the query status from UI? "History" page should include the query. If the warehouse is not ready, it may take a couple of minutes to start the query. (I guess that's very unlikely, though).
Try changing the connection to this:
connection = engine.connect()
results = connection.execute(query)
print (results.rowcount)
connection.close()
engine.dispose()
SQLAlchemy's dispose doesn't close the connection if the connection is not explicitly closed. I inquired before, but so far the workaround is just close the connection.
https://groups.google.com/forum/#!searchin/sqlalchemy/shige%7Csort:date/sqlalchemy/M7IIJkrlv0Q/HGaQLBFGAQAJ
Lastly, if the issue still persist, add the logger to top:
import logging
for logger_name in ['snowflake','botocore']:
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
ch = logging.FileHandler('log')
ch.setLevel(logging.DEBUG)
ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
logger.addHandler(ch)
and collect log.
If the output is too long to fit into here, I can take it at the issue page at https://github.com/snowflakedb/snowflake-sqlalchemy.
Note I tried it myself but cannot reproduce the issue so far.
Have you tried using a with statement instead to make your connection
instead of this:
engine = create_engine(URL(account=xxxx,user=xxxxx,password=xxxxx,database=xxxxx,schema=xxxxxx,warehouse=xxxxx))
results = engine.execute(query)
do the following:
with create_engine(URL(account=xxxx,user=xxxxx,password=xxxxx,database=xxxxx,schema=xxxxxx,warehouse=xxxxx)) as engine:
# do work
results = engine.execute(query)
...
After the with .. the engine object should be automatically closed out.
This is my code shown below that i use to connect to the database and get some tables. I have used this for a while without problem but recently I keep on getting this two errors after it runs for a few times. I assume it is something to do with the database.
Basically what happens is that i can connect to the databases, retrieve my table but since my data is very huge I do it on a for loop where I loop through dates within a given range and get the data. For example I start from 20160401 and count one by one until 20170331(about 365 loops).
However after i run like 20 loops i get this error s
OperationError "Lost connection to MySQL server during query (%s)" %
(e,))
and sometimes this one below
BaseSSHTunnelForwarderError: Could not establish session to SSH
gateway
I would like to solve this errors but if it is not possible I would like to wait for a few seconds then reconnect to the databases automatically then execute the same query. Any help in solving this will be highly appreciated.
import pymysql
from sshtunnel import SSHTunnelForwarder
with SSHTunnelForwarder(
(ssh_host, ssh_port),
ssh_username=ssh_user,
ssh_pkey=mypkey,
remote_bind_address=(sql_ip, sql_port)) as tunnel:
conn = pymysql.connect(host='127.0.0.1',
user=sql_username,
passwd=sql_password,
port=tunnel.local_bind_port,
db=db,charset='ujis')
data = pd.read_sql_query(query_2, conn,)
This is what I have tried so far but it is not working
while True:
try:
with SSHTunnelForwarder(
(ssh_host, ssh_port),
ssh_username=ssh_user,
ssh_pkey=mypkey,
remote_bind_address=(sql_ip, sql_port)) as tunnel:
conn = pymysql.connect(host='127.0.0.1',
user=sql_username,
passwd=sql_password,
port=tunnel.local_bind_port,
db=db, charset='ujis')
data = pd.read_sql_query(query, conn)
break
except pymysql.err.OperationalError as e:
conn.ping(reconnect=True)