Can't query datetime column in SQLAlchemy with postgreSQL - python-3.x

I want to delete rows based on a datetime filter.
I created a table with DateTime column without timezone using similar script.
class VolumeInfo(Base):
...
date: datetime.datetime = Column(DateTime, nullable=False)
Then I try to delete rows using such filter
days_interval = 10
to_date = datetime.datetime.combine(
datetime.datetime.utcnow().date(),
datetime.time(0, 0, 0, 0),
).replace(tzinfo=None)
from_date = to_date - datetime.timedelta(days=days_interval)
query = delete(VolumeInfo).where(Volume.date < from_date)
Unexpectedly, sometimes there is no error, but sometimes there is the error:
Traceback (most recent call last):
...
File "script.py", line 381, in delete_volumes
db.execute(query)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1660, in execute
) = compile_state_cls.orm_pre_session_exec(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1843, in orm_pre_session_exec
update_options = cls._do_pre_synchronize_evaluate(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 2007, in _do_pre_synchronize_evaluate
matched_objects = [
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 2012, in <listcomp>
and eval_condition(state.obj())
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/evaluator.py", line 211, in evaluate
return operator(eval_left(obj), eval_right(obj))
TypeError: can't compare offset-naive and offset-aware datetimes
Using python3.10 in docker (image python:3.10-slim) and postgreSQL database with psycopg2 driver.
I have already tried all possible options, but this error appears every once in a while
How can i solve this? or where I made a mistake?
UPD1:

Related

Error downloading historical stcok data using pandas_datareader Anconda3, Spyder 5.3.3

I have watch list of 30 stocks. The list is in a text file called "WatchList". I initialize the list as:
stock = []
and read the symbols line by line. I specify a location to store the data in csv format for each symbol.
I have the latest version of pandas_datareader and it is 0.10.0. I have used a while loop and pandas_datareader before. However, now I am experiencing problems. I receive the following error message:
runfile('E:/Stock_Price_Forecasting/NewStockPriceFetcher.py', wdir='E:/Stock_Price_Forecasting')
Enter the name of file to access WatchList
WatchList.txt
0 AAPL <class 'str'>
Traceback (most recent call last):
File "C:\Users\Om\anaconda3\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
exec(code, globals, locals)
File "e:\stock_price_forecasting\newstockpricefetcher.py", line 60, in
df = web.DataReader(stock[i], data_source='yahoo', start=start_date, end=end_date)
File "C:\Users\Om\anaconda3\lib\site-packages\pandas\util_decorators.py", line 211, in wrapper
return func(*args, **kwargs)
File "C:\Users\Om\anaconda3\lib\site-packages\pandas_datareader\data.py", line 370, in DataReader
return YahooDailyReader(
File "C:\Users\Om\anaconda3\lib\site-packages\pandas_datareader\base.py", line 253, in read
df = self._read_one_data(self.url, params=self._get_params(self.symbols))
File "C:\Users\Om\anaconda3\lib\site-packages\pandas_datareader\yahoo\daily.py", line 153, in _read_one_data
data = j["context"]["dispatcher"]["stores"]["HistoricalPriceStore"]
TypeError: string indices must be integers
The portion of my code that shows the while loop is shown below:
i = 0
while i < len(stock):
print(i, stock[i], type(stock[i]))
# Format the filename for each security to use in full path
stock_data_file = stock[i] + '.csv'
# Complete the path definition for stock data storage including filename
full_file_path = (file_path/stock_data_file)
# Specify the order for the columns
columnTitles = ('Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close')
# Pull the data for the stock from the Web
df = web.DataReader(stock[i], data_source='yahoo', start=start_date,
end=end_date) ** error in this line!!
# Reorder the columns for plotting Candlesticks
df=df.reindex(columns=columnTitles)
if i == 0:
df.to_csv(full_file_path)
print(i, stock[i], 'has data stored to csv file')
else:
df.to_csv(full_file_path, header=True)
print(i, stock[i], 'has data stored to csv file')
i += 1
I have looked at the parameter requirements for the Datareader and Yahoo. I belive the first paramataer is the ticker and a string value. I have been unable to find out where I am making a mistake. Any suggestions in solving this issue would be greatly appreciated. Thank you.

TypeError: not all arguments converted during string formatting in python connecting with postgresql

It seems like all no error in the code but no idea why I'm getting this.
I was creating a simple GUI app which store user details to a database(postgresql) and also they will be able to search for entries in database. This particular error ocures in this search() function so I haven't added the rest of the code. If necessary I can add them.
Hope I will get some solutions from this community.
def search(id):
conn = psycopg2.connect(dbname="postgres",user="postgres",password="1018",host="localhost",port="5432")
mycursor = conn.cursor()
query = '''select * from demotab where id=%s '''
mycursor.execute(query, (id))
row = mycursor.fetchone()
print(row)
conn.commit()
conn.close()
Getting this error below
Exception in Tkinter callback
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\tkinter\__init__.py", line 1702, in __call__
return self.func(*args)
File "appwithDB.py", line 51, in <lambda>
search_button = Button(newframe, text="Search", command=lambda : search(entry_search.get()))
File "appwithDB.py", line 71, in search
mycursor.execute(query, (id))
TypeError: not all arguments converted during string formatting
The second argument to mycursor.execute must be an iterable containing the values to insert in the query
You can use a list: mycursor.execute(query, [id])
or a one-element tuple: mycursor.execute(query, (id,))
Notice the comma. (id) is the same than id. In python, the comma is making the tuple, not the parenthesis.

What is causing this issue when trying to get yahoo_fin to return prices for a list of tickers?

I have a list of tickers that I want to retrieve the prices for by running the following:
from yahoo_fin import stock_info as si
for x in watchlist:
print(si.get_live_price(x))
When I run this I get the following error:
File "", line 1, in
runfile('C:/Users/User/OneDrive/Documents/Stuff/fluff 2.py', wdir='C:/Users/User/OneDrive/Documents/Stuff')
File
"D:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 705, in runfile
execfile(filename, namespace)
File
"D:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/User/OneDrive/Documents/Stuff/fluff 2.py", line 46,
in
print(si.get_live_price(x))
File "D:\Anaconda3\lib\site-packages\yahoo_fin\stock_info.py", line
338, in get_live_price
df = get_data(ticker, end_date = pd.Timestamp.today() + pd.DateOffset(10))
File "D:\Anaconda3\lib\site-packages\yahoo_fin\stock_info.py", line
68, in get_data
temp = loads(needed)
ValueError: Expected object or value
However, when I refer to a ticker directly, it runs normally:
print(si.get_live_price('tsla'))
348.8399963378906
What could be causing this issue? Is it due to me using a different html parser than that used with yahoo_fin in an earlier part of the code?
Try this out, It gives you complete dataframe for last 6 months data
import yfinance as yf
for x in ['TSLA','AAPL']:
data = yf.download( tickers = x)
print(data['Close'][-1])
Output :
348.8399963378906
268.4800109863281
If you want last 6 month data then you can store individual dataframe. In above case I have printed only last index as you wanted LTP.
This issue should be fixed now in the latest version of yahoo_fin (0.8.4). It was due to a change in Yahoo Finance's structure. See here for news about recent updates: http://theautomatic.net/2019/12/16/updates-to-yahoo_fin-package/

Errors writing a dataframe to DB2 using Pandas to_sql

I am trying to load data from a pandas dataframe to an IBM DB2 Data Warehouse environment. The table already exists so I am just appending rows to the table. I have built the dataframe to mirror every field in the table exactly.
I am using Pandas to_sql method to try to get the dataframe data to the table. I already know that I am connected to the database, but when I run the code I am getting the following error:
AttributeError: 'function' object has no attribute 'cursor'
I didn't see anything in the pandas documentation about having to define a cursor when using to_sql. Any help would be appreciated.
I tried writing a direct sql insert statement rather than using to_sql but couldn't get that to work properly either. I already have a to_csv method where I'm writing the dataframe to a csv file, so I would like to just use the same dataframe to insert to the table.
I cannot add too much code as this is a company project, but the table has 15 columns with differing datatypes (decimal, character, timestamp).
This is my to_sql statement:
`output_df.to_sql(name='PD5', con=self.db2_conn, schema='REBTEAM', if_exists='append', index=False)`
I expect the table to be loaded with the rows. The test file I'm using has 880 rows, so I would expect the table to have 880 rows.
Here is the entire error message I'm getting:
Warning (from warnings module):
File "C:\Users\dt24358\lib\site-packages\pandas\core\generic.py", line 2531
dtype=dtype, method=method)
UserWarning: The spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\dt24358\lib\tkinter\__init__.py", line 1705, in __call__
return self.func(*args)
File "C:\Users\dt24358\Scripts\Pricing Tool\Rebate_GUI_SQL.py", line 100, in <lambda>
command= lambda: self.submit_click(self.path, self.fileName, self.save_location, self.request_var.get(), self.execution_var.get(),self.dt_user_id, self.rebateAggregator))
File "C:\Users\dt24358\Scripts\Pricing Tool\Rebate_GUI_SQL.py", line 210, in submit_click
output_df.to_sql(name='PD5', con=self.db2_conn, schema='REBTEAM', if_exists='append', index=False)
File "C:\Users\dt24358\lib\site-packages\pandas\core\generic.py", line 2531, in to_sql
dtype=dtype, method=method)
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 460, in to_sql
chunksize=chunksize, dtype=dtype, method=method)
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 1546, in to_sql
table.create()
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 572, in create
if self.exists():
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 560, in exists
return self.pd_sql.has_table(self.name, self.schema)
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 1558, in has_table
return len(self.execute(query, [name, ]).fetchall()) > 0
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 1426, in execute
cur = self.con.cursor()
AttributeError: 'function' object has no attribute 'cursor'

python pandas merging excel sheets not working

I'm trying to merge two excel sheets using the common filed Serial but throwing some errors. My program is as below :
(user1_env)root#ubuntu:~/user1/test/compare_files# cat compare.py
import pandas as pd
source1_df = pd.read_excel('a.xlsx', sheetname='source1')
source2_df = pd.read_excel('a.xlsx', sheetname='source2')
joined_df = source1_df.join(source2_df, on='Serial')
joined_df.to_excel('/root/user1/test/compare_files/result.xlsx')
getting error as below :
(user1_env)root#ubuntu:~/user1/test/compare_files# python3.5 compare.py
Traceback (most recent call last):
File "compare.py", line 5, in <module>
joined_df = source1_df.join(source2_df, on='Serial')
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/core/frame.py", line 4385, in join
rsuffix=rsuffix, sort=sort)
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/core/frame.py", line 4399, in _join_compat
suffixes=(lsuffix, rsuffix), sort=sort)
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/tools/merge.py", line 39, in merge
return op.get_result()
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/tools/merge.py", line 223, in get_result
rdata.items, rsuf)
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/core/internals.py", line 4445, in items_overlap_with_suffix
to_rename)
ValueError: columns overlap but no suffix specified: Index(['Serial'], dtype='object')
I'm referring below SO link for the issue :
python compare two excel sheet and append correct record
Small modification worked for me,
import pandas as pd
source1_df = pd.read_excel('a.xlsx', sheetname='source1')
source2_df = pd.read_excel('a.xlsx', sheetname='source2')
joined_df = pd.merge(source1_df,source2_df,on='Serial',how='outer')
joined_df.to_excel('/home/gk/test/result.xlsx')
It is because of the overlapping column names after join. You can either set your index to Serial and join, or specify a rsuffix= or lsuffix= value in your join function so that the suffix value would be appended to the common column names.

Resources