I want to delete rows based on a datetime filter.
I created a table with DateTime column without timezone using similar script.
class VolumeInfo(Base):
...
date: datetime.datetime = Column(DateTime, nullable=False)
Then I try to delete rows using such filter
days_interval = 10
to_date = datetime.datetime.combine(
datetime.datetime.utcnow().date(),
datetime.time(0, 0, 0, 0),
).replace(tzinfo=None)
from_date = to_date - datetime.timedelta(days=days_interval)
query = delete(VolumeInfo).where(Volume.date < from_date)
Unexpectedly, sometimes there is no error, but sometimes there is the error:
Traceback (most recent call last):
...
File "script.py", line 381, in delete_volumes
db.execute(query)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1660, in execute
) = compile_state_cls.orm_pre_session_exec(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1843, in orm_pre_session_exec
update_options = cls._do_pre_synchronize_evaluate(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 2007, in _do_pre_synchronize_evaluate
matched_objects = [
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 2012, in <listcomp>
and eval_condition(state.obj())
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/evaluator.py", line 211, in evaluate
return operator(eval_left(obj), eval_right(obj))
TypeError: can't compare offset-naive and offset-aware datetimes
Using python3.10 in docker (image python:3.10-slim) and postgreSQL database with psycopg2 driver.
I have already tried all possible options, but this error appears every once in a while
How can i solve this? or where I made a mistake?
UPD1:
Related
I have watch list of 30 stocks. The list is in a text file called "WatchList". I initialize the list as:
stock = []
and read the symbols line by line. I specify a location to store the data in csv format for each symbol.
I have the latest version of pandas_datareader and it is 0.10.0. I have used a while loop and pandas_datareader before. However, now I am experiencing problems. I receive the following error message:
runfile('E:/Stock_Price_Forecasting/NewStockPriceFetcher.py', wdir='E:/Stock_Price_Forecasting')
Enter the name of file to access WatchList
WatchList.txt
0 AAPL <class 'str'>
Traceback (most recent call last):
File "C:\Users\Om\anaconda3\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
exec(code, globals, locals)
File "e:\stock_price_forecasting\newstockpricefetcher.py", line 60, in
df = web.DataReader(stock[i], data_source='yahoo', start=start_date, end=end_date)
File "C:\Users\Om\anaconda3\lib\site-packages\pandas\util_decorators.py", line 211, in wrapper
return func(*args, **kwargs)
File "C:\Users\Om\anaconda3\lib\site-packages\pandas_datareader\data.py", line 370, in DataReader
return YahooDailyReader(
File "C:\Users\Om\anaconda3\lib\site-packages\pandas_datareader\base.py", line 253, in read
df = self._read_one_data(self.url, params=self._get_params(self.symbols))
File "C:\Users\Om\anaconda3\lib\site-packages\pandas_datareader\yahoo\daily.py", line 153, in _read_one_data
data = j["context"]["dispatcher"]["stores"]["HistoricalPriceStore"]
TypeError: string indices must be integers
The portion of my code that shows the while loop is shown below:
i = 0
while i < len(stock):
print(i, stock[i], type(stock[i]))
# Format the filename for each security to use in full path
stock_data_file = stock[i] + '.csv'
# Complete the path definition for stock data storage including filename
full_file_path = (file_path/stock_data_file)
# Specify the order for the columns
columnTitles = ('Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close')
# Pull the data for the stock from the Web
df = web.DataReader(stock[i], data_source='yahoo', start=start_date,
end=end_date) ** error in this line!!
# Reorder the columns for plotting Candlesticks
df=df.reindex(columns=columnTitles)
if i == 0:
df.to_csv(full_file_path)
print(i, stock[i], 'has data stored to csv file')
else:
df.to_csv(full_file_path, header=True)
print(i, stock[i], 'has data stored to csv file')
i += 1
I have looked at the parameter requirements for the Datareader and Yahoo. I belive the first paramataer is the ticker and a string value. I have been unable to find out where I am making a mistake. Any suggestions in solving this issue would be greatly appreciated. Thank you.
It seems like all no error in the code but no idea why I'm getting this.
I was creating a simple GUI app which store user details to a database(postgresql) and also they will be able to search for entries in database. This particular error ocures in this search() function so I haven't added the rest of the code. If necessary I can add them.
Hope I will get some solutions from this community.
def search(id):
conn = psycopg2.connect(dbname="postgres",user="postgres",password="1018",host="localhost",port="5432")
mycursor = conn.cursor()
query = '''select * from demotab where id=%s '''
mycursor.execute(query, (id))
row = mycursor.fetchone()
print(row)
conn.commit()
conn.close()
Getting this error below
Exception in Tkinter callback
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\tkinter\__init__.py", line 1702, in __call__
return self.func(*args)
File "appwithDB.py", line 51, in <lambda>
search_button = Button(newframe, text="Search", command=lambda : search(entry_search.get()))
File "appwithDB.py", line 71, in search
mycursor.execute(query, (id))
TypeError: not all arguments converted during string formatting
The second argument to mycursor.execute must be an iterable containing the values to insert in the query
You can use a list: mycursor.execute(query, [id])
or a one-element tuple: mycursor.execute(query, (id,))
Notice the comma. (id) is the same than id. In python, the comma is making the tuple, not the parenthesis.
I have a list of tickers that I want to retrieve the prices for by running the following:
from yahoo_fin import stock_info as si
for x in watchlist:
print(si.get_live_price(x))
When I run this I get the following error:
File "", line 1, in
runfile('C:/Users/User/OneDrive/Documents/Stuff/fluff 2.py', wdir='C:/Users/User/OneDrive/Documents/Stuff')
File
"D:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 705, in runfile
execfile(filename, namespace)
File
"D:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/User/OneDrive/Documents/Stuff/fluff 2.py", line 46,
in
print(si.get_live_price(x))
File "D:\Anaconda3\lib\site-packages\yahoo_fin\stock_info.py", line
338, in get_live_price
df = get_data(ticker, end_date = pd.Timestamp.today() + pd.DateOffset(10))
File "D:\Anaconda3\lib\site-packages\yahoo_fin\stock_info.py", line
68, in get_data
temp = loads(needed)
ValueError: Expected object or value
However, when I refer to a ticker directly, it runs normally:
print(si.get_live_price('tsla'))
348.8399963378906
What could be causing this issue? Is it due to me using a different html parser than that used with yahoo_fin in an earlier part of the code?
Try this out, It gives you complete dataframe for last 6 months data
import yfinance as yf
for x in ['TSLA','AAPL']:
data = yf.download( tickers = x)
print(data['Close'][-1])
Output :
348.8399963378906
268.4800109863281
If you want last 6 month data then you can store individual dataframe. In above case I have printed only last index as you wanted LTP.
This issue should be fixed now in the latest version of yahoo_fin (0.8.4). It was due to a change in Yahoo Finance's structure. See here for news about recent updates: http://theautomatic.net/2019/12/16/updates-to-yahoo_fin-package/
I am trying to load data from a pandas dataframe to an IBM DB2 Data Warehouse environment. The table already exists so I am just appending rows to the table. I have built the dataframe to mirror every field in the table exactly.
I am using Pandas to_sql method to try to get the dataframe data to the table. I already know that I am connected to the database, but when I run the code I am getting the following error:
AttributeError: 'function' object has no attribute 'cursor'
I didn't see anything in the pandas documentation about having to define a cursor when using to_sql. Any help would be appreciated.
I tried writing a direct sql insert statement rather than using to_sql but couldn't get that to work properly either. I already have a to_csv method where I'm writing the dataframe to a csv file, so I would like to just use the same dataframe to insert to the table.
I cannot add too much code as this is a company project, but the table has 15 columns with differing datatypes (decimal, character, timestamp).
This is my to_sql statement:
`output_df.to_sql(name='PD5', con=self.db2_conn, schema='REBTEAM', if_exists='append', index=False)`
I expect the table to be loaded with the rows. The test file I'm using has 880 rows, so I would expect the table to have 880 rows.
Here is the entire error message I'm getting:
Warning (from warnings module):
File "C:\Users\dt24358\lib\site-packages\pandas\core\generic.py", line 2531
dtype=dtype, method=method)
UserWarning: The spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\dt24358\lib\tkinter\__init__.py", line 1705, in __call__
return self.func(*args)
File "C:\Users\dt24358\Scripts\Pricing Tool\Rebate_GUI_SQL.py", line 100, in <lambda>
command= lambda: self.submit_click(self.path, self.fileName, self.save_location, self.request_var.get(), self.execution_var.get(),self.dt_user_id, self.rebateAggregator))
File "C:\Users\dt24358\Scripts\Pricing Tool\Rebate_GUI_SQL.py", line 210, in submit_click
output_df.to_sql(name='PD5', con=self.db2_conn, schema='REBTEAM', if_exists='append', index=False)
File "C:\Users\dt24358\lib\site-packages\pandas\core\generic.py", line 2531, in to_sql
dtype=dtype, method=method)
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 460, in to_sql
chunksize=chunksize, dtype=dtype, method=method)
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 1546, in to_sql
table.create()
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 572, in create
if self.exists():
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 560, in exists
return self.pd_sql.has_table(self.name, self.schema)
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 1558, in has_table
return len(self.execute(query, [name, ]).fetchall()) > 0
File "C:\Users\dt24358\lib\site-packages\pandas\io\sql.py", line 1426, in execute
cur = self.con.cursor()
AttributeError: 'function' object has no attribute 'cursor'
I'm trying to merge two excel sheets using the common filed Serial but throwing some errors. My program is as below :
(user1_env)root#ubuntu:~/user1/test/compare_files# cat compare.py
import pandas as pd
source1_df = pd.read_excel('a.xlsx', sheetname='source1')
source2_df = pd.read_excel('a.xlsx', sheetname='source2')
joined_df = source1_df.join(source2_df, on='Serial')
joined_df.to_excel('/root/user1/test/compare_files/result.xlsx')
getting error as below :
(user1_env)root#ubuntu:~/user1/test/compare_files# python3.5 compare.py
Traceback (most recent call last):
File "compare.py", line 5, in <module>
joined_df = source1_df.join(source2_df, on='Serial')
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/core/frame.py", line 4385, in join
rsuffix=rsuffix, sort=sort)
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/core/frame.py", line 4399, in _join_compat
suffixes=(lsuffix, rsuffix), sort=sort)
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/tools/merge.py", line 39, in merge
return op.get_result()
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/tools/merge.py", line 223, in get_result
rdata.items, rsuf)
File "/home/user1/miniconda3/envs/user1_env/lib/python3.5/site-packages/pandas/core/internals.py", line 4445, in items_overlap_with_suffix
to_rename)
ValueError: columns overlap but no suffix specified: Index(['Serial'], dtype='object')
I'm referring below SO link for the issue :
python compare two excel sheet and append correct record
Small modification worked for me,
import pandas as pd
source1_df = pd.read_excel('a.xlsx', sheetname='source1')
source2_df = pd.read_excel('a.xlsx', sheetname='source2')
joined_df = pd.merge(source1_df,source2_df,on='Serial',how='outer')
joined_df.to_excel('/home/gk/test/result.xlsx')
It is because of the overlapping column names after join. You can either set your index to Serial and join, or specify a rsuffix= or lsuffix= value in your join function so that the suffix value would be appended to the common column names.