I have a data frame with the entry Logs.loc[0,1])[0:18] outputs '13:51:32.006655755' and I would like to convert this to milliseconds.
How would one convert this into milliseconds. I was trying to use the following:
dt.datetime.strptime((Logs.loc[0,1])[0:18], '%H:%M:%S.%f')
Traceback (most recent call last):
File "", line 1, in
dt.datetime.strptime((Logs.loc[0,1])[0:18], '%H:%M:%S.%f')
File "C:\Program Files\Anaconda3\lib_strptime.py", line 510, in _strptime_datetime
tt, fraction = _strptime(data_string, format)
File "C:\Program Files\Anaconda3\lib_strptime.py", line 346, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: 755
use pd.to_timedelta and the total_seconds method
pd.to_timedelta(Logs.loc[0,1])[0:18]).total_seconds() * 1000
If you wanted to convert the entire column
pd.to_timedelta(Logs.iloc[:, 1].str[0:18]).dt.total_seconds() * 1000
Related
I am seriously struggling with the issue of reading multiple csvs from a dir where I have them all listed.
All files I am trying to first read, and then load start with around 15 lines of report info on that file. Trying to eliminate this with skiprows=15, although this only delivers random luck.
Using e.g. glob, I keep getting either "pandas.errors.EmptyDataError: No columns to parse from file" or "Error reading line XXXX..... saw 23", "Python int too large to covert to C long", "Error: field larger than field limit (131072)". I am trying--on a per month basis--to merge hundreds of files loaded on a per hour load in a share, and then merge each month's loads into Dec, Jan, Feb...and so on. Things worked fine for all months except Dec which is why I am really after some bullit-proof solution able to read any csv file.
I have this:
import glob
import pandas as pd
import sys
import csv
maxInt = sys.maxsize
while True:
# decrease the maxInt value by factor 10
# as long as the OverflowError occurs.
try:
csv.field_size_limit(maxInt)
break
except OverflowError:
maxInt = int(maxInt/10)
df = pd.concat([pd.read_csv(f, encoding="ISO-8859-1", sep=",", skiprows=7, engine="python") for f in glob.glob('C:\\...._dec*.csv')], ignore_index=True)
df.to_csv("C:\\...\\dec_logons_total_per_some_date.csv", sep=';', index=False)
I get this:
Traceback (most recent call last):
File "C:/.../PycharmProjects/FFA_AllFiles/load_multiple_csvs.py", line 51, in <module>
df = pd.read_csv(file_, index_col=None, error_bad_lines=False, skiprows=15, header=0, low_memory=False)
File "C:\Users\...\PycharmProjects\F...\venv\lib\site-packages\pandas\io\parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\Users\...\PycharmProjects\FFA_AllFiles\venv\lib\site-packages\pandas\io\parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "C:\Users\...\PycharmProjects\FFA_AllFiles\venv\lib\site-packages\pandas\io\parsers.py", line 948, in __init__
self._make_engine(self.engine)
File "C:\Users\...\PycharmProjects\FFA_AllFiles\venv\lib\site-packages\pandas\io\parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\...\PycharmProjects\FFA_AllFiles\venv\lib\site-packages\pandas\io\parsers.py", line 2010, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas\_libs\parsers.pyx", line 540, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file
I tried a lot of attempts, including also skiprows=whatever,
I need to get the day value from a str 10/08/2020.
I'm getting dates from a List containing a years worth of dates. I'm using the index num to do some date manipulation. First I need to get the day 08 from the date str.
Code segment:
print("Today is = ",re.sub('[^!-~]+',' ',calendarData[i]).strip())
print("indexTarget is = ",indexTarget)
dateTarget = re.sub('[^!-~]+',' ',calendarData[indexTarget]).strip()
print("Target date is = ",dateTarget)
dayTarget = datetime.strptime(dateTarget,"%d")
print("Day Target = ",dayTarget)
Console output:
Today is = 10/01/2020
indexTarget is = 281
Target date is = 10/08/2020
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\tkinter\__init__.py", line 1702, in __call__
return self.func(*args)
File "C:\Users\micha\source\repos\makeAReservation\makeAReservation\makeAReservation.py", line 183,
in actual_time
alarm(set_alarm_timer)
File "C:\Users\micha\source\repos\makeAReservation\makeAReservation\makeAReservation.py", line 173, in alarm
makeAReservation()
File "C:\Users\micha\source\repos\makeAReservation\makeAReservation\makeAReservation.py", line 62, in makeAReservation
getIndex4TagetDate()
File "C:\Users\micha\source\repos\makeAReservation\makeAReservation\makeAReservation.py", line 46, in getIndex4TagetDate
dayTarget = datetime.strptime(dateTarget,"%d")
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\_strptime.py", line 565, in _strptime_datetime
tt, fraction = _strptime(data_string, format)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\_strptime.py", line 365, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: /08/2020
Your dateTarget is a string, however datetime.strprime is for creating date object from string. Since you already have a string, you can just slice it and then print.
print("Today is = ",re.sub('[^!-~]+',' ',calendarData[i]).strip())
print("indexTarget is = ",indexTarget)
dateTarget = re.sub('[^!-~]+',' ',calendarData[indexTarget]).strip()
print("Target date is = ",dateTarget)
dayTarget = dateTarget[:2]
print("Day Target = ",dayTarget)
I have been trying to calculate Stocastic RSI on multi-index dataframe by using "groupby" symbol, and then calling inline function. Following is the code:
df['fastk'],df['fastd'] = df.groupby('Symbol')['Close'].apply(lambda y: talib.STOCHRSI(y, timeperiod=14, fastk_period=5, fastd_period=3, fastd_matype=0))
The error is:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\jafre\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\frame.py", line 2519, in __setitem__
self._set_item(key, value)
File "C:\Users\jafre\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\frame.py", line 2585, in _set_item
value = self._sanitize_column(key, value)
File "C:\Users\jafre\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\frame.py", line 2760, in _sanitize_column
value = _sanitize_index(value, self.index, copy=False)
File "C:\Users\jafre\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\series.py", line 3121, in _sanitize_index
raise ValueError('Length of values does not match length of ' 'index')
ValueError: Length of values does not match length of index
Can someone help with the changes in the code?
I am getting an input file which contains datetime. My purpose is to read the file and filter the db.
Input file sample
2019-05-31 15:21:53
Below code work fine in my PC (python 3.6.4)
f=open("lastRun.txt", "r+")
lastHour=(f.read())
f.close()
lHourDate = datetime.datetime.strptime(lastHour, '%Y-%m-%d %H:%M:%S')
print(lHourDate)
however in production we have (Pyhton 3.6.7)
where the same code is throwing below error.
Traceback (most recent call last):
File "hourlyLoadEscenic.py", line 37, in <module>
lHourDate = datetime.strptime(lastHour, "%Y-%m-%d %H:%M:%S")
File "/usr/local/lib/python3.6/_strptime.py", line 565, in _strptime_datetime
tt, fraction = _strptime(data_string, format)
File "/usr/local/lib/python3.6/_strptime.py", line 365, in _strptime
data_string[found.end():])
ValueError: unconverted data remains:
I can do 'print' and see the lastHour has the input field. but in date conversion it is throwing error.
Please suggest
I am getting error message when im trying to write to my excel file.
On this line: df_Percent_Change.to_excel(writer, sheet_name=x, startcol=8)
Traceback (most recent call last):
File "/Users/david.soderstrom/Dropbox (Diagona)/DS/Python/Byggmarknad_index/Byggmarknad_index.py", line 125, in
df_Percent_Change.to_excel(writer, sheet_name=x, startcol=8)
File "/Users/david.soderstrom/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py", line 1766, in to_excel
engine=engine)
File "/Users/david.soderstrom/anaconda3/lib/python3.6/site-packages/pandas/io/formats/excel.py", line 652, in write
freeze_panes=freeze_panes)
File "/Users/david.soderstrom/anaconda3/lib/python3.6/site-packages/pandas/io/excel.py", line 1742, in write_cells
wks = self.book.add_worksheet(sheet_name)
File "/Users/david.soderstrom/anaconda3/lib/python3.6/site-packages/xlsxwriter/workbook.py", line 179, in add_worksheet
return self._add_sheet(name, worksheet_class=worksheet_class)
File "/Users/david.soderstrom/anaconda3/lib/python3.6/site-packages/xlsxwriter/workbook.py", line 666, in _add_sheet
name = self._check_sheetname(name, isinstance(worksheet, Chartsheet))
File "/Users/david.soderstrom/anaconda3/lib/python3.6/site-packages/xlsxwriter/workbook.py", line 717, in _check_sheetname
if len(sheetname) > 31:
TypeError: object of type 'int' has no len()
Exception ignored in: >
Traceback (most recent call last):
File "/Users/david.soderstrom/anaconda3/lib/python3.6/site-packages/xlsxwriter/workbook.py", line 154, in del
Exception: Exception caught in workbook destructor. Explicit close() may be required for workbook.
# Calculate and save the percent change for each asset
if 'Percent_Change' not in excel_db.columns:
print('Percent_Change does not exist in excel file.')
print('Calculating...')
# Loop through and read each sheet
x = 0
for x in range(countSheets):
# Read in data for the calculation
data = pd.read_excel('databas.xlsx', sheet_name=x, index_col='Date')
# Calculate the percent change from day to day
Percent_Change = data['Adj Close'].pct_change()*100
print(type(Percent_Change))
df_Percent_Change = pd.DataFrame(Percent_Change)
print(type(df_Percent_Change))
writer = pd.ExcelWriter('databas.xlsx', engine='xlsxwriter')
df_Percent_Change.to_excel(writer, sheet_name=x, startcol=8)
# Save the result
writer.save()
writer.close()
x += 1