I have df that with 3 columns on timestamps:
X. ...
01/01/2013 12:00:20 AM. ...
so I have been trying to convert these columns into the DateTime format for some further analysis
When I run:
df.dtype()
the info comes back with each of these columns as objects. I have been reading the data in from a csv so they should be string objects.
When converting them to DateTime I have been using:
df['X'] = pd.to_datetime(df['X'])
and
df['X'] = df['X'].astype('datetime64[ns]')
But in every case, the kernel just keeps running and I am not getting anywhere... I want to be able to use these dates and times to calculate the difference between timestamp columns in minutes and such.
Any help would be greatly appreciated. Thank You.
Here is a full example that works with me.You can try it out in your own setup:
import pandas as pd
df=pd.DataFrame([["1/1/2016 12:00:20 AM","3/1/2016"],
["6/15/2016 4:00:20 AM","7/14/2016"],
["7/14/2016 11:00:20 AM","8/15/2016"],
["8/7/2016 00:00:20 AM","9/6/2016"]]
,columns=['X','Y'])
print(df)
#convert one column
df['X'] = pd.to_datetime(df['X'])
print(df)
#convert all columns
df[df.columns] = df[df.columns].apply(pd.to_datetime)
print(df)
Related
I have an Excel .xlsb sheet with data, some columns have number as output data, other columns should have dates as output. After uploading the data in Python, some columns have a number in stead of date. How can I change the format of the number in that specific column to a date?
I use Pandas and ddf
The output of the dataframe of column date of birth ('dob_l1') shows '12150', which should be date '6-4-1933'.
I tried to solve this, but unfortunately I only managed to get the date '2050-01-12' which is incorrect.
I used code 'ddf['nwdob_l1'] = pd.to_datetime(ddf['dob_l1'], format='%d%m%y',errors='coerce')'
Who can help me. I was happy to received some good feedback from joe90. He showed me a function that could help for singular dates:
import datetime
def xldate2date(xl):
# valid for dates from 1900-03-01
basedate = datetime.date(1899,12,30)
d = basedate + datetime.timedelta(days=xl)
return d
# Example:
# >>> print(xldate2date(44948))
# 2023-01-22
That is correct, however, I need to change all values in the column (> 500.000), so I cannot do that 1-by-1.
As that question is closed, I hereby open a new question.
Is there anyone who can help me to find the correct code to get the right date in the whole column?
When you read the data in using pandas there are tools for the dates. You want to use parse_dates
Documentation for read_excel
example:
import pandas as pd
df = pd.read_excel('file/path/the.xlsx', parse_dates=['Date'])
This will change the date to be datetime64 format which is better than a number.
How do I convert Excel date format to number in Python? I'm importing a number of Excel files into Pandas dataframe in a loop and some values are formatted incorrectly in Excel. For example, the number column is imported as date and I'm trying to convert this date value into numeric.
Original New
1912-04-26 00:00:00 4500
How do I convert the date value in original to the numeric value in new? I know this code can convert numeric to date, but is there any similar function that does the opposite?
df.loc[0]['Date']= xlrd.xldate_as_datetime(df.loc[0]['Date'], 0)
I tried to specify the data type when I read in the files and also tried to simply change the data type of the column to 'float' but both didn't work.
Thank you.
I found that the number means the number of days from 1900-01-00.
Following code is to calculate how many days passed from 1900-01-00 until the given date.
import pandas as pd
from datetime import datetime, timedelta
df = pd.DataFrame(
{
'date': ['1912-04-26 00:00:00'],
}
)
print(df)
# date
#0 1912-04-26 00:00:00
def date_to_int(given_date):
given_date = datetime.strptime(given_date, '%Y-%m-%d %H:%M:%S')
base_date = datetime(1900, 1, 1) - timedelta(days=2)
delta = given_date - base_date
return delta.days
df['date'] = df['date'].apply(date_to_int)
print(df)
# date
#0 4500
I am trying to parse data from a csv file, sort them by date and write the sorted dataframe in a new csv file.
Say we have a very simple csv file with date entries following the pattern day/month/year:
Date,Reference
15/11/2020,'001'
02/11/2020,'002'
10/11/2020,'003'
26/11/2020,'004'
23/10/2020,'005'
I read the csv into a Pandas dataframe. When I attempt to order the dataframe based on the dates in ascending order I expect the data to be ordered as follows:
23/10/2020,'005'
02/11/2020,'002'
10/11/2020,'003'
15/11/2020,'001'
26/11/2020,'004'
Sadly, this is not what I get.
If I attempt to convert the date to datetime and then sort, then some date entries are converted to the month/day/year (e.g. 2020-10-23 instead of 2020-23-10) which messes up the ordering:
date reference
2020-02-11 '002'
2020-10-11 '003'
2020-10-23 '005'
2020-11-15 '001'
2020-11-26 '004'
If I sort without converting to datetime, then the ordering is also wrong:
date reference
02/11/2020 '002'
10/11/2020 '003'
15/11/2020 '001'
23/10/2020 '005'
26/11/2020 '004'
Here is my code:
import pandas as pd
df = pd.read_csv('order_dates.csv',
header=0,
names=['date', 'reference'],
dayfirst=True)
df.reset_index(drop=True, inplace=True)
# df.date = pd.to_datetime(df.date)
df.sort_val
df.sort_values(by='date', ascending=True, inplace=True)
print(df)
df.to_csv('sorted.csv')
Why is sorting by date so hard? Can someone explain why the above sorting attempts fail?
Ideally, I would like the sorted.csv to have the date entries in the day/month/year format.
Try:
df.loc[:,'date'] = pd.to_datetime(df.loc[:, 'date'], format='%d/%m-%Y')
What you can do is to specify the datetime format while reading the csv file. To do this try that:
>>> df = pd.read_csv('filename.csv', parse_dates=['Date'],infer_datetime_format='%d/%m/%Y').sort_values(by='Date')
This will read your dates from csv and give you this output where dates are sorted.
Date Reference
4 2020-10-23 '005
1 2020-11-02 '002'
2 2020-11-10 '003'
0 2020-11-15 '001'
3 2020-11-26 '004'
What's left now is to simply change the formatting to the desired one
>>> df['Date'] = df['Date'].dt.strftime('%d/%m/%Y')
Keep in mind however that this will change the Date back to string (object)
>>> df
Date Reference
4 23/10/2020 '005
1 02/11/2020 '002'
2 10/11/2020 '003'
0 15/11/2020 '001'
3 26/11/2020 '004'
>>> df.dtypes
Date object
I have searched but not found excatly what I need. I have a dataframe which has 50 columns. The first one is a date dtype, the rest are floats dtypes.
Now I want to convert ONLY the float columns into integer but NOT the date column.
Can someone guide please?
When I slice the df like this df_sub1=df_sub.iloc[:, 1:].apply(np.int64) and then concat with the date column after, it crashes my laptop and did therefore not work. I hope there is a better way.
Well assuming that date is your first column
import pandas as pd
cols = df.columns
df[cols[1:]] = df[cols[1:]].apply(pd.to_numeric, errors='coerce')
you can do it like this.
new_df = df.drop(nameoffirstcolumn,1)
new_df.apply(np.int64)
then you can do something like.
final_df = pd.concat([df1['nameoffirstcolumn'],new_df], axis=1)
I have a dataframe called prices, with historical stocks prices for the following companies:
['APPLE', 'AMAZON', 'GOOGLE']
So far on, with the help of a friendly user, I was able to create a dataframe for each of this periods with the following code:
import pandas as pd
import numpy as np
from datetime import datetime, date
prices = pd.read_excel('database.xlsx')
companies=prices.columns
companies=list(companies)
del companies[0]
timestep = 250
prices_list = [prices[day:day + step] for day in range(len(prices) - step)]
Now, I need to evaluate the change in price for every period of 251 days (Price251/Price1; Price252/Price2; Price 253/Price and so on) for each one of the companies, and create a column for each one of them.
I would also like to put the column name dynamic, so I can replicate this to a much longer database.
So, I would get a dataframe similar to this:
open image here
Here you can find the dataframe head(3): Initial Dataframe
IIUC, try this:
def create_cols(df,num_dates):
for col in list(df)[1:]:
df['{}%'.format(col)] = - ((df['{}'.format(col)].shift(num_dates) - df['{}'.format(col)]) / df['{}'.format(col)].shift(num_dates)).shift(- num_dates)
return df
create_cols(prices,251)
you only would have to format the columns to percentages.