Having a dataframe like this:
I would like to know what would be the most efficient way to transform it into this othe one:
I tried to generate all the combinations between Time column and days and then manually create the Value column by checking the given Day-Time cell, but Im sure it must be a more efficient way
IF the original index is not important for you,
You could also use the .melt() method which has the advantage of grouping the days so you have the values for 1 day after another:
df1 = df.melt(id_vars='Time', var_name='Day', value_name='Value')
Result:
index
Time
Day
Value
0
6am-2pm
Day1
15.4
1
2pm-10pm
Day1
15.0
2
10pm-6am
Day1
14.0
3
6am-2pm
Day2
13.4
4
2pm-10pm
Day2
2.1
5
10pm-6am
Day2
22.0
6
6am-2pm
Day3
45.0
7
2pm-10pm
Day3
3.4
8
10pm-6am
Day3
35.0
You could even rearrange the columns index like this to make it more readable in my own opinion:
df1 = df1.reindex(columns=['Day','Time','Value'])
Result:
index
Day
Time
Value
0
Day1
6am-2pm
15.4
1
Day1
2pm-10pm
15.0
2
Day1
10pm-6am
14.0
3
Day2
6am-2pm
13.4
4
Day2
2pm-10pm
2.1
5
Day2
10pm-6am
22.0
6
Day3
6am-2pm
45.0
7
Day3
2pm-10pm
3.4
8
Day3
10pm-6am
35.0
Use set_index and stack:
out = (df.set_index('Time').stack().rename_axis(index=['Time', 'Day'])
.rename('Value').reset_index())
print(out)
# Output
Time Day Value
0 6am – 2pm Day1 15.4
1 6am – 2pm Day2 13.4
2 6am – 2pm Day3 45.0
3 2pm – 10pm Day1 15.0
4 2pm – 10pm Day2 2.1
5 2pm – 10pm Day3 3.4
6 10pm – 6am Day1 14.0
7 10pm – 6am Day2 22.0
8 10pm – 6am Day3 35.0
Related
I have two time series data frame:
df1 = pd.DataFrame({'Date': [pd.to_datetime('1980-01-03'), pd.to_datetime('1980-01-04'),
pd.to_datetime('1980-01-05'), pd.to_datetime('1980-01-06'),
pd.to_datetime('1980-01-07'), pd.to_datetime('1980-01-8')],
'Temp': [13.5,10,14,12,10,9]})
df1
Date Temp
0 1980-01-03 13.5
1 1980-01-04 10.0
2 1980-01-05 14.0
3 1980-01-06 12.0
4 1980-01-07 10.0
5 1980-01-08 9.0
and
df2 = pd.DataFrame({'Date': [pd.to_datetime('1980-01-01'), pd.to_datetime('1980-01-02'),
pd.to_datetime('1980-01-03'), pd.to_datetime('1980-01-04')],
'Temp': [10,17,13.5,10]})
df2
Date Temp
0 1980-01-01 10.0
1 1980-01-02 17.0
2 1980-01-03 13.5
3 1980-01-04 10.0
Now my task is to join these data frames based on Dates such that the resultant data frame has the dates that are unique to both data frames and also has single entry for common (present in both data frames) dates and are arranged in proper date sequence.
To that effect I tried the following:
df = pd.concat([df1, df2])
df.reset_index().drop(columns = ['index'], axis = 1)
Date Temp
0 1980-01-03 13.5
1 1980-01-04 10.0
2 1980-01-05 14.0
3 1980-01-06 12.0
4 1980-01-07 10.0
5 1980-01-08 9.0
6 1980-01-01 10.0
7 1980-01-02 17.0
8 1980-01-03 13.5
9 1980-01-04
But this is incorrect result. What I am trying to get is:
Date Temp
0 1980-01-01 10.0
1 1980-01-02 17.0
2 1980-01-03 13.5
3 1980-01-04 10.0
4 1980-01-05 14.0
5 1980-01-06 12.0
6 1980-01-07 10.0
7 1980-01-08 9.0
What can I do? May be the pd.concat() is not the way to go?
A possible solution:
pd.merge(df1, df2, how="outer").sort_values(by="Date").reset_index(drop=True)
Output:
Date Temp
0 1980-01-01 10.0
1 1980-01-02 17.0
2 1980-01-03 13.5
3 1980-01-04 10.0
4 1980-01-05 14.0
5 1980-01-06 12.0
6 1980-01-07 10.0
7 1980-01-08 9.0
I have two dataframes and I want to accumulate the value of one of the dataframes in the other. How can I do it?
Dataframe 1:
Product
Amount
Price
Total
A
1
12.0
15
B
4
20.0
15
C
2
4.0
15
D
5
30.0
15
Dataframe 2:
Product
Amount
Price
B
3
20.0
C
2
4.0
Result:
Product
Amount
Price
Total
A
1
12.0
15
B
7
20.0
15
C
4
4.0
15
D
5
30.0
15
Thanks!
Use concat with aggregate sum - necessary all numeric columns without Product:
df = pd.concat([df1, df2]).groupby('Product', as_index=False).sum()
print (df)
Product Amount Price Total
0 A 1 12.0 15.0
1 B 7 40.0 15.0
2 C 4 8.0 15.0
3 D 5 30.0 15.0
I hope you can help me with this:
Trying to append missing weekends to the df['StartDate'] column and show the rest of the columns with the data except for Hours to show 0 or NaN
I don't need to see every single missing date between each displayed date in df['StartDate']. Need to only add the weekends 'Saturday' and 'Sunday' where ever they are missing.
Original Dataframe:
EmployeeId StartDate weekday Hours
111 1/20/2017 Friday 6
111 1/25/2017 Wednesday 5
111 1/30/2017 Monday 2
Final output would like this;
Desired Final output
EmployeeId StartDate weekday Hours
111 1/20/2017 Friday 6
111 1/21/2017 Saturday NaN
111 1/22/2017 Sunday NaN
111 1/25/2017 Wednesday 5
111 1/28/2017 Saturday NaN
111 1/29/2017 Sunday NaN
111 1/30/2017 Monday 2
One way is to create a separate data frame with the min and max values from your dataframe and just concatenate both frames together after filtering on weekends, we can handle duplicate values by dropping them and setting keep = 'first' which will keep the values from your first df.
s = pd.DataFrame(
{"StartDate": pd.date_range(df.StartDate.min(), df.StartDate.max(), freq="D")}
)
s["weekday"] = s.StartDate.dt.day_name()
s = s.loc[s["weekday"].isin(["Saturday", "Sunday"])]
df_new = (
pd.concat([df, s], sort=False)
.drop_duplicates(keep="first")
.sort_values("StartDate")
)
print(df_new)
EmployeeId StartDate weekday Hours
0 111.0 2017-01-20 Friday 6.0
1 NaN 2017-01-21 Saturday NaN
2 NaN 2017-01-22 Sunday NaN
1 111.0 2017-01-25 Wednesday 5.0
8 NaN 2017-01-28 Saturday NaN
9 NaN 2017-01-29 Sunday NaN
2 111.0 2017-01-30 Monday 2.0
to fill in NaN Employee IDs with the ones above them you can use fillna and ffill
df_new['EmployeeId'] = df_new['EmployeeId'].fillna(df_new['EmployeeId'].ffill())
print(df_new)
EmployeeId StartDate weekday Hours
0 111.0 2017-01-20 Friday 6.0
1 111.0 2017-01-21 Saturday NaN
2 111.0 2017-01-22 Sunday NaN
1 111.0 2017-01-25 Wednesday 5.0
8 111.0 2017-01-28 Saturday NaN
9 111.0 2017-01-29 Sunday NaN
2 111.0 2017-01-30 Monday 2.0
I am trying to create a report that is grouped by day of week for each year.
I have a df that looks like this:
s1 s2 srd
dt
2004-02-04 11:21:00 2365.79 2372.37 -7.0
2004-02-05 10:15:00 2365.79 2368.03 -2.0
2004-02-17 06:43:00 2421.05 2425.26 -4.0
2004-02-17 12:43:00 2418.42 2420.53 -2.0
2004-02-17 12:44:00 2420.39 2420.53 -0.0
The dt index is in datetime format.
What I am looking for is a dataframe that looks like this (I only need srd column and function to group can be anything, like sum, count, etc.):
srd
dayOfWeek year
Mon 2004 10
2005 11
2006 8
2007 120
Tues 2004 105
2005 105
I have tried dayOfWeekDf = df.resample('B') , but I get a dataframe that looks like it is split by week number.
I also tried df.groupby([df.index.weekday, df.index.year])['srd'].transform('sum'), but it does not even group for some reason, as I get the following (Feb 17th appears 3 times).
srd
dt
2004-02-04 11:21:00 81.0
2004-02-05 10:15:00 203.0
2004-02-17 06:43:00 37.0
2004-02-17 12:43:00 37.0
2004-02-17 12:44:00 37.0
If you want the dayOfWeek and year names in the index, you can assign them:
>>> df.assign(year=df.index.year, dayOfWeek = df.index.weekday_name).groupby(['dayOfWeek','year']).srd.sum()
dayOfWeek year
Thursday 2004 -2.0
Tuesday 2004 -6.0
Wednesday 2004 -7.0
Name: srd, dtype: float64
Otherwise, you can use the way you were doing, but omit the transform:
>>> df.groupby([df.index.weekday_name, df.index.year])['srd'].sum()
dt dt
Thursday 2004 -2.0
Tuesday 2004 -6.0
Wednesday 2004 -7.0
Name: srd, dtype: float64
I am working with this dataset:
TPdata:
id Tp1 Sp2 time
A 1 7 08:00:00
B 2 8 09:00:00
C 3 9 18:30:00
D 4 10 20:00:00
E 5 11 08:00:00
F 6 12 09:00:00
I would like to change the entries 08:00:00 in column time to 'early'. I thought this would work but it isn't:
TPdata$time[TPdata$time == 18:30:00] <- "early"
Can anyone help?