I have the following dataframe (named test) in pandas:
Group 1 Group 2 Species Adj. P-value
0 a b Parabacteroides goldsteinii 7
1 a b Parabacteroides johnsonii 8
2 a b Parabacteroides merdae 9
3 a b Parabacteroides sp 10
4 c d Bacteroides coprocola 1
5 c d Bacteroides dorei 2
I would like to transform this table in latex format, but with the repeated values in Group 1 and Group 2 centred (see figure below for an example). In latex this is done with the package \multirow, and df.to_latex has a parameter called multirow to enable this (to_latex)
However, a MultiIndex has to be created in order to use the multirow option in to_latex.
So I did this:
test.index = pd.MultiIndex.from_frame(test[["Group 1","Group 2"]])
test = test.drop(["Group 1","Group 2"], axis=1)
test
Species Adj. P-value
Group 1 Group 2
a b Parabacteroides goldsteinii 7
b Parabacteroides johnsonii 8
b Parabacteroides merdae 9
b Parabacteroides sp 10
c d Bacteroides coprocola 1
d Bacteroides dorei 2
And finally I stored the table:
test.to_latex("la_tex_tab.txt",multirow=True, index=True,float_format="{:0.3f}".format).
However, this yields:
It works just for level 0 (Group 1) but not for level 1 (Group 2) of the MultiIndex. Do you have any suggestions about how to avoid the repetitions of the values b and d in the MultiIndex?
Thank you.
Kind of a hack if you want:
test['Group 2'] = test['Group 2'].mask(test['Group 2'].duplicated(),'')
test.set_index(["Group 1","Group 2"])
Species Adj. P-value
Group 1 Group 2
a b Parabacteroides goldsteinii 7
Parabacteroides johnsonii 8
Parabacteroides merdae 9
Parabacteroides sp 10
c d Bacteroides coprocola 1
Bacteroides dorei 2
We can do it for display only by use assign with blank column
test = test.assign(help='').set_index('help',append=True).drop(["Group 1","Group 2"], axis=1)
Related
I have a dataframe like below in which I need to replace the 0s with the mean of the rows where the parent_key matches the self_key.
Input DataFrame: df= pd.DataFrame ({'self_key':['a','b','c','d','e','e','e','f','f','f'],'parent_key':[np.nan,'a','b','b','c','c','c','d','d','d'], 'value':[0,0,0,0,4,6,14,12,8,22],'level':[1,2,3,3,4,4,4,4,4,4]})
The row 3 has self_key of 'd' so I would need to replace its 0 value in column 'value' with the mean of rows 7,8,9 to fill with the correct value of 14. Since the lower levels feed into the higher levels I would need to do it from lowest level to highest to fill out the dataframe as well but when I do the below code it doesn't work and I get the error "ValueError: Grouper for '<class 'pandas.core.frame.DataFrame'>' not 1-dimensional". How can I fill in the 0s with the means from lowest level to highest?
df['value']=np.where((df['value']==0) & (df['level']==3), df['value'].groupby(df.where(df['parent_key']==df['self_key'])).transform('mean'), df['value'])
Input
self_key parent_key value level
0 a NaN 0 1
1 b a 0 2
2 c b 0 3
3 d b 0 3
4 e c 4 4
5 e c 6 4
6 e c 14 4
7 f d 12 4
8 f d 8 4
9 f d 22 4
My approach is to repeat the above code 3 times and change the level from 3 to 2 to 1, but its not working for even level 3.
Expected Ouput:
self_key parent_key value level
0 a NaN 11 1
1 b a 11 2
2 c b 8 3
3 d b 14 3
4 e c 4 4
5 e c 6 4
6 e c 14 4
7 f d 12 4
8 f d 8 4
9 f d 22 4
If I understand your problem correctly, you are trying to compute mean in a bottom-up fashion by filtering dataframe on certain keys. If so, then following should solve it:
for l in range(df["level"].max()-1, 0, -1):
df_sub = df[(df["level"] == l) & (df["value"] == 0)]
self_keys = df_sub["self_key"].tolist()
for k in self_keys:
df.loc[df_sub[df_sub["self_key"] == k].index, "value"] = df[df["parent_key"] == k]["value"].mean()
[Out]:
self_key parent_key value level
0 a 11 1
1 b a 11 2
2 c b 8 3
3 d b 14 3
4 e c 4 4
5 e c 6 4
6 e c 14 4
7 f d 12 4
8 f d 8 4
9 f d 22 4
I have the following DataFrame:
Segments Airline_pct_tesco Airline_pct_asda food_pct_tesco food_pct_asda Airline_diff food_diff
A 1 2 4 2 -1 2
B 2 2 4 4 0 0
c 10 5 12 10 5 2
I want to convert it to this format:
Segments Category Asda% Tesco% Diff%
A Airline 2 1 -1
b Food 4 4 0
c Airline 5 10 5
A Food 2 4 2
(only partially showing). Note
category is the col name without the '_pct_tesco' or '_diff' or '_pct_asda'
I am unsure how to go about this - I have tried transform but I just don't know how I can get it in a way which is easy for any user to use. I am doing this in pandas and am not sure how to even begin! The Asda% are related to '_pct_asda' columns and same for diff and tesco columns respectively..
Let's try set_index to save columns, then create a MultiIndex.from_frame using str.extract on the columns to create a MultiIndex based on the values before a list of suffixes, then stack to go to long-form.
new_df = df.set_index('Segments')
# Define allowed suffixes here
suffixes = ['_pct_asda', '_pct_tesco', '_diff']
# Extract Values
new_df.columns = (
pd.MultiIndex.from_frame(
new_df.columns.str.extract(rf'(.*?)({"|".join(suffixes)})'),
names=['Category', None]
)
)
new_df = new_df.stack(0)
new_df:
_diff _pct_asda _pct_tesco
Segments Category
A Airline -1 2 1
food 2 2 4
B Airline 0 2 2
food 0 4 4
c Airline 5 5 10
food 2 10 12
To get cleaner output add reset_index + rename to fix column names and index and also re-order columns.
new_df = new_df.reset_index().rename(columns={
'_pct_asda': 'Asda%',
'_pct_tesco': 'Tesco%',
'_diff': 'Diff%'
})[['Segments', 'Category', 'Asda%', 'Tesco%', 'Diff%']]
new_df:
Segments Category Asda% Tesco% Diff%
0 A Airline 2 1 -1
1 A food 2 4 2
2 B Airline 2 2 0
3 B food 4 4 0
4 c Airline 5 10 5
5 c food 10 12 2
I have the following DataFrame
Input:
A B C D E
2 3 4 5 6
1 1 2 3 2
2 3 4 5 6
I want to add a new column that has the minimum of A, B and C for that row
Output:
A B C D E Goal
2 3 4 5 6 2
1 1 2 3 2 1
2 3 4 5 6 2
I have tried to use
df = df[['A','B','C]].min()
but I get errors about hashing lists and also I think this will be the min of the whole column I only want the min of the row for those specific columns.
How can I best accomplish this?
Use min along the columns with axis=1
Inline solution that produces copy that doesn't alter the original
df.assign(Goal=lambda d: d[['A', 'B', 'C']].min(1))
A B C D E Goal
0 2 3 4 5 6 2
1 1 1 2 3 2 1
2 2 3 4 5 6 2
Same answer put different
Add column to existing dataframe
new = df[['A', 'B', 'C']].min(axis=1)
df['Goal'] = new
df
A B C D E Goal
0 2 3 4 5 6 2
1 1 1 2 3 2 1
2 2 3 4 5 6 2
Add axis = 1 to your min
df['Goal'] = df[['A','B','C']].min(axis = 1)
you have to define an axis across which you are applying the min function, which would be 1 (columns).
df['ABC_row_min'] = df[['A', 'B', 'C']].min(axis = 1)
I have dataframe1 with columns a,b,c,d with 5 rows.
I also have another dataframe2 with columns e,f,g,h
Let's say I want to copy columns a,b in row 3 from dataframe1 to columns f,g in row 3 at dataframe2.
I tried to use this code:
dataframe2.loc[3,['f','g']] = dataframe1.loc[3,['a','b']].
The results was NaN in dataframe2.
Any ideas how can I solve it?
One idea is convert to numpy array for avoid alignment data by columns names:
dataframe2.loc[3,['f','g']] = dataframe1.loc[3,['a','b']].values
Sample:
dataframe1 = pd.DataFrame({'a':list('abcdef'),
'b':[4,5,4,5,5,4],
'c':[7,8,9,4,2,3]})
print (dataframe1)
a b c
0 a 4 7
1 b 5 8
2 c 4 9
3 d 5 4
4 e 5 2
5 f 4 3
dataframe2 = pd.DataFrame({'f':list('HIJK'),
'g':[0,0,7,1],
'h':[0,1,0,1]})
print (dataframe2)
f g h
0 H 0 0
1 I 0 1
2 J 7 0
3 K 1 1
dataframe2.loc[3,['f','g']] = dataframe1.loc[3,['a','b']].values
print (dataframe2)
f g h
0 H 0 0
1 I 0 1
2 J 7 0
3 d 5 1
Given the following data frame:
import pandas as pd
df=pd.DataFrame({'A':['A','A','A','B','B','B'],
'B':[1,1,2,1,1,1],
'C':[2,4,6,3,5,7]})
df
A B C
0 A 1 2
1 A 1 4
2 A 2 6
3 B 1 3
4 B 1 5
5 B 1 7
Wherever there are duplicate rows per columns 'A' and 'B', I'd like to combine those rows and sum the value under column 'C' like this:
A B C
0 A 1 6
2 A 2 6
3 B 1 15
So far, I can at least identify the duplicates like this:
df['Dup']=df.duplicated(['A','B'],keep=False)
Thanks in advance!
use groupby() and sum():
In [94]: df.groupby(['A','B']).sum().reset_index()
Out[94]:
A B C
0 A 1 6
1 A 2 6
2 B 1 15