Add columns based on a changing number of rows below - python-3.x

Question

I'm trying to solve a machine learning problem for an university project. As a input I got a excel table.
It is needed to access information below specific rows (condition: df[c1] !=0) and create new columns with it. But the number of rows after the specific row is not fixed.
There are various pandas functions I tried to get running (e.g.: While-Loops combined with iloc, iterrows.) But nothing seemed to work. Now I wonder if I need to create a function where I create a new df for every group below each top element. I asume there must be a better option. I use Python 3.6 and Pandas 0.25.0.
I try to get the following result.
Input:
| name | c1 | c2 |
|------|-------|--------------|
| ab | 1 | info |
| tz | 0 | more info |
| ka | 0 | more info |
| cd | 2 | info |
| zz | 0 | more info |
The output should look like this:
Output:
| name | c1 | c2 | tz3 | ka4 | zz5 |
|------|-------|--------------|-----------|-----------|------------|
| ab | 1 | info | more info | more info | |
| tz | 0 | more info | | | |
| ka | 0 | more info | | | |
| cd | 2 | info | | | more info |
| zz | 0 | more info | | | |

Answer 1

You can do this as follows:
# make sure c1 is of type int (if it isn't already)
# if it is string, just change the comparison further below
df['c1']= df['c1'].astype('int32')
# create two temporary aux columns in the original dataframe
# the first contains 1 for each row where c1 is nonzero
df['nonzero']= (df['c1'] != 0).astype('int')
# the second contains a "group index" to give
# all rows that belong together the same number
df['group']= df['nonzero'].cumsum()
# create a working copy from the original dataframe
df2= df[['c1', 'c2', 'group']].copy()
# add another column which contains the name of the
# column under which the text should appear
df2['col']= df['name'].where(df['nonzero']==0, 'c2')
# add a dummy column with all ones
# (needed to merge the original dataframe
# with the "transposed" dataframe later)
df2['nonzero']= 1
# now the main part
# use the prepared copy and index it on
# group, nonzero(1) and col
df3= df2[['group', 'nonzero', 'col', 'c2']].set_index(['group', 'nonzero', 'col'])
# unstack it, meaning col is "split off" to create a new column
# level (like pivoting), the rest remains in the index
df3= df3.unstack()
# now df3 has a multilevel column index
# to get rid of it and have regular column names
# just rename the columns and remove c2 which
# we get from the original dataframe
df3_names= ['{1}'.format(*tup) for tup in df3.columns]
df3.columns= df3_names
df3.drop(['c2'], axis='columns', inplace=True)
# df3 now contains the "transposed" infos in column c1
# which should appear in the row for which 'nonzero' contains 1
# to get this, use merge
result= df.merge(df3, left_on=['group', 'nonzero'], right_index=True, how='left')
# if you don't like the NaN values (for the rows with nonzero=0), use fillna
result.fillna('', inplace=True)
# remove the aux columns and the merged c2_1 column
# for c2_1 we can use the original c2 column from df
result.drop(['group', 'nonzero'], axis='columns', inplace=True)
# therefore we rename it to get the same naming schema
result.rename({'c2': 'c2_1'}, axis='columns', inplace=True)
The result looks like this:
Out[191]:
name c1 c2 ka tz zz
0 ab 1 info even more info more info
1 tz 0 more info
2 ka 0 even more info
3 cd 2 info more info
4 zz 0 more info
For this input data:
Out[166]:
name c1 c2
0 ab 1 info
1 tz 0 more info
2 ka 0 even more info
3 cd 2 info
4 zz 0 more info
# created by the following code:
import io
raw=""" name c1 c2
0 ab 1 info
1 tz 0 more_info
2 ka 0 even_more_info
3 cd 2 info
4 zz 0 more_info"""
df= pd.read_csv(io.StringIO(raw), sep='\s+', index_col=0)
df['c2']=df['c2'].str.replace('_', ' ')