Pandas multi-column match and then backward shift to find value - python-3.x

I would really appreciate your help with this. I'm a Pandas python noob and have been thrown into the deepend with this problem.
I searched through around 100 different queries on this website and cannot find something befitting. The closest I got was applying boolean masks.
Please click on the text below to find my dataframe.
I would like to run a query on the dataset to find the previous row where the 'AwayTeam' string is found in the 'HomeTeam' column -> I would then like to pull in the value of 'home_form' for that matching incidence as an additional column
Date HomeTeam AwayTeam home_form away_form new_column
25/08/2019 Strasbourg Rennes 1.0 3.0 Nan (Row 25, just an example)
01/09/2019 Rennes Nice 3.0 3.0 3.0 (Row 37, just an example)
I want to pull in the previous 'away_form' value for the last row where a HomeTeam appeared in the AwayTeam column

This is not a complete solution, but I think I found a way to help you make some progress.
Here the steps:
Create a sample dataframe, just for illustration
Convert the 'HomeTeam' column into a list … this is the target column.
Create an empty list to store the results of searching 'HomeTeam' column
Loop through the teams in the 'AwayTeam' column
Use Python's list.index() method to return the index of the match … but use a try-except just in case you don't find a match.
Store the result into list
When finished with the for-loop, add the list as a new column in the pandas dataframe.
import pandas as pd
import numpy as np
# create sample dataframe
df = pd.DataFrame({
'Date': ['2019-08-18', '2019-08-25'],
'HomeTeam': ['Rennes', 'Strasbourg'],
'AwayTeam': ['Paris SG', 'Rennes'],
'home_form': [np.NaN, 1.0],
'away_form': [np.NaN, 3.0],
})
# convert your 'HomeTeam' column into a Python list
list_HomeTeam = list(df['HomeTeam'])
print(list_HomeTeam)
# create an empty list to capture the index position of matches in 'HomeTeam'
list_results_in_home = []
# loop through each team in the 'AwayTeam column'
for each_team in df['AwayTeam']:
# if you find a match in the list, store index as a result
try:
result = list_HomeTeam.index(each_team)
# if you don't find a match, store a string
except:
result = 'team not in list'
# add the result to the list that is capturing the index position in 'HomeTeam'
list_results_in_home.append(result)
print(list_index_home)
# add column to dataframe with the index position
df['index_match_in_HomeTeam'] = list_results_in_home

Related

How do I drop complete rows (including all values in it) that contain a certain value in my Pandas dataframe?

I'm trying to write a python script that finds unique values (names) and reports the frequency of their occurrence, making use of Pandas library. There's a total of around 90 unique names, which I've anonymised in the head of the dataframe pasted below.
,1,2,3,4,5
0,monday09-01-2022,tuesday10-01-2022,wednesday11-01-2022,thursday12-01-2022,friday13-01-2022
1,Anonymous 1,Anonymous 1,Anonymous 1,Anonymous 1,
2,Anonymous 2,Anonymous 4,Anonymous 5,Anonymous 5,Anonymous 5
3,Anonymous 3,Anonymous 3,,Anonymous 6,Anonymous 3
4,,,,,
I'm trying to drop any row (the full row) that contains the regex expression "^monday.*", intending to indicate the word "monday" followed by any other number of random characters. I want to drop/deselect any cell/value within that row.
To achieve this goal, I've tried using the line of code below (and many other approaches I found on SO).
df = df[df[1].str.contains("^monday.*", case = True, regex=True) == False]
To clarify, I'm trying to search values of column "1" for the value "^.monday.*" and then deselecting the rows and all values in that row that match the regex expression. I've succesfully removed "monday09-01-2022" and "tuesday10-01-2022" etc.. but I'm also losing random names that are not in the matching rows.
Any help would be very much appreciated! Thank you!

Splitting the data of one excel column into two columns sing python

I have problem of splitting the content of one excel column which contains numbers and letters into two columns the numbers in one column and the letters in the other.
As can you see in the first photo there is no space between the numbers and the letters, but the good thing is the letters are always "ms". I need a method split them as in the second photo.
Before
After
I tried to use the replace but it did not work. it did not split them.
Is there any other method.
You can use the extract method. Here is an example:
df = pd.DataFrame({'time': ['34ms', '239ms', '126ms']})
df[['time', 'unit']] = df['time'].str.extract('(\d+)(\D+)')
# convert time column into integer
df['time'] = df['time'].astype(int)
print(df)
# output:
# time unit
# 0 343 ms
# 1 239 ms
# 2 126 ms
It is pretty simple.
You need to use pandas.Series.str.split
Attaching the Syntax here :- pandas.Series.str.split
The Code should be
import pandas as pd
data_before = {'data' : ['34ms','56ms','2435ms']}
df = pd.DataFrame(data_before)
result = df['data'].str.split(pat='(\d+)',expand=True)
result = result.loc[:,[1,2]]
result.rename(columns={1:'number', 2:'string'}, inplace=True)
Output : -
print(result)
Output

{Python} - [Pandas] - How sum columns by condition less than in columns name

First explaining the dataframe, the values of columns '0-156', '156-234', '234-546' .... '> 76830' is the percentage distribution for each range of distances in meters, totaling 100%.
Column 'Cell Name' refers to the data element of the other columns and the column 'Distance' is the column that will trigger the desired sum.
I need to sum the values of the columns '0-156', '156-234', '234-546' .... '> 76830' which are less than the value of the 'Distance' (Meters) column.
Below creation code for testing.
import pandas as pd
# initialize list of lists
data = [['Test1',0.36516562,19.065996,49.15094,24.344206,0.49186087,1.24217,5.2812457,0.05841639,0,0,0,0,158.4122868],
['Test2',0.20406325,10.664485,48.70978,14.885571,0.46103176,8.75815,14.200708,2.1162114,0,0,0,0,192.553074],
['Test3',0.13483211,0.6521175,6.124511,41.61725,45.0036,5.405257,1.0494527,0.012979688,0,0,0,0,1759.480042]
]
# Create the pandas DataFrame
df = pd.DataFrame(data, columns = ['Cell Name','0-156','156-234','234-546','546-1014','1014-1950','1950-3510','3510-6630','6630-14430','14430-30030','30030-53430','53430-76830','>76830','Distance'])
Example of what should be done:
The value of column 'Distance' = 158.412286772863 therefore would have to sum the values <= of the following columns, 0-156, '156-234' totalizing 19.43116162 %.
Thanks so much!
As I understand it, you want to sum up all the percentage values in a row, where the lower value of the column-description (in case of '0-156' it would be 0, in case of '156-234' it would be 156, and so on...) is smaller than the value in the distance column.
First I would suggest, that you transform your string-like column-names into values, as an example:
lowerlimit=df.columns[2]
>>'156-234'
Then read the string only till the '-' and make it a number
int(lowerlimit[:lowerlimit.find('-')])
>> 156
You can loop this through all your columns and make a new row for the lower limits.
For a bit more simplicity I left out the first column for your example, and added another first row with the lower limits of each column, that you could generate as described above. Then this code works:
data = [[0,156,234,546,1014,1950,3510,6630,11430,30030,53430,76830,1e-23],[0.36516562,19.065996,49.15094,24.344206,0.49186087,1.24217,5.2812457,0.05841639,0,0,0,0,158.4122868],
[0.20406325,10.664485,48.70978,14.885571,0.46103176,8.75815,14.200708,2.1162114,0,0,0,0,192.553074],
[0.13483211,0.6521175,6.124511,41.61725,45.0036,5.405257,1.0494527,0.012979688,0,0,0,0,1759.480042]
]
# Create the pandas DataFrame
df = pd.DataFrame(data, columns = ['0-156','156-234','234-546','546-1014','1014-1950','1950-3510','3510-6630','6630-14430','14430-30030','30030-53430','53430-76830','76830-','Distance'])
df['lastindex']=None
df['sum']=None
After creating basically your dataframe, I add two columns 'lastindex' and 'sum'.
Then I am searching for the last index in every row, that is has its lower limit below the distance given in that row (df.iloc[x,-3]); afterwards I'm summing up the respective columns in that row.
for i in np.arange(1,len(df)):
df.at[i,'lastindex']=np.where(df.iloc[0,:-3]<df.iloc[i,-3])[0][-1]
df.at[i,'sum']=sum(df.iloc[i][0:df.at[i,'lastindex']+1])
I hope, this is helpful. Best, lepakk

Take odd one out from a list when comparing with a filtered column using python

I have an excel as shown below:
Input File
Now I want to filter fruits first from "Items" column and check which one in list of "list" column is not present in the list. For example: here "grapes" is not present in "Name" column. So I want grapes as output in next column as shown below.
Expected Output Shown
The same is to be done for many by filtering each items one by one as I have many items.
Please suggest or give some hints so that i can start this code.
I am naming the excel as Book1
import pandas as pd
frame = pd.read_excel("Book1.xlsx")
frame_list_as_String = frame.list.tolist()
frame_list = [x.split(',') for x in frame_list_as_String]
frame_Name = frame.Name.tolist()
frame_col3=[]
for item in frame_list :
frame_col3.append(list(set(items)-set(frame_Name)))
frame["col3"]=frame_col3
frame.to_excel("df.xlsx", index = False)

pandas iterate rows and then break until condition

I have a column that's unorganized like this;
Name
Jack
James
Riddick
Random value
Another random value
What I'm trying to do is get only the names from this column, but struggling to find a way to differentiate real names to random values. Fortunately the names are all together, and the random values are all together as well. The only thing I can do is iterate the rows until it gets to 'Random value' and then break off.
I've tried using lambda's for this but with no success as I don't think there's a way to break. And I'm not sure how comprehension could work in this case.
Here's the example I've been trying to play with;
df['Name'] = df['Name'].map(lambda x: True if x != 'Random value' else break)
But the above doesn't work. Any suggestions on what could work based on what I'm trying to achieve? Thanks.
Find index of row containing 'Random value':
index_split = df[df.Name == 'Random value'].index.values[0]
Save your random values column for use later if you want:
random_values = df.iloc[index_split+1:,].values[0]
Remove random values from the Names column:
df = df[0:index_split]

Resources