Average values with Pandas GroupBy - python-3.x

I have a data frame similar to this:
name | points | tries | game
Alfred | 3.4 | 2 | sudoku
Alfred | 3 | 1 | sudoku
Alfred | 2.1 | 4 | tetris
Barry | 2.7 | 3 | tetris
Barry | 1.1 | 2 | sudoku
Cathy | 4.1 | 2 | tetris
Cathy | 3.3 | 2 | tetris
I am trying to first group by Games and inside every different "game", group by "name" and calculate total "points", total "tries", and the average points per try.
I have been reading about groupby but I don't find the day to do all this and at the same time calculate points/try for each group.
Any help would be appreciated.
This is for trying to analyze more easily some csv files. I have been able to open the data, read it and do some simple groupby commands but this multiple selection and average calculation is driving me crazy.


If you are on pandas 0.25 or later and want to play with thew new NamedAgg:
result = df.groupby(['game', 'name']).agg(
total_points = pd.NamedAgg('points', 'sum'),
total_tries = pd.NamedAgg('tries', 'sum')
)
result['avg_point_per_try'] = result['total_points'] / result['total_tries']
If you are on pandas < 0.25, you just have to rename the columns manually:
result = df.groupby(['game', 'name']).sum()
result.columns = ['total_points', 'total_tries']
result['avg_point_per_try'] = result['total_points'] / result['total_tries']
Result:
total_points total_tries avg_point_per_try
game name
sudoku Alfred 6.4 3 2.133333
Barry 1.1 2 0.550000
tetris Alfred 2.1 4 0.525000
Barry 2.7 3 0.900000
Cathy 7.4 4 1.850000

Related

Counting 15's in Cribbage Hand

Background
This is a followup question to my previous finding a straight in a cribbage hand question and Counting Pairs in Cribbage Hand
Objective
Count the number of ways cards can be combined to a total of 15, then score 2 points for each pair. Ace worth 1, and J,Q,K are worth 10.
What I have Tried
So my first poke at a solution required 26 different formulas. Basically I checked each possible way to combine cards to see if the total was 15. 1 way to add 5 cards, 5 ways to add 4 cards, 10 ways to add 3 cards, and 10 ways to add 2 cards. I thought I had this licked until I realized I was only looking at combinations, I had not considered the fact that I had to cap the value of cards 11, 12, and 13 to 10. I initially tried an array formula something along the lines of:
MIN(MOD(B1:F1-1,13)+1,10)
But the problem with this is that MIN takes the minimum value of all results not the individual results compared to 10.
I then tried it with an IF function, which worked, but involved the use of CSE formula even wehen being used with SUMPRODUCT which is something I try to avoid when I can
IF(MOD(B1:F1-1,13)+1<11,MOD(B1:F1-1,13)+1,10)
Then I stumble on an answer to a question in code golf which I modified to lead me to this formula, which I kind of like for some strange reason, but its a bit long in repetitive use:
--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2)
My current working formulas are:
5 card check
=(SUMPRODUCT(--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2))=15)*2
4 card checks
=(SUM(AGGREGATE(15,6,--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2),{1,2,3,4}))=15)*2
=(SUM(AGGREGATE(15,6,--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2),{1,2,3,5}))=15)*2
=(SUM(AGGREGATE(15,6,--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2),{1,2,4,5}))=15)*2
=(SUM(AGGREGATE(15,6,--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2),{1,3,4,5}))=15)*2
=(SUM(AGGREGATE(15,6,--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2),{2,3,4,5}))=15)*2
3 card checks
same as 4 card checks using all combinations for 3 cards in the {1,2,3}.
There are 10 different combinations, so 10 different formulas.
The 2 card check was based on the solution by Tom in Counting Pairs in Cribbage Hand and all two cards are checked with a single formula. (yes it is CSE)
2 card check
{=SUM(--(--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2)+TRANSPOSE(--MID("01020304050607080910101010",1+(MOD(B1:F1-1,13)*2),2))=15))}
Question
Can the 3 and 4 card combination sum check be brought into a single formula similar to the 2 card check?
Is there a better way to convert cards 11,12,13 to a value of 10?
Sample Data
| B | C | D | E | F | POINTS
+----+----+----+----+----+
| 1 | 2 | 3 | 17 | 31 | <= 2 (all 5 add to 15)
| 1 | 2 | 3 | 17 | 32 | <= 2 (Last 4 add to 15)
| 11 | 18 | 31 | 44 | 5 | <= 16 ( 4x(J+5), 4X(5+5+5) )
| 6 | 7 | 8 | 9 | 52 | <= 4 (6+9, 7+8)
| 1 | 3 | 7 | 8 | 52 | <= 2 (7+8)
| 2 | 3 | 7 | 9 | 52 | <= 2 (2+3+K)
| 2 | 4 | 6 | 23 | 52 | <= 0 (nothing add to 15)
Excel Version
Excel 2013

For 5:
=(SUMPRODUCT(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10))=15)*2
For 4:
=SUMPRODUCT(--(MMULT(INDEX(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)*ROW($1:$10)^0,ROW($1:$5),{1,2,3,4;1,2,3,5;1,2,4,5;1,3,4,5;2,3,4,5}),ROW($1:$4)^0)=15))*2
For 3
=SUMPRODUCT(--(MMULT(INDEX(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)*ROW($1:$10)^0,ROW($1:$10),{1,2,3;1,2,4;1,2,5;1,3,4;1,3,5;1,4,5;2,3,4;2,3,5;2,4,5;3,4,5}),ROW($1:$3)^0)=15))*2
For 2:
SUMPRODUCT(--((CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10))+(TRANSPOSE(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)))=15))
All together:
=(SUMPRODUCT(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10))=15)*2+
SUMPRODUCT(--(MMULT(INDEX(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)*ROW($1:$10)^0,ROW($1:$5),{1,2,3,4;1,2,3,5;1,2,4,5;1,3,4,5;2,3,4,5}),ROW($1:$4)^0)=15))*2+
SUMPRODUCT(--(MMULT(INDEX(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)*ROW($1:$10)^0,ROW($1:$10),{1,2,3;1,2,4;1,2,5;1,3,4;1,3,5;1,4,5;2,3,4;2,3,5;2,4,5;3,4,5}),ROW($1:$3)^0)=15))*2+
SUMPRODUCT(--((CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10))+(TRANSPOSE(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)))=15))
For older versions we need to "trick" INDEX into accepting the arrays as Row and Column References:
We do that by using N(IF({1},[thearray]))
=(SUMPRODUCT(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10))=15)*2+
SUMPRODUCT(--(MMULT(INDEX(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)*ROW($1:$10)^0,N(IF({1},ROW($1:$5))),N(IF({1},{1,2,3,4;1,2,3,5;1,2,4,5;1,3,4,5;2,3,4,5}))),ROW($1:$4)^0)=15))*2+
SUMPRODUCT(--(MMULT(INDEX(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)*ROW($1:$10)^0,N(IF({1},ROW($1:$10))),N(IF({1},{1,2,3;1,2,4;1,2,5;1,3,4;1,3,5;1,4,5;2,3,4;2,3,5;2,4,5;3,4,5}))),ROW($1:$3)^0)=15))*2+
SUMPRODUCT(--((CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10))+(TRANSPOSE(CHOOSE(MOD(A1:E1-1,13)+1,1,2,3,4,5,6,7,8,9,10,10,10,10)))=15))
This is a CSE That must be confirmed with Ctrl-Shift-Enter instead of Enter when exiting edit mode.

How do I search for a string in a pandas column and append to the row based on that string?

I have a pandas dataframe and I want to search through strings in column A, if there's a match I want to append 1 to a new column, if there is no match I want to append a 0.
My df currently looks like:
Column A | Column B | Column C
company one | 314 | 0.9
company one toast | 190 | 0.3
www.companyone | 380 | 0.87
companyone home | 850 | 0.1
toaster supplies | 1100 | 0.5
toast rack | 200 | 0.7
...
I'm trying to write a function which will read through column A, and if there's a match with either company one or companyone, then append 1 on the end of the row. If there is no match, then append 0. The output I'm looking for is:
Column A | Column B | Column C | Branded
company one | 314 | 0.9 | 1
company one toast | 190 | 0.3 | 1
www.companyone | 380 | 0.87 | 1
companyone home | 850 | 0.1 | 1
toaster supplies | 1100 | 0.5 | 0
toast rack | 200 | 0.7 | 0
...
I've tried this function:
def branded(table):
if 'company.*?one' in table[table['Column A']]:
table['Branded'] = 1
else:
table['Branded'] = 0
return table.head()
However I get a KeyError. I'm not sure what I'm missing though.

You can do it like this:
df['Branded'] = df['Column A'].str.contains('company.*?one')*1

The solution posted by zipa is better in my opinion. However, thought of sharing this which is a tweak version in case the strings to be looked for are entirely of different pattern. You can add the words to the list and then perform something similar:
import pandas as pd
df = pd.DataFrame({'column':['company one','companyone', 'company two']})
search = ['company one', 'companyone']
string_search = '|'.join(search)
df['flag'] = df['column'].str.contains(string_search)
df['flag'] = df['flag'].map({True: 1, False: 0})

How to produce a table with statistics according to categorical variable?

I have a lot of variables for which I need descriptive statistics (means). However, as columns I want to use values of categorical variable (AlcCons1).
I used the following code in order to do this:
tabstat Age25_29 Age30_34 ... SmokeY religAtndY, statistics( mean ) by(AlcCons1)
And I have got the result like this:
AlcCons1 | Age25_29 Age30_34 Age35_39 Age40_44 Age45_49 Age50_54 Age55_59
---------+----------------------------------------------------------------------
1 | .0987326 .0936242 .1243994 .1668614 .1579665 .1481626 .1258278
2 | .1037879 .11853 .1451863 .1415631 .1317288 .1231884 .1387164
3 | .0905679 .1151016 .1405161 .1624963 .1506231 .137278 .123246
4 | .0649853 .0716117 .1094201 .1606857 .1786286 .1630888 .1401794
---------+----------------------------------------------------------------------
Total | .091001 .0986022 .1286311 .1617972 .156643 .144962 .1289952
------------------------------
How can I swap columns and rows? (transpose the table)

In principle, the answer is c(statistics). For this kind of example, that is legal and it produces a kind of transposition, but the result is not an exact transposition. Here is one way to do something better.
There is no reproducible example in the question, so we need to find one.
Using means is incidental. The same issue would arise with any other statistic.
This is the kind of table we might want to transpose.
. sysuse census, clear
(1980 Census data by state)
. tabstat poplt5-pop65p , s(p50) by(region)
Summary statistics: p50
by categories of: region (Census region)
region | poplt5 pop5_17 pop18p pop65p
--------+----------------------------------------
NE | 185188 637731 2284657 364864
N Cntrl | 327094.5 936449 3126055 521880.5
South | 289571.5 880546 2803536 407053.5
West | 114731 303176 884987 109220
--------+----------------------------------------
Total | 227467.5 629654 2175130 370495
-------------------------------------------------
Trick 1: Simplify the problem by getting a dataset consisting only of what we want to tabulate.
. collapse (p50) poplt5-pop65p, by(region)
. l
+---------------------------------------------------------+
| region poplt5 pop5_17 pop18p pop65p |
|---------------------------------------------------------|
1. | NE 185,188 637,731 2,284,657 364,864 |
2. | N Cntrl 327,094.5 936,449 3,126,054.5 521,880.5 |
3. | South 289,571.5 880,546 2,803,536 407,053.5 |
4. | West 114,731 303,176 884,987 109,220 |
+---------------------------------------------------------+
Trick 2: Use a reshape to map the distinct variables for different categories to a single categorical variable.
. reshape long pop, i(region) j(age) string
(note: j = 18p 5_17 65p lt5)
Data wide -> long
-----------------------------------------------------------------------------
Number of obs. 4 -> 16
Number of variables 5 -> 3
j variable (4 values) -> age
xij variables:
pop18p pop5_17 ... poplt5 -> pop
-----------------------------------------------------------------------------
. l, sepby(region)
+------------------------------+
| region age pop |
|------------------------------|
1. | NE 18p 2,284,657 |
2. | NE 5_17 637,731 |
3. | NE 65p 364,864 |
4. | NE lt5 185,188 |
|------------------------------|
5. | N Cntrl 18p 3,126,054.5 |
6. | N Cntrl 5_17 936,449 |
7. | N Cntrl 65p 521,880.5 |
8. | N Cntrl lt5 327,094.5 |
|------------------------------|
9. | South 18p 2,803,536 |
10. | South 5_17 880,546 |
11. | South 65p 407,053.5 |
12. | South lt5 289,571.5 |
|------------------------------|
13. | West 18p 884,987 |
14. | West 5_17 303,176 |
15. | West 65p 109,220 |
16. | West lt5 114,731 |
+------------------------------+
Trick 3: Use tabdisp directly.
. tabdisp age region, c(pop)
--------------------------------------------------------------
| Census region
age | NE N Cntrl South West
----------+---------------------------------------------------
18p | 2,284,657 3,126,054.5 2,803,536 884,987
5_17 | 637,731 936,449 880,546 303,176
65p | 364,864 521,880.5 407,053.5 109,220
lt5 | 185,188 327,094.5 289,571.5 114,731
--------------------------------------------------------------
Trick 4: Some cleaning up may be needed.
. label def age 1 lt5 2 5_17 3 18p 4 65p
. encode age , gen(ageclass) label(age)
. tab ageclass
ageclass | Freq. Percent Cum.
------------+-----------------------------------
lt5 | 4 25.00 25.00
5_17 | 4 25.00 50.00
18p | 4 25.00 75.00
65p | 4 25.00 100.00
------------+-----------------------------------
Total | 16 100.00
. label def age 1 "<5" 2 "5-17" 3 "18-64" 4 "65+", modify
. tabdisp ageclass region, c(pop)
--------------------------------------------------------------
| Census region
ageclass | NE N Cntrl South West
----------+---------------------------------------------------
<5 | 185,188 327,094.5 289,571.5 114,731
5-17 | 637,731 936,449 880,546 303,176
18-64 | 2,284,657 3,126,054.5 2,803,536 884,987
65+ | 364,864 521,880.5 407,053.5 109,220
--------------------------------------------------------------

I found the answer under following link: https://www.stata.com/statalist/archive/2005-09/msg00561.html
I tried to transpose the table, so I installed the command:
ssc install tabstatmat, replace
tabstat Age25_29 Age30_34 CurntSmokeY religAtndY, by(AlcCons1) stat(mean) col(stat) long format(%9.2f) save
qui tabstatmat B
matrix B = B'
matrix list B, f(%9.2f)
And I have got what I needed:
B[41,5]
1: 2: 3: 4: Total:
mean mean mean mean mean
Age25_29 0.10 0.10 0.09 0.06 0.09
Age30_34 0.09 0.12 0.12 0.07 0.10
Age35_39 0.12 0.15 0.14 0.11 0.13
Age40_44 0.17 0.14 0.16 0.16 0.16
The question now is how to make it look better (delete "mean", Change 1,2,3,4 by words) and then to use putexcel command?

Retrieving Max in range of one column dictated by another column

My set up is fairly simple. I have paired data where one column is time and the next is a value corresponding to that time point. This recurs for many trials with each trial having a different number of time points
Time Freq
0.216 0.000
0.423 4.835
0.620 5.067
0.784 6.108
0.971 5.355
1.156 5.395
1.311 6.470
1.433 8.170
1.575 7.034
1.752 5.673
1.925 5.758
2.077 6.602
2.180 9.675
2.363 5.477
2.487 8.022
2.616 7.795
2.773 6.344
2.915 7.050
3.074 6.283
3.208 7.495
3.395 5.344
3.535 7.111
3.682 6.839
3.830 6.730
4.023 5.185
This is an example from a table. What I want to do is to create a formulate that will pull the Max Frequency when Time is greater that 1 and less than 3. I know this can be done by manually selecting the range, but I have many different ranges that I want to find the max freq for would like to be able to just input the column.

You can reference upper and lower bounds for the time variable like this:
+---+----+----+-------+
| | D | E | F |
+---+----+----+-------+
| 1 | LB | UB |MaxFreq|
| 2 | 1 | 3 | 9.675 |
| 3 | 0 | 1 | 6.108 |
| 4 | 1 | 2 | 8.17 |
| 5 | 2 | 3 | 9.675 |
+---+----+----+-------+
F2: =MAX(IF(($A$1:$A$26>$D2)*($A$1:$A$26<$E2),$B$1:$B$26))
F2 is an array formula--confirm the entry with the combination Ctrl+Shift+Enter (not just Enter). It can be copied down as far as needed.

Excel Solver with Multiple Input Variables

I have an Excel sheet that has a list of sports players, their positions, a salary assigned to those players, and a projected amount of points.
I'm currently using solver to create the best group of sports players possible with certain position restrictions and within a predefined salary cap.
In my specific example, the different player positions are: PG, SG, SF, PF, C, G, and F. Note that a G can be either a PG or a SG, and a F can be either a SF or PF.
This solver is working currently, but in some cases a player might have multiple positions. So, for example, Player A can be used as a SF OR a SG.
Is there any way to account for this within a Solver function so that the player in question can be used in either slot?
To help explain, I'll provide some examples of the data being used. Here is a small sample of the player list:
Position Player Salary Game Points Pos 1 Pos 2
PF/C Kevin Love $10,400 Pho#Min 09:30PM ET 53.17 PF C
PG/SG Stephen Curry $10,000 GS#Bkn 07:30PM ET 47.50 PG SG
SG/SF James Harden $9,500 LAL#Hou 08:00PM ET 41.13 SG SF
PF/C LaMarcus Aldridge $9,500 Orl#Por 10:00PM ET 45.22 PF C
PF/C Anthony Davis $9,200 Was#NO 08:00PM ET 42.97 PF C
PF/C Blake Griffin $9,000 Bos#LAC 10:30PM ET 42.66 PF C
PG John Wall $8,900 Was#NO 08:00PM ET 42.09 PG
PF/C Dwight Howard $8,700 LAL#Hou 08:00PM ET 41.19 PF C
SG/SF Paul George $8,600 Ind#Atl 07:30PM ET 40.06 SG SF
PF Paul Millsap $8,400 Ind#Atl 07:30PM ET 37.96 PF
PF/C Al Horford $8,300 Ind#Atl 07:30PM ET 37.33 PF C
To solve the G and F issue, I have a list that tallies the "selected" players, and that list has a min/max of each. So in this example I need 1 PG, 1 SG, and 1 G. So I have a PG column with a min of 1 and a max of 2, a SG column with a min of 1 and a max of 2, and a G column, which adds PG/SG together that has a min of 3 and a max of 3. Then part of the solver criteria is all the min/max must match.
I have split the first column into POS1 and POS2 to separate the two possible positions, but I can't figure out how to include both in the solver function. For a player like Kevin Love, I want the solver to take into account that he can be placed in either the PF, F, or C spots.
Here is that table:
PG SG SF PF C G F Ttl
Min 1 1 1 1 1 3 3 8
Max 3 3 3 3 2 4 4 8
Cur 1 3 1 2 1 4 3 8
So for example, the Solver criteria is:
Sum of Salary <= Salary Cap (50000)
PG Cur >= PG Min
Repeat for all Min
PG Cur <= PG Max
Repeat for all Max
Maximize Points
And finally, here's an example of a solution:
PG Stephen Curry $10,000 47.5
SG James Harden $9,500 41.13
SF Paul George $8,600 40.06
PF Dwight Howard $8,700 41.19
C Blake Griffin $9,000 42.66
G John Wall $8,900 42.09
F Al Horford $8,300 37.33
Total $63,000 292.5
Each player fits into their slot because one of their two positions (POS1 or POS2) fits the criteria. Obviously the total doesn't fit, so this wouldn't be a viable solution, but it's just an example.
Hopefully I've provided enough details, but if not please let me know and I'll be happy to explain further. Thanks in advance.

Your question is still a bit vague because your Salary Cap is $50,000 but you show $63,000 of players selected.
Assumptions:
1. You want 5 players since that is the number of players floor and the only way to get a team consisting of your player-list below $50,000. This is probably Fantasy hoops and you want 7 or 8, but the solution should still work for 8 players with either a broader list of options and lower salaries.
2. I'll also assume that there is little-no VBA in here since you didn't provide any - or at least none of that needs to be adjusted.
My solution (which is still in testing) is to throw/cast (or VBA equivalent) for Pos1 and Pos2 to a text of position number (i.e., 1 = PG, 2=SG, etc.).
Next, =concatenate(Pos1, Pos2) so that a PG is an 11, a PG/SG is a 12, etc. ("G" is also an 11 in this scenario which might put a glitch in my logic).
Change your Position Table with Min/Max/Cur values looks something like this:
Pos. | PG | G | PG/SG | SG | SG/SF | SF/PF | F | PF | PF/C | C
--------------------------------------------------------------------
Pos1 | 1 | 1 | 1 | 2 | 2 | 3 | 3 | 4 | 4 | 5
Pos2 | 1 | 2 | 2 | 2 | 3 | 4 | 4 | 4 | 5 | 5
PosNbr | 11 | 12 | 12 | 22 | 23 | 34 | 34 | 44 | 45 | 55
--------------------------------------------------------------------
Min | 1 | 3 | 3 | 1 | | 3 | 4 | 1 | | 1
Max | 3 | 4 | 4 | 3 | | 4 | 4 | 3 | | 2
Curr | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 3 | 0
(values are my own).

Resources