In this dataframe, how to go about appending a column named "class_name", with a text string, that is based on another column.
x
y
z
not used
Label
-3.8481877
-0.47685334
0.63422906
1.0396314
1
-2.320888
0.65347993
1.1519914
0.12997247
1
1.5827686
1.4119303
-1.7410104
-4.6962333
2
-0.1337152
0.13315737
-1.6648949
-1.4205348
2
-0.4028037
1.332986
1.3618442
0.3292255
1
-0.015517877
1.346349
1.4083523
0.87017965
0
-0.2669228
0.5478992
-0.06730786
-1.5959451
0
-0.03318152
0.3263167
-2.116833
-5.4616213
1
There are the values the new column will take based on the values in the 'Label' column:
0 == 'avocados'
1 == 'apples'
2 == ' grapes
This is my code so far:
import numpy as np
import matplotlib.pyplot as plt
import plotly.express as px
import seaborn as sns
import pandas as pd
df = pd.read_csv('embed1_2.csv')
df.loc[df.y_train == 103, 'class_name'] = 'avocados'
df.loc[df.y_train == 103, 'class_name'] = 'apples'
df.loc[df.y_train == 103, 'class_name'] = 'grapes'
How to get the appended column to show up with the converted text?
Thanks for your help!
create a dictionary and then use map in creating a new columns
dict = {
0 : 'avocados',
1 : 'apples',
2 : 'grapes'
}
df['val']=df['Label'].map(dict)
df
x y z not used Label val
0 -3.848188 -0.476853 0.634229 1.039631 1 apples
1 -2.320888 0.653480 1.151991 0.129972 1 apples
2 1.582769 1.411930 -1.741010 -4.696233 2 grapes
3 -0.133715 0.133157 -1.664895 -1.420535 2 grapes
4 -0.402804 1.332986 1.361844 0.329226 1 apples
5 -0.015518 1.346349 1.408352 0.870180 0 avocados
6 -0.266923 0.547899 -0.067308 -1.595945 0 avocados
7 -0.033182 0.326317 -2.116833 -5.461621 1 apples
Related
In my python code, using pandas i have to resample a datetimedata series and calculate diffs between a column values (the sum of diffs between values), i write this piece of code:
import pandas as pd
import datetime
from .models import Results, VarsResults
start_date = datetime.date(2021, 6, 21)
end_date = datetime.date(2021, 6, 24)
def calc_q(start_d, end_d):
start_d = start_date
end_d = end_date
var_results = VarsResults.objects.filter(
id_res__read_date__range=(start_d, end_d)
).select_related(
"id_res"
).values(
"id_res__read_date",
"id_res__unit_id",
"id_res__device_id",
"id_res__proj_code",
"var_val",
)
df = pd.DataFrame(list(var_results))
df['id_res__read_date'] = pd.to_datetime(df['id_res__read_date'])
df = df.set_index('id_res__read_date')
df_15 = df.resample('15min').sum()
return df_15
but i get the sum of the values itself.
example
... | 5
... | 3
... | 1
i get 9
i would the sum of the difference between values not the sum of the values:
in this case 4 (5-3 = 2 + 3-1 = 2, 2+2)
Is there a method in pandas using resample for manage this kind of clcultion?
So many thanks in advance
Manuel
The sum of all the differences is equal to the difference between the first element and the last one: if you work it out, all the other elements cancel out. In your data for example the 3 cancels out:
(5-3) + (3-1)
= 5 - 3 + 3 - 1 # - 3 and + 3 cancel out
= 5 - 1
I don't know how Pandas works, but you can simply do the equivalent of first_value - last_value.
I am unable to understand the mask variable in the code below. The code is basically to filter out the words in the given series in which there are more than 2 variables.
# Input
ser = pd.Series(['Apple', 'Orange', 'Plan', 'Python', 'Money'])
# Solution
from collections import Counter
mask = ser.map(lambda x: sum([Counter(x.lower()).get(i, 0) for i in list('aeiou')]) >= 2)
ser[mask]
Can someone please help me in understanding this in a better way?
Use Series.str.count with regex for test all values in [] with ignore lowercase/uppercase:
print (ser[ser.str.count('(?i)[aeiou]') >=2])
0 Apple
1 Orange
4 Money
dtype: object
Another solution:
import re
print (ser[ser.str.count('[aeiou]', re.I) >=2])
0 Apple
1 Orange
4 Money
dtype: object
Try this:
import re
mask = ser.str.count('a|e|i|o|u', re.IGNORECASE) >= 2
ser[mask]
Output:
0 Apple
1 Orange
4 Money
dtype: object
import pandas as pd
series=pd.Series(['red','Green','orange','pink','yellow','white'])
for i in series:
a=0
for j in i:
if j in ['a','e','i','o','u','A','E','I','O','U']:
a=a+1
if a>=2:
print(i)
break
I'm trying to apply a function to different readings of each measure. Is it possible to do it without transforming the dataframe?
import random
import pandas as pd
df = pd.DataFrame({
'index': sorted(['A', 'B']*3),
'measure': [i for i in range(0,3)]*2,
'reading': [random.random() for i in range(0,6)]
})
index measure reading
0 A 0 0.260492
1 A 1 0.805028
2 A 2 0.548699
3 B 0 0.014042
4 B 1 0.719705
5 B 2 0.398824
How can I apply a function like basic difference to different readings for each index?
Here I assumed function applied to reading 0 and 1. It should be part of the call as I need to calculate it for different values of measure.
Desired output looks like this:
index applied
0 A 0.5445359999999999
1 B 0.705663
Try this
import random
import pandas as pd
import numpy as np
df = pd.DataFrame({
'index': sorted(['A', 'B']*3),
'measure': [i for i in range(0,3)]*2,
'reading': [random.random() for i in range(0,6)]
})
print(df)
# index measure reading
# 0 A 0 0.869707
# 1 A 1 0.120680
# 2 A 2 0.772035
# 3 B 0 0.565548
# 4 B 1 0.577074
# 5 B 2 0.290668
start = 0
stop = 1
# I decided to specify start and stop value separately, the absolute difference is
# calculated via np.sum(). If the difference between start and stop is always 1, you
# can omit the np.sum() call.
df = df.groupby('index').agg(applied=('reading', lambda x: np.sum(np.diff(x)
[start:stop])))
print(df)
# applied
# index
# A -0.749027
# B 0.011526
After forming the below python pandas dataframe (for example)
import pandas
data = [['Alex',10],['Bob',12],['Clarke',13]]
df = pandas.DataFrame(data,columns=['Name','Age'])
If I iterate through it, I get
In [62]: for i in df.itertuples():
...: print( i.Index, i.Name, i.Age )
...:
0 Alex 10
1 Bob 12
2 Clarke 13
What I would like to achieve is to replace the value of a particular cell
In [67]: for i in df.itertuples():
...: if i.Name == "Alex":
...: df.at[i.Index, 'Age'] = 100
...:
Which seems to work
In [64]: df
Out[64]:
Name Age
0 Alex 100
1 Bob 12
2 Clarke 13
The problem is that when using a larger different dataset, and do:
First, I create a new column named like NETELEMENT with a default value of ""
I would like to replace the default value "" with the string that the function lookup_netelement returns
df['NETELEMENT'] = ""
for i in df.itertuples():
df.at[i.Index, 'NETELEMENT'] = lookup_netelement(i.PEER_SRC_IP)
print( i, lookup_netelement(i.PEER_SRC_IP) )
But what I get as a result is:
Pandas(Index=769, SRC_AS='', DST_AS='', COMMS='', SRC_COMMS=nan, AS_PATH='', SRC_AS_PATH=nan, PREF='', SRC_PREF='0', MED='0', SRC_MED='0', PEER_SRC_AS='0', PEER_DST_AS='', PEER_SRC_IP='x.x.x.x', PEER_DST_IP='', IN_IFACE='', OUT_IFACE='', PROTOCOL='udp', TOS='0', BPS=35200.0, SRC_PREFIX='', DST_PREFIX='', NETELEMENT='', IN_IFNAME='', OUT_IFNAME='') routerX
meaning that it should be:
NETELEMENT='routerX' instead of NETELEMENT=''
Could you please advise what I am doing wrong ?
EDIT: for reasons of completeness the lookup_netelement is defined as
def lookup_netelement(ipaddr):
try:
x = LOOKUP['conn'].hget('ipaddr;{}'.format(ipaddr), 'dev') or b""
except:
logger.error('looking up `ipaddr` for netelement caused `{}`'.format(repr(e)), exc_info=True)
x = b""
x = x.decode("utf-8")
return x
Hope you are looking for where for conditional replacement i.e
def wow(x):
return x ** 10
df['new'] = df['Age'].where(~(df['Name'] == 'Alex'),wow(df['Age']))
Output :
Name Age new
0 Alex 10 10000000000
1 Bob 12 12
2 Clarke 13 13
3 Alex 15 576650390625
Based on your edit your trying to apply the function i.e
df['new'] = df['PEER_SRC_IP'].apply(lookup_netelement)
Edit : For your comment on sending two columns, use lambda with axis 1 i.e
def wow(x,y):
return '{} {}'.format(x,y)
df.apply(lambda x : wow(x['Name'],x['Age']),1)
What I am trying to do is to get bootstrap confidence limits by row regardless of the number of rows and make a new dataframe from the output.I currently can do this for the entire dataframe, but not by row. The data I have in my actual program looks similar to what I have below:
0 1 2
0 1 2 3
1 4 1 4
2 1 2 3
3 4 1 4
I want the new dataframe to look something like this with the lower and upper confidence limits:
0 1
0 1 2
1 1 5.5
2 1 4.5
3 1 4.2
The current generated output looks like this:
0 1
0 2.0 2.75
The python 3 code below generates a mock dataframe and generates the bootstrap confidence limits for the entire dataframe. The result is a new dataframe with just 2 values, a upper and a lower confidence limit rather than 4 sets of 2(one for each row).
import pandas as pd
import numpy as np
import scikits.bootstrap as sci
zz = pd.DataFrame([[[1,2],[2,3],[3,6]],[[4,2],[1,4],[4,6]],
[[1,2],[2,3],[3,6]],[[4,2],[1,4],[4,6]]])
print(zz)
x= zz.dtypes
print(x)
a = pd.DataFrame(np.array(zz.values.tolist())[:, :, 0],zz.index, zz.columns)
print(a)
b = sci.ci(a)
b = pd.DataFrame(b)
b = b.T
print(b)
Thank you for any help.
scikits.bootstrap operates by assuming that data samples are arranged by row, not by column. If you want the opposite behavior, just use the transpose, and a statfunction that doesn't combine columns.
import pandas as pd
import numpy as np
import scikits.bootstrap as sci
zz = pd.DataFrame([[[1,2],[2,3],[3,6]],[[4,2],[1,4],[4,6]],
[[1,2],[2,3],[3,6]],[[4,2],[1,4],[4,6]]])
print(zz)
x= zz.dtypes
print(x)
a = pd.DataFrame(np.array(zz.values.tolist())[:, :, 0],zz.index, zz.columns)
print(a)
b = sci.ci(a.T, statfunction=lambda x: np.average(x, axis=0))
print(b.T)
Below is the answer I ended up figuring out to create bootstrap ci by row.
import pandas as pd
import numpy as np
import numpy.random as npr
zz = pd.DataFrame([[[1,2],[2,3],[3,6]],[[4,2],[1,4],[4,6]],
[[1,2],[2,3],[3,6]],[[4,2],[1,4],[4,6]]])
x= zz.dtypes
a = pd.DataFrame(np.array(zz.values.tolist())[:, :, 0],zz.index, zz.columns)
print(a)
def bootstrap(data, num_samples, statistic, alpha):
n = len(data)
idx = npr.randint(0, n, (num_samples, n))
samples = data[idx]
stat = np.sort(statistic(samples, 1))
return (stat[int((alpha/2.0)*num_samples)],
stat[int((1-alpha/2.0)*num_samples)])
cc = list(a.index.values) # informs generator of the number of rows
def bootbyrow(cc):
for xx in range(1):
xx = list(a.index.values)
for xx in range(len(cc)):
k = a.apply(lambda y: y[xx])
k = k.values
for xx in range(1):
kk = list(bootstrap(k,10000,np.mean,0.05))
yield list(kk)
abc = pd.DataFrame(list(bootbyrow(cc))) #bootstrap ci by row
# the next 4 just show that its working correctly
a0 = bootstrap((a.loc[0,].values),10000,np.mean,0.05)
a1 = bootstrap((a.loc[1,].values),10000,np.mean,0.05)
a2 = bootstrap((a.loc[2,].values),10000,np.mean,0.05)
a3 = bootstrap((a.loc[3,].values),10000,np.mean,0.05)
print(abc)
print(a0)
print(a1)
print(a2)
print(a3)