Related
I have a data like the following:
NAME ETHNICITY_RECAT TOTAL_LENGTH 3LETTER_SUBSTRINGS
joseph fr 14 jos, ose, sep, eph
ann en 16 ann
anne ir 14 ann, nne
tom en 18 tom
tommy fr 16 tom, omm, mmy
ann ir 19 ann
... more rows
The 3LETTER_SUBSTRINGS values are string which captures all the 3-letter substrings of the NAME variable. I would like to aggregate it into a single list, with each comma-separated item appended to the list by each row, and to be considered as a single list item. As follows:
ETHNICITY_RECAT TOTAL_LENGTH 3LETTER_SUBSTRINGS
min max mean <lambda>
fr 2 26 13.22 [jos, ose, sep, eph, tom, oom, mmy, ...]
en 3 24 11.92 [ann, tom, ...]
ir 4 23 12.03 [ann, nne, ann, ...]
I kind of "did" it using the following code:
aggregations = {
'TOTAL_LENGTH': [min, max, 'mean'],
'3LETTER_SUBSTRINGS': lambda x: list(x),
}
self.df_agg = self.df.groupby('ETHNICITY_RECAT', as_index=False).agg(aggregations)
The problem is the whole string "ann, anne" is considered one single list item in the final list, instead of considering each as single list item, such as "ann", "anne".
I would like to see the highest frequency of the substrings, but now I am getting the frequency of the whole string (instead of the individual 3-letter substring), when I run the following code:
from collections import Counter
x = self.df_agg_eth[self.df_agg_eth['ETHNICITY_RECAT']=='en']['3LETTER_SUBSTRINGS']['<lambda>']
x_list = x[0]
c = Counter(x_list)
I get this:
[('jos, ose, sep, eph', 19), ('ann, nee', 5), ...]
Instead of what I want:
[('jos', 19), ('ose', 19), ('sep', 23), ('eph', 19), ('ann', 15), ('nee', 5), ...]
I tried:
'3LETTER_SUBSTRINGS': lambda x: list(i) for i in x.split(', '),
But it says invalid syntax.
First thing you want to do is to convert the string into list, then it's just a groupby with agg:
df['3LETTER_SUBSTRINGS'] = df['3LETTER_SUBSTRINGS'].str.split(', ')
df.groupby('ETHNICITY_RECAT').agg({'TOTAL_LENGTH':['min','max','mean'],
'3LETTER_SUBSTRINGS':'sum'})
Output:
TOTAL_LENGTH 3LETTER_SUBSTRINGS
min max mean sum
ETHNICITY_RECAT
en 16 18 17.0 [ann, tom]
fr 14 16 15.0 [jos, ose, sep, eph, tom, omm, mmy]
ir 14 19 16.5 [ann, nne, ann]
I think most of your code is alright, you just misinterpreted the error: it has nothing to do with string conversion. You have lists/tuples in each cell of the 3LETTER_SUBSTRING column. When you use the lambda x:list(x) function, you create a list of tuples. Hence there is nothing like split(",") to do and going to cast to string and back to table ...
Instead, you just need to unnest your table when you create your new list. So here's a small reproducible code: (note that I focused on your tuple/aggregation issue as I'm sure you will quickly find the rest of the code)
import pandas as pd
# Create some data
names = [("joseph","fr"),("ann","en"),("anne","ir"),("tom","en"),("tommy","fr"),("ann","fr")]
df = pd.DataFrame(names, columns=["NAMES","ethnicity"])
df["3LETTER_SUBSTRING"] = df["NAMES"].apply(lambda name: [name[i:i+3] for i in range(len(name) - 2)])
print(df)
# Aggregate the 3LETTER per ethnicity, and unnest the result in a new table for each ethnicity:
df.groupby('ethnicity').agg({
"3LETTER_SUBSTRING": lambda x:[z for y in x for z in y]
})
Using the counter you specify, I got
dfg = df.groupby('ethnicity', as_index=False).agg({
"3LETTER_SUBSTRING": lambda x:[z for y in x for z in y]
})
from collections import Counter
print(Counter(dfg[dfg["ethnicity"] == "en"]["3LETTER_SUBSTRING"][0]))
# Counter({'ann': 1, 'tom': 1})
To get it as a list of tuples, just use a dictionary built-in function such as dict.items().
UPDATE : using preformated string list as in the question:
import pandas as pd
# Create some data
names = [("joseph","fr","jos, ose, sep, eph"),("ann","en","ann"),("anne","ir","ann, nne"),("tom","en","tom"),("tommy","fr","tom, omm, mmy"),("ann","fr","ann")]
df = pd.DataFrame(names, columns=["NAMES","ethnicity","3LETTER_SUBSTRING"])
def transform_3_letter_to_table(x):
"""
Update this function with regard to your data format
"""
return x.split(", ")
df["3LETTER_SUBSTRING"] = df["3LETTER_SUBSTRING"].apply(transform_3_letter_to_table)
print(df)
# Applying aggregation
dfg = df.groupby('ethnicity', as_index=False).agg({
"3LETTER_SUBSTRING": lambda x:[z for y in x for z in y]
})
print(dfg)
# test on some data
from collections import Counter
c = Counter(dfg[dfg["ethnicity"] == "en"]["3LETTER_SUBSTRING"][0])
print(c)
print(list(c.items()))
I have been trying to take integer inputs seperated by any number of white spaces or line breaks. I know how to take space seperated outputs and outputs having line breaks. In C based languages we don't have to care about where the input is, it automatically takes the input when found, but I don't think this is the case with Python(correct me if I am wrong). Can anybody help?
I tried using a While statement till True and using a try statement in it. But it doesn't work.
a = []
try:
while(True):
a.append(int(input()))
except:
pass
print(a)
when i input
12 12
12
it returns an empty list. If i remove the int in the input it returns a list [12 12, 12].
Try this: The Shortest way possible
a = []
s=input()
while s != '':
i = s.split()
[a.append(int(j)) for j in i]
s=input()
print(a)
Input:
1 2 3
4 5
6
Output:
[1, 2, 3, 4, 5, 6]
You can also try:
a = []
s=input()
while s != '':
i = s.split()
a.extend(map(lambda s: int(s),i))
s=input()
print(a)
Wait, so I think I understand it now. You want to accept any amount of input, but save each input separated by whitespace as its own entry? There is actually a string method for that. Here's an example script for it. It's not the best, but it demonstrates the method pretty well.
list = []
string = "user input goes here"
splitString = string.split()
for word in splitString:
list.append(word)
print(list)
Output:
["user", "input", "goes", "here"]
The string.split() method uses space by default, but you can specify another delimiter like the # sign.
List = []
String = "Hi#my#name#is#bob"
newString = String.split("#")
for word in newString:
list.append(word)
EDIT: Here is a full working implementation that will work whether the thing separating two inputs is whitespace, newlines, or anything else you'd like.
import re
list = []
while True:
string = input()
if string == "break":
break
splitString = re.split("[\s | \r\n]", string)
for word in splitString:
list.append(word)
cleanList = []
for word in list:
if word != '':
cleanList.append(word)
print(cleanList)
Input:
12 94 17
56
3
Output:
[12, 94, 17, 56, 3]
Functional proof: Click here
Hope you will some insight in this example & have added my personal view of how to code.
Firstly, giving input with multi-spaces is understandable but not multi-lines. Prefer taking input one by one.
For testing & debugging purposes, prefer separate collecting user and processing input data.
Now, say you have collected your user input and stored as data using raw_input, which is handy when you need to collect multiline inputs. Please explore raw_input, it is supported in Python3 and Python2.
>>>
>>> data = '''12
...
... 12 12
...
...
... 12'''
>>> data
'12 \n\n12 12\n\n\n12'
>>> print(data)
12
12 12
12
step1: clear all line separations
>>> double_spaces = ' '
>>> single_space = ' '
>>> data = data.strip().replace('\n', single_space)
>>> data
'12 12 12 12'
step2: Fix multiple spaces
>>> while double_spaces in data:
... data = data.replace(double_spaces, single_space)
...
>>> data
'12 12 12 12'
>>> print(list(map(int, data.split()))
...
... )
[12, 12, 12, 12]
>>>
Problems with you code
>>> a = []
>>> try:
... while(True):
... a.append(int(input()))
... except:
... pass
...
12
1
12 12
>>> a
[12, 1]
>>>
When you enter 12 12, below is supposed to happen.
>>> int('12 12')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '12 12'
Since this code had bad exception handling except:, your use case is returning an empty list as expected.
Since I have changed the input, you see this difference.
In Python 3, I want to return the units place of an integer value, then tens, then hundreds and so on. Suppose I have an integer 456, first I want to return 6, then 5 then 4. Is there any way? I tried floor division and for loop but didn't work.
If you look at the list of basic operators from the documentation, for example here,
Operator Description Example
% Modulus Divides left hand operand by right hand operand and returns remainder b % a = 1
// Floor Division - The division of operands where the result is the quotient in which the digits after the decimal point are removed. But if one of the operands is negative, the result is floored, i.e., rounded away from zero (towards negative infinity): 9//2 = 4 and 9.0//2.0 = 4.0, -11//3 = -4, -11.0//3 = -4.0
With that knowledge, you can get what you want as follows:
In [1]: a = 456
In [2]: a % 10
Out[2]: 6
In [3]: (a % 100) // 10
Out[3]: 5
In [4]: a // 100
Out[4]: 4
Write a generator if you want to retrieve digits in different places of your code based on requirement as follows.
If you are not much familiar with Python's generator, have a quick look at https://www.programiz.com/python-programming/generator.
» Here get_digits() is a generator.
def get_digits(n):
while str(n):
yield n % 10
n = n // 10
if not n:
break
digit = get_digits(1729)
print(next(digit)) # 9
print(next(digit)) # 2
print(next(digit)) # 7
print(next(digit)) # 1
» If you wish to iterate over digits, you can also do so as follows.
for digit in get_digits(74831965):
print(digit)
# 5
# 6
# 9
# 1
# 3
# 8
# 4
# 7
» Quick overview about its usage (On Python3's Interactive terminal).
>>> def letter(name):
... for ch in name:
... yield ch
...
>>>
>>> char = letter("RISHIKESH")
>>>
>>> next(char)
'R'
>>>
>>> "Second letter is my name is: " + next(char)
'Second letter is my name is: I'
>>>
>>> "3rd one: " + next(char)
'3rd one: S'
>>>
>>> next(char)
'H'
>>>
After forming the below python pandas dataframe (for example)
import pandas
data = [['Alex',10],['Bob',12],['Clarke',13]]
df = pandas.DataFrame(data,columns=['Name','Age'])
If I iterate through it, I get
In [62]: for i in df.itertuples():
...: print( i.Index, i.Name, i.Age )
...:
0 Alex 10
1 Bob 12
2 Clarke 13
What I would like to achieve is to replace the value of a particular cell
In [67]: for i in df.itertuples():
...: if i.Name == "Alex":
...: df.at[i.Index, 'Age'] = 100
...:
Which seems to work
In [64]: df
Out[64]:
Name Age
0 Alex 100
1 Bob 12
2 Clarke 13
The problem is that when using a larger different dataset, and do:
First, I create a new column named like NETELEMENT with a default value of ""
I would like to replace the default value "" with the string that the function lookup_netelement returns
df['NETELEMENT'] = ""
for i in df.itertuples():
df.at[i.Index, 'NETELEMENT'] = lookup_netelement(i.PEER_SRC_IP)
print( i, lookup_netelement(i.PEER_SRC_IP) )
But what I get as a result is:
Pandas(Index=769, SRC_AS='', DST_AS='', COMMS='', SRC_COMMS=nan, AS_PATH='', SRC_AS_PATH=nan, PREF='', SRC_PREF='0', MED='0', SRC_MED='0', PEER_SRC_AS='0', PEER_DST_AS='', PEER_SRC_IP='x.x.x.x', PEER_DST_IP='', IN_IFACE='', OUT_IFACE='', PROTOCOL='udp', TOS='0', BPS=35200.0, SRC_PREFIX='', DST_PREFIX='', NETELEMENT='', IN_IFNAME='', OUT_IFNAME='') routerX
meaning that it should be:
NETELEMENT='routerX' instead of NETELEMENT=''
Could you please advise what I am doing wrong ?
EDIT: for reasons of completeness the lookup_netelement is defined as
def lookup_netelement(ipaddr):
try:
x = LOOKUP['conn'].hget('ipaddr;{}'.format(ipaddr), 'dev') or b""
except:
logger.error('looking up `ipaddr` for netelement caused `{}`'.format(repr(e)), exc_info=True)
x = b""
x = x.decode("utf-8")
return x
Hope you are looking for where for conditional replacement i.e
def wow(x):
return x ** 10
df['new'] = df['Age'].where(~(df['Name'] == 'Alex'),wow(df['Age']))
Output :
Name Age new
0 Alex 10 10000000000
1 Bob 12 12
2 Clarke 13 13
3 Alex 15 576650390625
Based on your edit your trying to apply the function i.e
df['new'] = df['PEER_SRC_IP'].apply(lookup_netelement)
Edit : For your comment on sending two columns, use lambda with axis 1 i.e
def wow(x,y):
return '{} {}'.format(x,y)
df.apply(lambda x : wow(x['Name'],x['Age']),1)
Is there an easy way to modify this code which converts from base 2 into base 10 to work for converting base 16 into base 10? My objective is to build a dedicated function for conversion and not use any built-in Python features for the calculation. Thanks
BinaryVal = int(input('Enter:')
DecVal = 0
for n in range(len(str(BinaryVal))):
Power = len(str(BinX))-(n+1)
DecVal += int(str(BinaryVal)[n])*(2**Power)
print(DecVal)
Yikes.
int already can convert from any base to base 10 - just supply it as the second argument.
int('101010',2)
Out[64]: 42
int('2A',16)
Out[66]: 42
To convert hexadecimal string to int:
>>> hexstr = '101010'
>>> int(hexstr, 16)
1052688
The same -- without int constructor:
>>> import binascii
>>> int.from_bytes(binascii.unhexlify(hexstr), 'big')
1052688
The same -- similar to #SzieberthAdam's answer:
>>> hex2dec = {d: i for i, d in enumerate('0123456789abcdef')}
>>> sum(hex2dec[h] * 16**pos for pos, h in enumerate(reversed(hexstr.lower())))
1052688
or:
>>> from functools import reduce
>>> reduce(lambda n, h: n*16 + hex2dec[h], hexstr.lower(), 0)
1052688
that is equivalent to:
def hex2int(hexstr):
n = 0
for h in hexstr.lower():
n = n*16 + hex2dec[h]
return n
Example:
>>> hex2int('101010')
1052688
As an alternative, one could convert all digits to int first:
>>> reduce(lambda n, d: n*16 + d, map(hex2dec.get, hexstr.lower()))
1052688
It raises TypeError for empty strings.
Well, here you go then:
>>> binary_num = '101010'
>>> sum(int(b)*2**i for i, b in enumerate(reversed(binary_num)))
42