I am working with pandas dataframe. One of the columns has list of tuples in each row with some score. I am trying to get scores higher than 0.20. How do I put a threshold instead of max? I tried itemgetter and lambda if else. It didn't worked as I thought. What am I doing wrong?
from operator import itemgetter
import pandas as pd
# sample data
l1 = ['1','2','3']
l2 = ['test1','test2','test3']
l3 = [[(1,0.95),(5,0.05)],[(7,0.10),(1,0.20),(6,0.70)],[(7,0.30),(1,0.70)]]
df = pd.DataFrame({'id':l1,'text':l2,'score':l3})
print(df)
# # Preview from print statement above
id text score
1 test1 [(1, 0.95), (5, 0.05)]
2 test2 [(7, 0.1), (1, 0.2), (6, 0.7)]
3 test3 [(7, 0.3), (1, 0.7)]
# Try #1:
print(df['score'].apply(lambda x: max(x,key=itemgetter(0))))
# Preview from print statement above
(5, 0.05)
(7, 0.1)
(7, 0.3)
# Try #2: Gives `TypeError`
df['score'].apply(lambda x: ((x,itemgetter(0)) if x >= 0.20 else ''))
What I am trying to get for output:
id text probability output needed
1 test1 [(1, 0.95), (5, 0.05)] [(1, 0.95)]
2 test2 [(7, 0.1), (1, 0.2), (6, 0.7)] [(1, 0.2), (6, 0.7)]
3 test3 [(7, 0.3), (1, 0.7)] [(7, 0.3), (1, 0.7)]
You can use a pretty straightforward list comprehension to get the desired output. I'm not sure how you would use itemgetter for this:
df['score'] = df['score'].apply(lambda x: ([y for y in x if min(y) >= .2]))
df
id text score
0 1 test1 [(1, 0.95)]
1 2 test2 [(1, 0.2), (6, 0.7)]
2 3 test3 [(7, 0.3), (1, 0.7)]
If you wanted an alternative result (like an empty tuple, you can use:
df['score'] = df['score'].apply(lambda x: ([y if min(y) >= .2 else () for y in x ]))
Related
This is a common question but I have an extra condition: how do I remove matches based on a unique ID? Or, how to prevent matching against itself?
Given a dataframe:
df = pd.DataFrame({'id':[1, 2, 3],
'name':['pizza','pizza toast', 'ramen']})
I used solutions like this one to create a multi-index dataframe:
Fuzzy match strings in one column and create new dataframe using fuzzywuzzy
df_copy = df.copy()
compare = pd.MultiIndex.from_product([df['name'], df_copy['name']]).to_series()
def metrics(tup):
return pd.Series([fuzz.ratio(*tup),
fuzz.token_sort_ratio(*tup)],
['ratio', 'token'])
compare.apply(metrics)
So that's great but how can I use the unique ID to prevent matching against itself?
If there's a case of ID/name = 1/pizza and 10/pizza, obviously I want to keep those. But I need to remove the same ID in both indexes.
I suggest a slightly different approach for the same result using Python standard library difflib module, which provides helpers for computing deltas.
So, with the following dataframe in which pizza has two different ids (and thus should be checked against one another later on):
import pandas as pd
df = pd.DataFrame(
{"id": [1, 2, 3, 4], "name": ["pizza", "pizza toast", "ramen", "pizza"]}
)
Here is how you can find similarities between different id/name combinations, but avoid checking an id/name combination against itself:
from difflib import SequenceMatcher
# Define a simple helper function
def ratio(a, b):
return SequenceMatcher(None, a, b).ratio()
And then, with the following steps:
# Create a column of unique identifiers: (id, name)
df["id_and_name"] = list(zip(df["id"], df["name"]))
# Calculate ratio only for different id_and_names
df = df.assign(
match=df["id_and_name"].map(
lambda x: {
value: ratio(x[1], value[1])
for value in df["id_and_name"]
if x[0] != value[0] or ratio(x[1], value[1]) != 1
}
)
)
# Format results in a readable fashion
df = (
pd.DataFrame(df["match"].to_list(), index=df["id_and_name"])
.reset_index(drop=False)
.melt("id_and_name", var_name="other_id_and_name", value_name="ratio")
.dropna()
.sort_values(by=["id_and_name", "ratio"], ascending=[True, False])
.reset_index(drop=True)
.pipe(lambda df_: df_.assign(ratio=df_["ratio"] * 100))
.pipe(lambda df_: df_.assign(ratio=df_["ratio"].astype(int)))
)
You get the expected result:
print(df)
# Output
id_and_name other_id_and_name ratio
0 (1, pizza) (4, pizza) 100
1 (1, pizza) (2, pizza toast) 62
2 (1, pizza) (3, ramen) 20
3 (2, pizza toast) (4, pizza) 62
4 (2, pizza toast) (1, pizza) 62
5 (2, pizza toast) (3, ramen) 12
6 (3, ramen) (4, pizza) 20
7 (3, ramen) (1, pizza) 20
8 (3, ramen) (2, pizza toast) 12
9 (4, pizza) (1, pizza) 100
10 (4, pizza) (2, pizza toast) 62
11 (4, pizza) (3, ramen) 20
I have a sorted dataset by timestamps in seconds. However I need to somehow convert it to millisecond accuracy.
Example
dataset = [
# UNIX timestamps with reading data
(0, 0.48499),
(2, 0.48475),
(3, 0.48475),
(3, 0.48473),
(3, 0.48433),
(3, 0.48403),
(3, 0.48403),
(3, 0.48403),
(3, 0.48403),
(3, 0.48403),
(5, 0.48396),
(12, 0.48353),
]
Expected output (roughly)
interpolated = [
# Timestamps with millisecond accuracy
(0.0, 0.48499),
(2.0, 0.48475),
(3.0, 0.48475),
(3.14, 0.48473),
(3.28, 0.48433),
(3.42, 0.48403),
(3.57, 0.48403),
(3.71, 0.48403),
(3.85, 0.48403),
(3.99, 0.48403),
(5.0, 0.48396),
(12.0, 0.48353),
]
I don't have much experience with Pandas and I've gone through interpolate and drop_duplicates but couldn't figure out how to go about this.
I would think this is a common problem so any help appreciated. Ideally I want to spread evenly the numbers.
You can use groupby and apply methods. I didn't come up with a specific method like interpolate in this case, but there might be a more pythonic way.
Code:
import numpy as np
import pandas as pd
# Create a sample dataframe
dataset = [(0, 0.48499), (2, 0.48475), (3, 0.48475), (3, 0.48473), (3, 0.48433), (3, 0.48403), (3, 0.48403), (3, 0.48403), (3, 0.48403), (3, 0.48403), (5, 0.48396), (12, 0.48353)]
df = pd.DataFrame(dataset, columns=['t', 'value'])
# Convert UNIX timestamps into the desired format
df.t = df.groupby('t', group_keys=False).apply(lambda df: df.t + np.linspace(0, 1, len(df)))
Output:
t
value
0
0.48499
2
0.48475
3
0.48475
3.14286
0.48473
3.28571
0.48433
3.42857
0.48403
3.57143
0.48403
3.71429
0.48403
3.85714
0.48403
4
0.48403
5
0.48396
12
0.48353
(Input:)
t
value
0
0.48499
2
0.48475
3
0.48475
3
0.48473
3
0.48433
3
0.48403
3
0.48403
3
0.48403
3
0.48403
3
0.48403
5
0.48396
12
0.48353
Input: 1 2 3 4
Output: (1,2)(3,4)
(1,3)(2,4)
(1,4)(2,3)
I have been able to come up with a solution for the problem but it is efficient. It needs to be optimized.
comb = combinations(Arr,int(n/2))
l = []
for i in comb:
l.append(i)
final_comb = combinations(l,int(n/2))
for i in final_comb:
if is_unique(n,i):
print(i)
def is_unique(n,tup):
k = []
for i in tup:
for j in i:
k.append(j)
if len(set(k)) == n:
return True
return False
The output must be combinations of tuples such that they all have the numbers given as input
Use itertools
from itertools import combinations
list(combinations([1, 2, 3, 4], 2))
>>> [(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
Lets say I have a list like this
my_list = [(1, 'A'), (1, 'A'), (1, 'A'), (1, 'B'), (20, 'BB'),
(20, 'BB'), (100, 'CC'), (100, 'CC'), (100, 'CC')]
now i want to write a code in python 3 so that it returns the frequency of second element of item with accordance to the first item.
for example I expect the output to print something like below :
1: 3 (A), 1 : 1 (B), 20: 2(BB), 100: 3(CC)
Sorry for bad English. Any help would be appreciated.
Use the Counter method from the Collections module.
from collections import Counter
my_list = [(1, 'A'), (1, 'A'), (1, 'A'), (1, 'B'), (20, 'BB'),
(20, 'BB'), (100, 'CC'), (100, 'CC'), (100, 'CC')]
print(Counter(my_list))
Output:
Counter({(1, 'A'): 3, (100, 'CC'): 3, (20, 'BB'): 2, (1, 'B'): 1})
You can also use Dict to achieve this instead of importing collections. Since, the items of your list are tuples, they can be used as keys of a dictionary.
Here's how you can do this:
list = [(1, 'A'), (1, 'A'), (1, 'A'), (1, 'B'), (20, 'BB'), (20, 'BB'), (100, 'CC'), (100, 'CC'), (100, 'CC')]
dict = {}
for item in list:
if item in dict:
dict[item] += 1
else:
dict[item] = 1
print(dict)
Suppose I have a data set like below that shows an undirected graph:
1 2
1 3
1 4
3 5
3 6
7 8
8 9
10 11
I have a python script like it:
for s in ActorGraph.degree():
print(s)
that is a dictionary consist of key and value that keys are node names and values are degree of nodes:
('9', 1)
('5', 1)
('11', 1)
('8', 2)
('6', 1)
('4', 1)
('10', 1)
('7', 1)
('2', 1)
('3', 3)
('1', 3)
In networkx documentation suggest to use values() for having nodes degree.
now I like to have just keys that are degree of nodes and I use this part of script but it does't work and say object has no attribute 'values':
for s in ActorGraph.degree():
print(s.values())
how can I do it?
You are using version 2.0 of networkx. Which changed from using a dict for G.degree() to using a dict-like (but not dict) DegreeView. See this guide.
To have the degrees in a list you can use a list-comprehension:
degrees = [val for (node, val) in G.degree()]
I'd like to add the following: if you're initializing the undirected graph with nx.Graph() and adding the edges afterwards, just beware that networkx doesn't guarrantee the order of nodes will be preserved -- this also applies to degree(). This means that if you use the list comprehension approach then try to access the degree by list index the indexes may not correspond to the right nodes. If you'd like them to correspond, you can instead do:
degrees = [val for (node, val) in sorted(G.degree(), key=lambda pair: pair[0])]
Here's a simple example to illustrate this:
>>> edges = [(0, 1), (0, 3), (0, 5), (1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (2, 5)]
>>> g = nx.Graph()
>>> g.add_edges_from(edges)
>>> print(g.degree())
[(0, 3), (1, 4), (3, 3), (5, 2), (2, 4), (4, 2)]
>>> print([val for (node, val) in g.degree()])
[3, 4, 3, 2, 4, 2]
>>> print([val for (node, val) in sorted(g.degree(), key=lambda pair: pair[0])])
[3, 4, 4, 3, 2, 2]
You can also use a dict comprehension to get an actual dictionary:
degrees = {node:val for (node, val) in G.degree()}