I have two lists:
a = [1, 2, 3, 4]
b = [9, 8, 7, 6]
I'd like to have every combination of these two lists passed as argument to a function I'm multithreading:
def test(hello, world):
return hello + world
with ThreadPoolExecutor(max_workers=10) as executor:
future_to_stuff = { executor.submit(self._http_check_port, hello, world): ### }
for future in as_completed(future_to_port):
...
I'm trying to figure out how to "unpack" both my lists so that every combination of values in a and b are sent as params to the function.
I usually use following list comprehension.
future_to_stuff = [executor.submit(test, hello, world)
for hello, world in zip(a, b)]
Here is a modified code.
from concurrent.futures import ThreadPoolExecutor, as_completed
def test(hello, world):
return hello + world
def main(a, b):
with ThreadPoolExecutor(max_workers=10) as executor:
future_to_stuff = [executor.submit(test, hello, world)
for hello, world in zip(a, b)]
for future in as_completed(future_to_stuff):
print(future.result())
if __name__ == '__main__':
a = [1, 2, 3, 4]
b = [9, 8, 7, 6]
main(a, b)
Another way is to use .map method instead of .submit.
from concurrent.futures import ThreadPoolExecutor, as_completed
def test(hello, world):
return hello + world
def main(a, b):
with ThreadPoolExecutor(max_workers=10) as executor:
results = executor.map(test, a, b)
for result in results:
print(result)
if __name__ == '__main__':
a = [1, 2, 3, 4]
b = [9, 8, 7, 6]
main(a, b)
Related
I'm trying to use list comprehension to replace a for loop for the purpose of improving speed. Being new to list comprehensions, it appears that I do not have a complete grasp of how they work. My intent is to pass a list to a function that is held in memory to complete the processing that needs to be done, then return the list. In this simplified example, it looks like instead of returning the list I want it is returning a list of identical lists. So instead of [1, 2, 3, 4, 5] I get [[1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5]]. I looked at the result here but it looks like they had a different problem.
lst1 = [1,2,3,4,5]
k = []
def main():
lst2 = [processfile(x) for x in lst1]
print(lst2)
def processfile(i):
k.append(i)
return(k)
if __name__ == '__main__':
main()
Any help would be greatly appreciated.
If you want to return a list then , you don't need to append the items in the lst1 to k just return i
lst1 = [1,2,3,4,5]
def main():
lst2 = [processfile(x) for x in lst1]
print(lst2)
def processfile(i):
return(i)
if __name__ == '__main__':
main()
Since k is only being used in the process file() function, avoid declaring it in the global scope. I didn't quite understand what you wanted/tried to do, but I hope this example will help you with list comprehension, also, I'd recommend you add 'Python' to the title of your post.
Here I will use list comprehension to add 1 to all items in list1.
lst1 = [1,2,3,4,5]
def main():
lst2 = [add1(item) for item in lst1]
print(lst2)
def add1(num):
return num + 1
if __name__ == '__main__':
main()
Expected output is lst2 = [2,3,4,5,6]
Just
lst2 = [processfile(x) for x in lst1]
will be fine.
In that case you should do like below:
def processfile(i):
return(i)
This question already has answers here:
How can I iterate over overlapping (current, next) pairs of values from a list?
(12 answers)
Closed 10 months ago.
Is it possible to iterate a list in the following way in Python (treat this code as pseudocode)?
a = [5, 7, 11, 4, 5]
for v, w in a:
print [v, w]
And it should produce
[5, 7]
[7, 11]
[11, 4]
[4, 5]
You can zip the list with itself sans the first element:
a = [5, 7, 11, 4, 5]
for previous, current in zip(a, a[1:]):
print(previous, current)
This works even if your list has no elements or only 1 element (in which case zip returns an empty iterable and the code in the for loop never executes). It doesn't work on generators, only sequences (tuple, list, str, etc).
From the itertools recipes:
from itertools import tee
def pairwise(iterable):
"s -> (s0,s1), (s1,s2), (s2, s3), ..."
a, b = tee(iterable)
next(b, None)
return zip(a, b)
for v, w in pairwise(a):
...
To do that you should do:
a = [5, 7, 11, 4, 5]
for i in range(len(a)-1):
print [a[i], a[i+1]]
Nearly verbatim from Iterate over pairs in a list (circular fashion) in Python:
def pairs(seq):
i = iter(seq)
prev = next(i)
for item in i:
yield prev, item
prev = item
>>> a = [5, 7, 11, 4, 5]
>>> for n,k in enumerate(a[:-1]):
... print a[n],a[n+1]
...
5 7
7 11
11 4
4 5
I got an error when executing code below. The problem seems to be map doesn't support functions taking multiple inputs, just as in the python built-in multiprocessing package. But in the built-in package, there is a starmap that solves this issue. Does pathos.multiprocessing have the same?
import pathos.multiprocessing as mp
class Bar:
def foo(self, name):
return len(str(name))
def boo(self, x, y, z):
sum = self.foo(x)
sum += self.foo(y)
sum += self.foo(z)
return sum
if __name__ == '__main__':
b = Bar()
pool = mp.ProcessingPool()
results = pool.map(b.boo, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
print(results)
TypeError: boo() missing 2 required positional arguments: 'y' and 'z'
Update for lambda expression as suggested (didn't work):
if __name__ == '__main__':
b = Bar()
pool = mp.ProcessingPool()
results = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
print(results)
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File
"C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py",
line 121, in worker
result = (True, func(*args, **kwds))
File
"C:\Users\yg451\Anaconda3\lib\site-packages\multiprocess\pool.py",
line 44, in mapstar
return list(map(*args))
File
"C:\Users\yg451\Anaconda3\lib\site-packages\pathos\helpers\mp_helper.py",
line 15, in
func = lambda args: f(*args)
File "C:/Users/yg451/Code/foo/Machine
Learning/xPype/test/scratch.py", line 18, in
results = pool.map(lambda x: b.boo(*x), [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
NameError: name 'b' is not defined
"""
I'm the pathos author. pathos is older than starmap, and doesn't really need it. It solved multiple arguments in a pool exactly the same way that the built-in map does.
>>> import pathos.multiprocessing as mp
>>> class Bar:
... def foo(self, name):
... return len(str(name))
... def boo(self, x, y, z):
... sum = self.foo(x)
... sum += self.foo(y)
... sum += self.foo(z)
... return sum
...
>>> b = Bar()
>>> pool = mp.ProcessingPool()
>>> f = lambda x: b.boo(*x)
>>> results = pool.map(f, [(12, 3, 456), (8, 9, 10), ('a', 'b', 'cde')])
>>> results
[6, 4, 5]
>>> results = pool.map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])
>>> results
[6, 4, 5]
>>> results = map(b.boo, [12, 9, 'a'], [3, 9, 'b'], [456, 10, 'cde'])
>>> list(results)
[6, 4, 5]
>>>
So, essentially, starmap is unnecessary. However, as it's been recently added to the standard Pool interface in multiprocessing in certain versions of python, it probably should be more prominent in pathos. Note that it already is possible to get to an "augmented" version of starmap from pathos if you like.
>>> import pathos
>>> mp = pathos.helpers.mp
>>> p = mp.Pool()
>>> p.starmap
<bound method Pool.starmap of <multiprocess.pool.Pool object at 0x1038684e0>>
>>>
I was trying to write a program that would reverse a list in python3. I first tried:
def reverse(lst):
""" Reverses lst in place.
>>> x = [3, 2, 4, 5, 1]
>>> reverse(x)
>>> x
[1, 5, 4, 2, 3]
"""
n = len(lst)
for i in range(n//2):
lst[i], lst[n-i-1] = lst[n-i-1], lst[i]
It failed and the x I got was the original value. However, when I changed my code to this, it worked:
def reverse(lst):
""" Reverses lst in place.
>>> x = [3, 2, 4, 5, 1]
>>> reverse(x)
>>> x
[1, 5, 4, 2, 3]
"""
n = len(lst)
for i in range(n//2):
temp = lst[i]
lst[i] = lst[n-i-1]
lst[n-i-1] = temp
It works as expected:
>>> def reverse(lst):
... n = len(lst)
... for i in range(n//2):
... lst[i], lst[n-i-1] = lst[n-i-1], lst[i]
...
>>> lst = [1,2,3,4,5]
>>> reverse(lst)
>>> print(lst)
[5, 4, 3, 2, 1]
BTW, why don't you use list.reverse?
>>> lst = [1,2,3]
>>> lst.reverse()
>>> lst
[3, 2, 1]
The following java code exists but I'm trying to convert it to groovy. Should I simply keep it as is w/ the System.arraycopy or does groovy have a nicer way to combine arrays like this?
byte[] combineArrays(foo, bar, start) {
def tmp = new byte[foo.length + bar.length]
System.arraycopy(foo, 0, tmp, 0, start)
System.arraycopy(bar, 0, tmp, start, bar.length)
System.arraycopy(foo, start, tmp, bar.length + start, foo.length - start)
tmp
}
Thank you
def a = [1, 2, 3]
def b = [4, 5, 6]
assert a.plus(b) == [1, 2, 3, 4, 5, 6]
assert a + b == [1, 2, 3, 4, 5, 6]
If you want to use an array:
def abc = [1,2,3,4] as Integer[] //Array
def abcList = abc as List
def xyz = [5,6,7,8] as Integer[] //Array
def xyzList = xyz as List
def combined = (abcList << xyzList).flatten()
Using Lists:
def abc = [1,2,3,4]
def xyz = [5,6,7,8]
def combined = (abc << xyz).flatten()
def a = [1, 2, 3]
def b = [4, 5, 6]
a.addAll(b)
println a
>> [1, 2, 3, 4, 5, 6]
The trick is the flatten() method, that combined nested arrays into one:
def a = [1, 2, 3]
def b = [4, 5, 6]
def combined = [a, b].flatten()
assert combined == [1, 2, 3, 4, 5, 6]
println(combined)
To remove null values you can use findAll() like this:
def a = null
def b = [4, 5, 6]
def combined = [a, b].flatten().findAll{it}
assert combined == [4, 5, 6]
println(combined)
I'd go with
byte[] combineArrays(foo, bar, int start) {
[*foo[0..<start], *bar, *foo[start..<foo.size()]]
}
It could be done like this:
def newCombine(foo,bar,start) {
([].add + foo[0..<start]+bar+foo[start..<foo.size()]).flatten()
}
It works for all kinds of arrays (byte[]) or lists
All the solutions above fails if an array is undefined:
def a = [1,2]
def b
assert a+b == [1, 2, null]
which is probably not what you want.
Either test if the array exists before adding:
def a = [1,2,3,4]
def b // null array
def c = [0,4,null,6]
def abc = []
[a,b,c].each{ if (it) abc += it }
assert abc == [1, 2, 3, 4, 0, 4, null, 6]
,or add all and then filter the output:
(a+b+c).findAll{ it != null }
(assuming here that null isn't a valid value in the original arrays, which implies that the first solution is a lot better, even if it may not look Groovy enough.)