Print statement does not get executed when MPI Barrier introduced - python-3.x

I am using python and mpi4py, and have encountered a scenario I do not understand. The below code is a minimal working example mwe.py.
import numpy as np
from mpi4py import MPI
import time
import itertools
N=8
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
N_sub = comm.Get_size()-1
get_sub = itertools.cycle(range(1, N_slaves+1))
if rank == 0:
print("I am rank {}".format(rank))
data = []
for i in range(N):
nums = np.random.normal(size=6)
data.append(nums)
sleep_short = 0.0001
sleep_long = 0.1
sleep_dict = {}
for r in range(1, N_sub+1):
sleep_dict[str(r)] = 1
data_idx = 0
while len(data) > data_idx:
r = next(get_sub)
if comm.iprobe(r):
useless_info = comm.recv(source=r)
print(useless_info)
comm.send(data[data_idx], dest=r)
data_idx += 1
print("data_idx {}".format(data_idx))
sleep_dict[str(r)] = 1
else:
sleep_dict[str(r)] = 0
if all(value == 0 for value in sleep_dict.values()):
time.sleep(sleep_long)
else:
time.sleep(sleep_short)
for r in range(1, N_sub+1):
comm.send('Done', dest=r)
else:
print("rank {}".format(rank))
######################
# vvv This is the statement in question
######################
comm.Barrier()
while True:
comm.send("I am done", dest=0)
model = comm.recv(source=0)
if type(model).__name__ == 'str':
break
MPI.Finalize()
sys.exit(0)
When run with mpirun -np 4 python mwe.py, this code generates an array containing lists of random numbers, and then distributes these lists to the "sub" ranks until all arrays have been sent. Understandably, if I insert a comm.Barrier() call near the bottom (where I have indicated in the code), the code no longer completes execution, as the sub ranks (not equal to 0) never get to the statement where they are to receive what is being sent from rank 0. Rank 0 keeps trying to find a rank to pass the array to, but never does since the other ranks are held up, and the code hangs.
This makes sense to me. What doesn't make sense to me is that with the comm.Barrier() statement included, the preceding print statement also does not execute. Based on my understanding, the sub ranks should proceed normally until they hit the barrier statement, and then wait there until all of the ranks 'catch up', which in this case never happens because rank 0 is in its own loop. If this is the case, the preceding print statement should be executed, as it comes before those ranks have gotten to the barrier line. So why does the statement not get printed? Can anyone explain where my understanding fails?

Related

python return / print highest number of value in a row in a list

I am a beginner.
Trying to make a function that can return the highest number in a row a value in a list was picked.
Basicly I want to simulate a head or tails game, and find out how many times in a row it was for example tails.
import random
def simulations(runs):
sequence = []
in_a_row = []
tæller = 0
for run in range(0, int(runs)):
my_random = random.randint(0,1)
sequence.append(my_random)
if my_random == 0:
in_a_row.append(my_random)
tæller = tæller + 1
print()
elif my_random == 1:
tæller = 0
in_a_row = []
print()
print(f'{tæller}')
print(sequence)
# runs = input('How many simulations do you want to run ? ')
runs = 10
simulations(runs)
I think I am almost there. But I cant figure it out, when I run this code I first print the actual value that was picked, and then see the list that was created so far.
if for example my_random == 0 3 times in a row it will print each time it was 0 until it picks 1.
Want i want is for the code to print only the longest sequence where 0 was picked in the entire list
I know my code is spagetti, and the function is not finished, but i belive I can do that by myself. I just need some help with isolating the number of times a given value was picked in a row from the list.
Hmm I think I was finally able to create a solution by myself.
So happy i think i spend like 5 hours trying to figure it out.
Here is my solution. I would still appriciate it a lot if someone can comment on my solution, and maybe show another more simple way of doing it :-)
import random
def simulations(runs):
sequence = []
in_a_row = []
found_zero = 0
all_time_high = [0]
for run in range(int(runs)):
my_random = random.randint(0,1)
sequence.append(my_random)
if my_random == 0:
in_a_row.append(my_random)
found_zero = found_zero + 1
print(f'fundet 0 ja: {found_zero} Gange')
all_time_high.append(found_zero)
elif my_random == 1:
found_zero = 0
in_a_row = []
print()
print(sequence)
print(f'Higest number of times in a row zero was found: {max(all_time_high)}')
# runs = input('How many simulations do you want to run ? ')
runs = 10
simulations(runs)

I have tried solving this problem, encountered a maximum recursion depth exceeded error

I am a bit of a noob, please go easy on me.
I'm trying to solve this.
The input I'm using.
So I have written code using mammoth amount of input and it keeps on piling up every time function is called.
I want to understand how to approach this and not get the Maximum Recursion Depth Exceeded Error
count = 0
#a #I assign the whole input here
def count_bag(x):
#a #I assign the whole input here again
for x in a:
if x == "other bags":
continue
elif "shiny gold" in a[x]:
global count
count += 1
break
else:
for y in a[x]:
count_bag(y)
count_bag(a)
print(count)
In python the recursion depth is limited for most compilers, so if you want to do some deep recursion you can use sys.setrecursionlimit(some value) to get over this issue
import sys
sys.setrecursionlimit(10**6)
Why not use a queue and skip the recursion? That way, you don't even need to use a global count variable. I haven't compiled/linted this or anything, but something like this:
def count_bag(x, a):
count = 0
to_process = list()
to_process.extend(x)
while len(to_process) > 0:
item = to_process.pop(0) # pop item off front of list
if item == 'other bags':
continue
elif 'shiny gold' in a[item]:
count += 1
else:
for y in a[item]:
to_process.extend(y)
return count

A strategy-proof method of finding the time complexity of complex algorithms?

I have a question in regard to time complexity (big-O) in Python. I want to understand the general method I would need to implement when trying to find the big-O of a complex algorithm. I have understood the reasoning behind calculating the time complexity of simple algorithms, such as a for loop iterating over a list of n elements having a O(n), or having two nested for loops each iterating over 2 lists of n elements each having a big-O of n**2. But, for more complex algorithms that implement multiple if-elif-else statements coupled with for loops, I would want to see if there is a strategy to, simply based on the code, in an iterative fashion, to determine the big-O of my code using simple heuristics (such as, ignoring constant time complexity if statements or always squaring the n upon going over a for loop, or doing something specific when encountering an else statement).
I have created a battleship game, for which I would like to find the time complexity, using such an aforementioned strategy.
from random import randint
class Battle:
def __init__(self):
self.my_grid = [[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False],[False,False,False,False,False,False,False,False,False,False]]
def putting_ship(self,x,y):
breaker = False
while breaker == False:
r1=x
r2=y
element = self.my_grid[r1][r2]
if element == True:
continue
else:
self.my_grid[r1][r2] = True
break
def printing_grid(self):
return self.my_grid
def striking(self,r1,r2):
element = self.my_grid[r1][r2]
if element == True:
print("STRIKE!")
self.my_grid[r1][r2] = False
return True
elif element == False:
print("Miss")
return False
def game():
battle_1 = Battle()
battle_2 = Battle()
score_player1 = 0
score_player2 = 0
turns = 5
counter_ships = 2
while True:
input_x_player_1 = input("give x coordinate for the ship, player 1\n")
input_y_player_1 = input("give y coordinate for the ship, player 1\n")
battle_1.putting_ship(int(input_x_player_1),int(input_y_player_1))
input_x_player_2 = randint(0,9)
input_y_player_2 = randint(0,9)
battle_2.putting_ship(int(input_x_player_2),int(input_y_player_2))
counter_ships -= 1
if counter_ships == 0:
break
while True:
input_x_player_1 = input("give x coordinate for the ship\n")
input_y_player_1 = input("give y coordinate for the ship\n")
my_var = battle_1.striking(int(input_x_player_1),int(input_y_player_1))
if my_var == True:
score_player1 += 1
print(score_player1)
input_x_player_2 = randint(0,9)
input_y_player_2 = randint(0,9)
my_var_2 = battle_2.striking(int(input_x_player_2),int(input_y_player_2))
if my_var_2 == True:
score_player2 += 1
print(score_player2)
counter_ships -= 1
if counter_ships == 0:
break
print("the score for player 1 is",score_player1)
print("the score for player 2 is",score_player2)
print(game())
If it's just nested for loops and if/else statements, you can take the approach ibonyun has suggested - assume all if/else cases are covered and look at the deepest loops (being aware that some operations like sorting, or copying an array, might hide loops of their own.)
However, your code also has while loops. In this particular example it's not too hard to replace them with fors, but for code containing nontrivial whiles there is no general strategy that will always give you the complexity - this is a consequence of the halting problem.
For example:
def collatz(n):
n = int(abs(n))
steps = 0
while n != 1:
if n%2 == 1:
n=3*n+1
else:
n=n//2
steps += 1
print(n)
print("Finished in",steps,"steps!")
So far nobody has been able to prove that this will even finish for all n, let alone shown an upper bound to the run-time.
Side note: instead of the screen-breaking
self.my_grid = [[False,False,...False],[False,False,...,False],...,[False,False,...False]]
consider something like:
grid_size = 10
self.my_grid = [[False for i in range(grid_size)] for j in range(grid_size)]
which is easier to read and check.
Empirical:
You could do some time trials while increasing n (so maybe increasing the board size?) and plot the resulting data. You could tell by the curve/slope of the line what the time complexity is.
Theoretical:
Parse the script and keep track of the biggest O() you find for any given line or function call. Any sorting operations will give you nlogn. A for loop inside a for loop will give you n^2 (assuming their both iterating over the input data), etc. Time complexity is about the broad strokes. O(n) and O(n*3) are both linear time, and that's what really matters. I don't think you need to worry about the minutia of all your if-elif-else logic. Maybe just focus on worst case scenario?

String subpattern recognition optimization

In this kata you need to build a function to return either true/True or false/False if a string can be seen as the repetition of a simpler/shorter subpattern or not.
For example:
has_subpattern("a") == False #no repeated pattern
has_subpattern("aaaa") == True #created repeating "a"
has_subpattern("abcd") == False #no repeated pattern
has_subpattern("abababab") == True #created repeating "ab"
has_subpattern("ababababa") == False #cannot be entirely reproduced repeating a pattern
Strings will never be empty and can be composed of any character (just consider upper- and lowercase letters as different entities) and can be pretty long (keep an eye on performances!).
My solution is:
def has_subpattern(string):
string_size = len(string)
for i in range(1, string_size):
slice1 = string[:i]
appearence_count = string.count(slice1)
slice1_len = len(slice1)
if appearence_count > 0:
if appearence_count * slice1_len == string_size:
return True
return False
Obviously there are weak and too slow things like slice1 = string[:i] and string.count() in loop..
Is there better ways to solve an issue or ways to improve performance ?
Short regex approach:
import re
def has_subpattern_re(s):
return bool(re.search(r'^(\w+)\1+$', s))
It'll provide a close (to initial has_subpattern approach) performance on small strings:
import timeit
...
print(timeit.timeit('has_subpattern("abababab")', 'from __main__ import has_subpattern'))
0.7413144190068124
print(timeit.timeit('has_subpattern_re("abababab")', 'from __main__ import re, has_subpattern_re'))
0.856149295999785
But, a significant performance increase (in about 3-5 times faster) on long strings:
print(timeit.timeit('has_subpattern("ababababababababababababababababababababababababa")', 'from __main__ import has_subpattern'))
14.669428467008402
print(timeit.timeit('has_subpattern_re("ababababababababababababababababababababababababa")', 'from __main__ import re, has_subpattern_re'))
4.308312018998549
And one more test for a more longer string:
print(timeit.timeit('has_subpattern("ababababababababababababababababababababababababaababababababababababababababababababababababababab")', 'from __main__ import has_subpattern'))
35.998031173992786
print(timeit.timeit('has_subpattern_re("ababababababababababababababababababababababababaababababababababababababababababababababababababab")', 'from __main__ import re, has_subpattern_re'))
7.010367843002314
Within standard Python, the bottlenecks here will be count, which enjoys C speed implementation and the looping.
The looping itself may be hard to speed-up (althogh Cython may be of some help).
Hence, the most important optimization is to reduce the number of loopings.
One obvious way is to let range() do not exceed half the size of the input (+ 2: + 1 for rounding issues, + 1 for end extrema exclusion in range()):
Also, string is a standard Python module, so better not use it as a variable name.
def has_subpattern_loop(text):
for i in range(1, len(text) // 2 + 2):
subtext = text[:i]
num_subtext = text.count(subtext)
if num_subtext > 1 and num_subtext * len(subtext) == len(text):
return True
return False
A much more effective way of restricting the number of calls to count is to skip computation when i is not a multiple of the length of the input.
def has_subpattern_loop2(text):
for i in range(1, len(text) // 2 + 2):
if len(text) % i == 0:
subtext = text[:i]
num_subtext = text.count(subtext)
if num_subtext > 1 and num_subtext * len(subtext) == len(text):
return True
return False
Even better would be to generate only the divisors of the length of the input.
This could be done using sympy and the approach outlined here:
import sympy as sym
import functools
def get_divisors(n):
if n == 1:
yield 1
return
factors = list(sym.factor_.factorint(n).items())
nfactors = len(factors)
f = [0] * nfactors
while True:
yield functools.reduce(lambda x, y: x * y, [factors[x][0]**f[x] for x in range(nfactors)], 1)
i = 0
while True:
f[i] += 1
if f[i] <= factors[i][1]:
break
f[i] = 0
i += 1
if i >= nfactors:
return
def has_subpattern_divs(text):
for i in get_divisors(len(text)):
subtext = text[:i]
num_subtext = text.count(subtext)
if num_subtext > 1 and num_subtext * len(subtext) == len(text):
return True
return False
A completely different approach is the one proposed in #ВладДавидченко answer:
def has_subpattern_find(text):
return (text * 2).find(text, 1) != len(text)
or the more memory efficient (requires ~50% less additional memory compared to has_subpattern_find2()):
def has_subpattern_find2(text):
return (text + text[:len(text) // 2 + 2]).find(text, 1) > 0
and it is based on the idea that if there is a exactly self-repeating string, the string itself must be found in a circularly extended string:
Input: abab
Extension1: abababab
Found1: |-abab
Extension2: ababab
Found2: |-abab
Input: ababa
Extension1: ababaababa
Found1: |----ababa
Extension2: ababab
Found2: NOT FOUND!
The find-based method are the fastest, with has_subpattern_find() being fastest in the small input size regime, and has_subpattern_find2() gets generally faster in the intermediate and large input size regime (especially in the False case).
For shorter inputs, the direct looping approaches (especially has_subpattern_loop2()) are fastest, closely followed (but sometimes surpassed by has_subpattern_re()), but as soon as the input gets bigger (and especially for the False outcome), the has_subpattern_divs() method gets to be the fastest (aside of find-based ones) by far and large, as shown by the following benchmarks.
For the True outcome, has_subpattern_loop2() gets to be the fastest due to the very small number of loops required, which is independent of the input size.
The input is generated as a function of n using:
def gen_input(n, m=0):
g = string.ascii_lowercase
if not m:
m = n
offset = '!' if n % 2 else ''
return g[:n] * (m // min(n, len(g)) + 2) + offset
so that if n is even, the has_subpattern*() always return True and the opposite for odd n.
Note that, in general, the has_subpattern() function will depend not only on the raw size of the input but also on the length of the repeating string, if any. This is not explored in the benchmarks, except for the odd/even separation.
Even Inputs
Odd Inputs
(Full code available here).
(EDITED to include some more solutions as well as comparison with regex-based solution from #RomanPerekhrest)
(EDITED to include some more solutions based on the find from #ВладДавидченко)
Found another one solution, probably will be useful:
def has_subpattern(string):
return (string * 2).find(string, 1) != len(string)

How to determine which nested generator produces StopIteration exception?

I bumped into a situation where I need to determine in my try/except code which nested generator is raising a StopIteration exception. How do I do it? The following is a dummy example:
def genOne(iMax, jMax):
i = 0;
g2 = genTwo(jMax)
while i <= iMax:
print('genOne: ' + str(i))
next(g2)
yield
i = i + 1
def genTwo(jMax):
j = 0;
while j <= jMax:
print('genTwo: ' + str(j))
yield
j = j + 1
g1 = genOne(6, 3) # The inputs are arbitrary numbers
try:
while True:
next(g1)
except:
# Do some processing depending on who generates the StopIteration exception
Thanks!
This can be generalized to the problem of finding the origin of an arbitrary exception.
Use the traceback module to inspect the stacktrace of your exception object.
Here is a previous answer on a similar subject.
Some example code:
g1 = genOne(6, 3) # The inputs are arbitrary numbers
try:
while True:
next(g1)
except:
exc_type, exc_value, exc_traceback = sys.exc_info()
print(traceback.extract_tb(exc_traceback)[-1])
Shell output:
> ./test.py
genOne: 0
genTwo: 0
genOne: 1
genTwo: 1
genOne: 2
genTwo: 2
genOne: 3
genTwo: 3
genOne: 4
('./test.py', 12, 'genOne', 'next(g2)')
Note that the [-1] in the extract_tb() call explicitly checks only the first lower level of the stacktrace. With the print you can see which element of that output you'd need to check (genOne -> item index #2 in that list). In your particular example you'd probably want to check if the lowest level generator string genTwo exists in any of the elements of the traceback.extract_tb(exc_traceback) array.
Those hardcoded checks relying on internal code details are been frowned upon, especially since in your particular example you do not have control over their implementation.

Resources