I have written some code that uses the cvxpy library to solve an integer programming problem, however the code is taking so much time to run I was wondering if there is any way to make the code faster?
The integer programming problem in this case takes in a matrix of shape (1,569 x 3,071), and it has 3,071 constraints to satisfy. The code is as follows:
mat_f = sys.argv[1]
matIdx2genome_dic_f = sys.argv[2]
genomes_f = sys.argv[3]
with open(matIdx2genome_dic_f, 'r') as in_f:
matIdx2genome_dic = json.load(in_f)
M = np.load(mat_f)
selection = cp.Variable(M.shape[1], boolean = True)
ones_vec = np.ones(M.shape[1])
constraints = []
for i in range(len(M)):
constraints.append(M[i] * selection >= 1)
total_genomes = ones_vec * selection
problem = cp.Problem(cp.Minimize(total_genomes), constraints)
print('solving the integer programming problem: ')
time = time.time()
problem.solve(parallel = True)
print('problem solved in: '+ str(time.time() - time))
solution = selection.value
solution = list(map(round, solution))
solution = np.array(solution)
which_genomes = np.where(solution == 1.0)[0]
with open(genomes_f, 'w') as out_f:
for idx in which_genomes:
out_f.write(matIdx2genome_dic[idx]+'\n')
The first command line argument is what's important here, it's a numpy binary matrix that is of shape (1569, 3071).
The problem here is to minimize the number of columns of the matrix needed such that each and every row has at least a 1 in it's rows.
My question is how can I write this script so that it can run faster? is there a way to parallelize it? I have set the parallel parameter to True in the solve method by I don't think it's doing much since I'm monitoring the cpu utilization and it's only 100%, so I don't think that parallel option is doing much?
Or is there another way (solver that I should call maybe) that would solve this faster?
Related
I want to generate random numbers in Julia using multi-threading. I am using the
Threads.#threads macro to accomplish it. However, I struggle fixing the number of seeds to obtain the same result every time I run the code. Here is my trial:
Random.seed!(1234)
a = [Float64[] for _ in 1:10]
Threads.#threads for i = 1:10
push!(a[Threads.threadid()],rand())
end
sum(reduce(vcat, a))
The script above delivers different results every time I run it. By contrast, I get the same results if I use a plain for loop:
Random.seed!(12445)
b = []
for i = 1:10
push!(b,rand())
end
sum(b)
I have the impression that the solution to this issue must be easy. Still, I couldn't find it. Any help is much appreciated.
Thank you.
You need to generate a separate random stream for each thread.
The simplest way is to have a random number generator with a different seed:
using Random
rngs = [MersenneTwister(i) for i in 1: Threads.nthreads()];
Threads.#threads for i = 1:10
val = rand(rngs[Threads.threadid()])
# do something with val
end
If you do not want to risk correlation for different random number seeds you could actually jump around a single number generator:
julia> rngs2 = Future.randjump.(Ref(MersenneTwister(0)), big(10)^20 .* (1:Threads.nthreads()))
4-element Vector{MersenneTwister}:
MersenneTwister(0, (200000000000000000000, 0))
MersenneTwister(0, (400000000000000000000, 0))
MersenneTwister(0, (600000000000000000000, 0))
MersenneTwister(0, (800000000000000000000, 0))
Ciao Fabrizio. In BetaML I solved this problem with:
"""
generateParallelRngs(rng::AbstractRNG, n::Integer;reSeed=false)
For multi-threaded models, return n independent random number generators (one per thread) to be used in threaded computations.
Note that each ring is a _copy_ of the original random ring. This means that code that _use_ these RNGs will not change the original RNG state.
Use it with `rngs = generateParallelRngs(rng,Threads.nthreads())` to have a separate rng per thread.
By default the function doesn't re-seed the RNG, as you may want to have a loop index based re-seeding strategy rather than a threadid-based one (to guarantee the same result independently of the number of threads).
If you prefer, you can instead re-seed the RNG here (using the parameter `reSeed=true`), such that each thread has a different seed. Be aware however that the stream of number generated will depend from the number of threads at run time.
"""
function generateParallelRngs(rng::AbstractRNG, n::Integer;reSeed=false)
if reSeed
seeds = [rand(rng,100:18446744073709551615) for i in 1:n] # some RNGs have issues with too small seed
rngs = [deepcopy(rng) for i in 1:n]
return Random.seed!.(rngs,seeds)
else
return [deepcopy(rng) for i in 1:n]
end
end
The function above deliver the same results also independently of the number of threads used in Julia and can then be used for example like here:
using Test
TESTRNG = MersenneTwister(123)
println("** Testing generateParallelRngs()...")
x = rand(copy(TESTRNG),100)
function innerFunction(bootstrappedx; rng=Random.GLOBAL_RNG)
sum(bootstrappedx .* rand(rng) ./ 0.5)
end
function outerFunction(x;rng = Random.GLOBAL_RNG)
masterSeed = rand(rng,100:9999999999999) # important: with some RNG it is important to do this before the generateParallelRngs to guarantee independance from number of threads
rngs = generateParallelRngs(rng,Threads.nthreads()) # make new copy instances
results = Array{Float64,1}(undef,30)
Threads.#threads for i in 1:30
tsrng = rngs[Threads.threadid()] # Thread safe random number generator: one RNG per thread
Random.seed!(tsrng,masterSeed+i*10) # But the seeding depends on the i of the loop not the thread: we get same results indipendently of the number of threads
toSample = rand(tsrng, 1:100,100)
bootstrappedx = x[toSample]
innerResult = innerFunction(bootstrappedx, rng=tsrng)
results[i] = innerResult
end
overallResult = mean(results)
return overallResult
end
# Different sequences..
#test outerFunction(x) != outerFunction(x)
# Different values, but same sequence
mainRng = copy(TESTRNG)
a = outerFunction(x, rng=mainRng)
b = outerFunction(x, rng=mainRng)
mainRng = copy(TESTRNG)
A = outerFunction(x, rng=mainRng)
B = outerFunction(x, rng=mainRng)
#test a != b && a == A && b == B
# Same value at each call
a = outerFunction(x,rng=copy(TESTRNG))
b = outerFunction(x,rng=copy(TESTRNG))
#test a == b
Assuming you are on Julia 1.6 you can do e.g. the following:
julia> using Random
julia> foreach(i -> Random.seed!(Random.default_rng(i), i), 1:Threads.nthreads())
The point is that currently Julia already has a separate random number generator per thread so you do not need to generate your own (of course you could do it as in the other answers, but you do not have to).
Also note that in the future versions of Julia the:
Threads.#threads for i = 1:10
push!(a[Threads.threadid()],rand())
end
part is not guaranteed to produce reproducible results. In Julia 1.6 Threads.#threads uses static scheduling, but as you can read in its docstring it is subject to change.
I am trying to run a specific code to find the sum of all the possible combinations of the list that is coming from a .in file. The same code, when running with relatively small files, runs perfectly and with bigger files hangs and after a bit throws MEMORY ERROR
import itertools
file = open("c_medium.in","r")
if file.mode=='r':
content = file.readlines()
maxSlices,numberOfPizza = map(int,content[0].split())
numberOfSlices = tuple(map(int,content[1].split()))
print(maxSlices)
print(numberOfSlices)
sol = []
sumOfSlices = []
for x in range(1,len(numberOfSlices)+1):
print(x)
for y in itertools.combinations(numberOfSlices,x):
if sum(y) <= maxSlices:
sumOfSlices.append(sum(y))
sumOfSlices.sort()
print(sumOfSlices)
checkSum = sumOfSlices[len(sumOfSlices)-1]
print(checkSum)
found = False
if found == False:
for x in range(1,len(numberOfSlices)+1):
print(x)
for y in itertools.combinations(numberOfSlices,x):
if found == False:
if sum(y) == checkSum:
for z in y:
sol.append(numberOfSlices.index(z))
found = True
solution = tuple(map(str,sol))
print(solution)
The number of combinations of N elements grows very, very fast with N.
Regarding your code in particular, if (sum(y) <= maxSlices) is always true, then you'll generate a list with 2^(numberOfSlices) elements. i.e., you'll overflow a 32-bit integer if numberOfSlices=32.
I'd recommend trying to solve your task without explicitly building a list. If you describe what your code is doing, maybe someone can help.
I have a set of points (represented by complex values), and I need to find the shortest path through these. It looks a bit like the travelling salesman problem, but I can't seem to find (or understand) a solution that isn't in O(n!). I know how to compute short enough solutions in O(n^3), O(n²), but I wanted to know if it was possible to have THE best one. Thank you !
There's the code I use for a "Short Enough Path"
def insert(x,liste,taille):
max_add = 10**9
n = len(liste) -1
for i in range(n):
test = abs(liste[i] -x) + abs(liste[i+1] - x) - taille[i]
if test < max_add:
max_add = test
i_max = i
taille[i_max] = abs(liste[i_max]-x)
taille.insert(i_max+1,abs(liste[i_max+1] - x))
liste.insert(i_max+1,x)
def sort(x,i=0):
taille = [0]
tri = [x[i]]*2
for y in x[:i]+x[i+1:]:
inserer(y,tri,taille)
return tri, taille
def the_best(liste):
n = len(liste)
shortest = 10**9
for i in range(n):
a,b = sort(liste,i)
if sum(b) < shortest:
back = a,b
return back
`
Of course the "the_best" function is in O(n^3) so I usually use the "sort" function only
The list called "taille" is built like this:
taille[i] = abs(liste[i] - liste[i+1])
liste[-1] = liste[0]
From what I understand in your description, this is indeed the TSP problem. It is a well-known NP-hard problem, and as such an efficient algorithm to solve it does not exist (even if it does, we don't know of it yet). It's one of the famous open problems in Computer Science.
Indeed, do give it a try to solve it, but do not hold your breath :)
General reading: https://en.wikipedia.org/wiki/Travelling_salesman_problem
You may also want to give a quick read to: https://en.wikipedia.org/wiki/P_versus_NP_problem
I have successfully extracted the count of a specific word from a dataset but, it is taking too much time. I am new to parallel programming.
How can I create parallelism in the following code:
df = dd.read_csv('crime.csv', encoding="ISO-8859-1")
distinct_values = df.YEAR.unique().compute()
counter = len(distinct_values)
values_count = {}
for i in distinct_values:
count = df[df.YEAR == i].YEAR.value_counts().compute()
values_count.update(count)
list = []
for x, y in values_count.items():
dict = {}
for i in x, y:
dict['name'] = x
dict['value'] = y
# print(dict)
list.append(dict)
# print(list)
maximum = max(distinct_values)
mininmum = min(distinct_values)
Maybe you're looking for a groupby aggregation like the following?
df.groupby("YEAR").count.compute()
Or, if you need to do this as many operations, you should at least use the dask.compute function with many inputs rather than call the .compute method many times.
the function shown below is running quite slow even though I used swifter to call it. Does anyone know how to speed this up? My python knowledge is limited at this point and I would appreciate any help I could get. I tried using map() function but somehow it didnt work for me. I guess the nested for loop makes it rather slow, right?
BR,
Hannes
def polyData(uniqueIds):
for index in range(len(uniqueIds) - 1):
element = uniqueIds[index]
polyData1 = df[df['id'] == element]
poly1 = build_poly(polyData1)
poly1 = poly1.buffer(0)
for secondIndex in range(index + 1, len(uniqueIds)):
otherElement = uniqueIds[secondIndex]
polyData2 = df[df['id'] == otherElement]
poly2 = build_poly(polyData2)
poly2 = poly2.buffer(0)
# Calculate overlap percentage wise
overlap_pct = poly1.intersection(poly2).area/poly1.area
# Form new DF
df_ol = pd.DataFrame({'id_1':[element],'id_2':[otherElement],'overlap_pct':[overlap_pct]})
# Write to SQL database
df_ol.to_sql(name='df_overlap', con=e,if_exists='append',index=False)
This function is inherently slow for large amounts of data due to its complexity (trying every 2-combination of a set). However, you're calculating the 'poly' for the same ids multiple times, even though it seems that you can calculate them only once beforehand (which might be expensive) and store them for later usage. So try to extract the building of the polys.
def getPolyForUniqueId(uid):
polyData = df[df['id'] == uid]
poly = build_poly(polyData)
poly = poly.buffer(0)
return polyData
def polyData(uniqueIds):
polyDataList = [getPolyForUniqueId(uid) for uid in uniqueIds]
for index in range(len(uniqueIds) - 1):
id_1 = uniqueIds[index]
poly_1 = polyDataList[index]
for secondIndex in range(index + 1, len(uniqueIds)):
id_2 = uniqueIds[secondIndex]
poly_2 = polyDataList[secondIndex]
...