Is there a way to set particular values of parameters in the R package paradox? Say I do hyperparameter tuning for a random forest method and I want to test for mtry = c(2, 3, 7, 8) and min.node.size = c(2, 5, 7), i.e., a 4 x 3 grid with non-equal distances between the values.
Currently, I have to do a large 7 x 6 grid search to include these values, testing combinations that I'm not interested in:
tuner_params = ParamSet$new(list(
ParamInt$new("mtry", lower = 2, upper = 7),
ParamInt$new("min.node.size", lower = 2, upper = 6)
))
generate_design_grid(tuner_params, param_resolutions = c(mtry = 7, min.node.size = 5))
One way to overcome this is to not use grid search but TunerDesignPoints.
See example:
library(paradox)
library(mlr3)
library(mlr3tuning)
library(mlr3learners)
library(data.table)
tuner_params = ParamSet$new(list(
ParamInt$new("mtry", lower = 2, upper = 8),
ParamInt$new("min.node.size", lower = 2, upper = 7)
))
Specify custom design points:
design = data.table(expand.grid(mtry = c(2, 3, 7, 8),
min.node.size = c(2, 5, 7)))
tuner = tnr("design_points", design = design)
sonar_task = tsk("sonar")
r_lrn = lrn("classif.ranger", predict_type = "prob")
instance = TuningInstance$new(
task = sonar_task,
learner = r_lrn,
resampling = rsmp("cv", folds = 3),
measures = msr("classif.acc"),
param_set = tuner_params,
terminator = term("none")) #no terminator since you want all design points evaluated
tuner$tune(instance)
instance$archive()
#output
nr batch_nr resample_result task_id learner_id resampling_id iters params tune_x warnings errors classif.acc
1: 1 1 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8462388
2: 2 2 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8366460
3: 3 3 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8317460
4: 4 4 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8269151
5: 5 5 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8366460
6: 6 6 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8173913
7: 7 7 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8221532
8: 8 8 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8124914
9: 9 9 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8415459
10: 10 10 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8173223
11: 11 11 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8221532
12: 12 12 <ResampleResult> sonar classif.ranger cv 3 <list> <list> 0 0 0.8221532
12 points evaluated like we specified in the design grid.
Related
What I am doing wrong in the following code:
import dis
def count_operations(f):
operations = 0
for op in dis.get_instructions(f):
if op.opname in ('ADD', 'SUB', 'MULT', 'DIV', 'MOD'):
operations += 1
return operations
def solve_system(A, b):
x = np.linalg.solve(A, b)
return x
A = np.array([[2, 3],
[3, 4]])
b = np.array([8, 11])
operations = count_operations(solve_system)
print(f'Number of operations: {operations}')
I wrote two functions, one for counting operations and one for solving a system.
Numpy does a lot of the heavy lifting with (compiled) library routines written in C or Fortran. You won't see those in dis output:
In [1]: import numpy as np
In [2]: import dis
In [3]: def solve_system(A, b):
...: x = np.linalg.solve(A, b)
...: return x
...:
In [4]: dis.dis(solve_system)
2 0 LOAD_GLOBAL 0 (np)
2 LOAD_ATTR 1 (linalg)
4 LOAD_METHOD 2 (solve)
6 LOAD_FAST 0 (A)
8 LOAD_FAST 1 (b)
10 CALL_METHOD 2
12 STORE_FAST 2 (x)
3 14 LOAD_FAST 2 (x)
16 RETURN_VALUE
In [5]: dis.dis(np.linalg.solve)
179 0 LOAD_DEREF 0 (dispatcher)
2 LOAD_FAST 0 (args)
4 BUILD_MAP 0
6 LOAD_FAST 1 (kwargs)
8 DICT_MERGE 1
10 CALL_FUNCTION_EX 1
12 STORE_FAST 2 (relevant_args)
180 14 LOAD_GLOBAL 0 (implement_array_function)
181 16 LOAD_DEREF 1 (implementation)
18 LOAD_DEREF 2 (public_api)
20 LOAD_FAST 2 (relevant_args)
22 LOAD_FAST 0 (args)
24 LOAD_FAST 1 (kwargs)
180 26 CALL_FUNCTION 5
28 RETURN_VALUE
From the numpy.linalg.solve documentation:
The solutions are computed using LAPACK routine _gesv.
Those routines (sgesv and dgesv for single and double precision) are written in Fortran.
See e.g. the documentation for dgesv.
This is actually an interesting question. So you are making a wrapper function to show the amount of operations that are in a given tuple.
On inspection, the if statement is failing and you can verify this by including an else.
So i do this modification to verify:
import dis
import numpy as np
def count_operations(f):
operations = 0
not_operations = 0 # <---added this
for op in dis.get_instructions(f):
if op.opname in ('ADD', 'SUB', 'MULT', 'DIV', 'MOD'):
operations += 1
else:
not_operations += 1
return operations, not_operations
def solve_system(A, b):
x = np.linalg.solve(A, b)
return x
A = np.array([[2, 3],
[3, 4]])
b = np.array([8, 11])
operations = count_operations(solve_system)
print(f'Number of operations: {operations}')
The above code now returns a operations as a tuple (of ops and not ops).
When i run the modified code i get this:
Number of operations: (0, 9)
You can also add a print statement to expose what you get in each loop. An example of the first operation in the list is this:
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='np', argrepr='np', offset=0, starts_line=18, is_jump_target=False)
Which is <class 'dis.Instruction'>
So you could expose that...
Finally, i can show that the code does infact work, but adding LOAD_FAST into the tuple which then returns (3,6) as it occurs 3 times...
How to add aggregated error to keras model?
Having table:
g x y
0 1 1 1
1 1 2 2
2 1 3 3
3 2 1 2
4 2 2 1
I would like to be able to minimize sum((y - y_pred) ** 2) error along with
sum((sum(y) - sum(y_pred)) ** 2) per group.
I'm fine to have bigger individual sample errors, but it is crucial for me to have right totals.
SciPy example:
import pandas as pd
from scipy.optimize import differential_evolution
df = pd.DataFrame({'g': [1, 1, 1, 2, 2], 'x': [1, 2, 3, 1, 2], 'y': [1, 2, 3, 2, 1]})
g = df.groupby('g')
def linear(pars, fit=False):
a, b = pars
df['y_pred'] = a + b * df['x']
if fit:
sample_errors = sum((df['y'] - df['y_pred']) ** 2)
group_errors = sum((g['y'].sum() - g['y_pred'].sum()) ** 2)
total_error = sum(df['y'] - df['y_pred']) ** 2
return sample_errors + group_errors + total_error
else:
return df['y_pred']
pars = differential_evolution(linear, [[0, 10]] * 2, args=[('fit', True)])['x']
print('SAMPLES:\n', df, '\nGROUPS:\n', g.sum(), '\nTOTALS:\n', df.sum())
Output:
SAMPLES:
g x y y_pred
0 1 1 1 1.232
1 1 2 2 1.947
2 1 3 3 2.662
3 2 1 2 1.232
4 2 2 1 1.947
GROUPS:
x y y_pred
g
1 6 6 5.841
2 3 3 3.179
TOTALS:
g 7.000
x 9.000
y 9.000
y_pred 9.020
For grouping, as long as you keep the same groups throughout training, your loss function will not have problems about being not differentiable.
As a naive form of grouping, you can simply separate the batches.
I suggest a generator for that.
#suppose you have these three numpy arrays:
gTrain
xTrain
yTrain
#create this generator
def grouper(g,x,y):
while True:
for gr in range(1,g.max()+1):
indices = g == gr
yield (x[indices],y[indices])
For the loss function, you can make your own:
import keras.backend as K
def customLoss(yTrue,yPred):
return K.sum(K.square(yTrue-yPred)) + K.sum(K.sum(yTrue) - K.sum(yPred))
model.compile(loss=customLoss, ....)
Just be careful with the second term if you have negative values.
Now you train using the method fit_generator:
model.fit_generator(grouper(gTrain,xTrain, yTrain), steps_per_epoch=gTrain.max(), epochs=...)
I am implementing the A star algorithm with the Manhattan distance for the 8 puzzle. [ The solution is in spiral form]
1 2 3
8 0 4
7 6 5
In some case, going from A to B will not take the same number of steps as going from B to A.
I think this is because it does not pick the same state on the open list, when they have the same cost, thus, not expanding the same nodes.
From
7 6 4
1 0 8
2 3 5
(A -> B)
7 6 4
1 8 0
2 3 5
(B -> A)
7 6 4
1 3 8
2 0 5
Which both have the same value using Manhattan distance.
Should I explore all path with the same value?
Or should I change the heuristic to have some kind of tie-breaker?
Here is the relevant part of the code
def solve(self):
cost = 0
priority = 0
self.parents[str(self.start)] = (None, 0, 0)
open = p.pr() #priority queue
open.add(0, self.start, cost)
while open:
current = open.get()
if current == self.goal:
return self.print_solution(current)
parent = self.parents[str(current)]
cost = self.parents[str(current)][2] + 1
for new_state in self.get_next_states(current):
if str(new_state[0]) not in self.parents or cost < self.parents[str(new_state[0])][2]:
priority = self.f(new_state) + cost
open.add(priority, new_state[0], cost)
self.parents[str(new_state[0])] = (current, priority, cost)
After wasting so much time re-writing my "solve" function many different ways, for nothing,
I finally found the problem.
def get_next_states(self, mtx, direction):
n = self.n
pos = mtx.index(0)
if direction != 1 and pos < self.length and (pos + 1) % n:
yield (self.swap(pos, pos + 1, mtx),pos, 3)
if direction != 2 and pos < self.length - self.n:
yield (self.swap(pos, pos + n, mtx),pos, 4)
if direction != 3 and pos > 0 and pos % n:
yield (self.swap(pos, pos - 1, mtx),pos, 1)
if direction != 4 and pos > n - 1:
yield (self.swap(pos, pos - n, mtx),pos, 2)
It was in this function. The last if used to be "if 4 and pos > n:"
So there were unexplored states..
2 days for a "-1"
It will teach me to do more unit testing
I am creating a minesweeper program for school. Part of this is randomly assigning coordinates for 10 bombs. These will be placed in a 9x9 square. I don't want the bomb coordinates to repeat. Then I remove the coordinates of the bomb from the list buttonNames. I added print statements to try to figure out why this code didn't work.
This is my code:
for i in range(11):
Bombx = randrange(1,10)
Bomby = randrange(1,10)
newBomb = (Bombx,Bomby)
self.BombList.append(newBomb)
sumValues = []
for (x,y) in self.BombList:
value = x+y
print(x,y)
sumValues.append(value)
for value in sumValues:
count = sumValues.count(value)
while count != 1:
Bombx = randrange(1,10)
Bomby = randrange(1,10)
newBomb = (Bombx,Bomby)
oldBomb = (x, y)
print(oldBomb)
self.BombList.remove(oldBomb)
self.BombList.append(newBomb)
count = 1
self.buttonNames.remove(newBomb)
And it outputs this:
3 4
3 4
6 8
3 4
6 8
4 4
3 4
6 8
4 4
5 4
3 4
6 8
4 4
5 4
9 2
3 4
6 8
4 4
5 4
9 2
4 2
3 4
6 8
4 4
5 4
9 2
4 2
6 1
(6, 1)
(6, 1)
Traceback (most recent call last):
File "/Users/gould942/Documents/Intro Comp Sci/Minesweeper.py", line 222, in <module>
main()
File "/Users/gould942/Documents/Intro Comp Sci/Minesweeper.py", line 219, in main
gameBoard = Sweeper(win)
File "/Users/gould942/Documents/Intro Comp Sci/Minesweeper.py", line 98, in __init__
self.BombList.remove(oldBomb)
ValueError: list.remove(x): x not in list
Any help is appreciated.
I'd ordinarily prefer to try to help you work out why your code is doing what it's doing, but to be honest I'm having a hard time trying to follow the logic. Instead I'll say that since there are only 9x9=81 options to choose from, the simple solution is to generate a sequence of all possible options and use random.sample():
self.BombList = []
for x in range(1, 10):
for y in range(1, 10):
self.BombList.append((x, y))
self.BombList = random.sample(self.BombList, 9)
If you're familiar with list comprehensions, the whole thing can be done in one line:
self.BombList = random.sample([(x, y) for x in range(1, 10) for y in range(1, 10)], 9)
Here's the text document: The first string is the type of metal, the second is the amount of the metal bars, the third is the weight, and the fourth is the value.
Gold 1 5 750
Silver 1 1 400
Rhodium 1 4 500
Platinum 1 6 1000
I have to sort this list by value using insertion sort. Here's what I have so far
def sortMetalsByValuePerBar(metals):
for i in range(1,len(metals)):
j = i
while j > 0 and metals[j-1] > metals[j]:
metals[j - 1], metals[j] = metals[j], metals[j - 1]
j -= 1
return metals
Is this correct?
Try to learn from this solution.
data="""
Gold 1 5 750
Silver 1 1 400
Rhodium 1 4 500
Platinum 1 6 1000
"""
data = filter(None, data.splitlines())
data = [l.split() for l in data]
data = [ (l[0], int(l[1]), int(l[2]), int(l[3])) for l in data ]
def insertion_sort(l, keyfunc=lambda i:i):
for i in range(1, len(l)):
j = i-1
key = l[i]
while keyfunc(l[j]) > keyfunc(key) and (j >= 0):
l[j+1] = l[j]
j -= 1
l[j+1] = key
insertion_sort(data, keyfunc=lambda l: l[3])
for l in data:
print(l)
# Output:
# ('Silver', 1, 1, 400)
# ('Rhodium', 1, 4, 500)
# ('Gold', 1, 5, 750)
# ('Platinum', 1, 6, 1000)