Related
I Used SMOTE and Tomek methods for imbalanced classes that I have. I'm trying to do boosted regression tree.
It runs smoothly until I create the confusion matrix I have this error (
Error: data and reference should be factors with the same levels.
### SMOTE and Tomek
NOAA_SMOTE= read.csv("NOAA_SMOTE.csv", TRUE, ",")
train.index <- createDataPartition(NOAA_SMOTE$japon, p = .7, list = FALSE)
train <- NOAA_SMOTE[ train.index,]
test <- NOAA_SMOTE[-train.index,]
tomek = ubTomek(train[,-1], train[,1])
model_train_tomek = cbind(tomek$X,tomek$Y)
names(model_train_tomek)[1] = "japon"
removed.index = tomek$id.rm
train$japon = as.factor(train$japon)
train_tomek = train[-removed.index,]
## SMOTE after tomek links
traintomeksmote <- SMOTE(japon ~ ., train_tomek, perc.over = 2000,perc.under = 100)
fitControlSmoteTomek<- trainControl(## 10-fold CV
method = "repeatedcv",
number = 10,
repeats = 3,
## Estimate class probabilities
classProbs = TRUE,
## Evaluate performance using
## the following function
summaryFunction = twoClassSummary)
gbmGridSmoteTomek <- expand.grid(interaction.depth = c(3,4, 5, 6),
n.trees = (1:30)*50,
shrinkage = c(0.1,0.001,0.75,0.0001),
n.minobsinnode = 10)
gbmFitNOAASMOTETomek <- caret::train (make.names(japon) ~ ., data = traintomeksmote,
method = "gbm",
trControl = fitControlSmoteTomek,
distribution = "bernoulli",
verbose = FALSE,
tuneGrid = gbmGridSmoteTomek,
bag.fraction=0.5,
## Specify which metric to optimize
metric = "ROC")
test$japon = as.factor(test$japon)
PredNOAASMOTETomek <- predict(gbmFitNOAASMOTETomek, newdata= test ,type='prob')
cmSMOTETomekNOAA = confusionMatrix(PredNOAASMOTETomek , as.factor(test$japon), mode="everything")
part of the data
[enter image description here](https://i.stack.imgur.com/jPgI9.png)
For python3, I originally needed to extract odd and even positions from a list and assign it to new lists, then clear the original list. I thought lists were impacted by a function call through "pass by reference". Testing some scenarios, it works sometime. Could someone please explain how exactly python3 works here?
Case 1: empty list is populated with string as expected.
def func1(_in):
_in.append('abc')
mylist = list()
print(f"Before:\nmylist = {mylist}")
func1(mylist)
print(f"After:\nmylist = {mylist}")
Output case 1:
Before:
mylist = []
After:
mylist = ['abc']
Case 2: middle list element is replaced with string as expected.
def func2(_in):
_in[1] = 'abc'
mylist = list(range(3))
print(f"Before:\nmylist = {mylist}")
func2(mylist)
print(f"After:\nmylist = {mylist}")
Output case 2:
Before:
mylist = [0, 1, 2]
After:
mylist = [0, 'abc', 2]
Case 3: why is the list not empty after function call?
def func3(_in):
_in = list()
mylist = list(range(3))
print(f"Before:\nmylist = {mylist}")
func3(mylist)
print(f"After:\nmylist = {mylist}")
Output case 3:
Before:
mylist = [0, 1, 2]
After:
mylist = [0, 1, 2]
Case 4: working exactly as expected, but note I have returned all three lists from function.
def func4_with_ret(_src, _dest1, _dest2):
_dest1 = [val for val in _src[0:len(_src):2]]
_dest2 = [val for val in _src[1:len(_src):2]]
_src = list()
return _src, _dest1, _dest2
source = list(range(6))
evens, odds = list(), list()
print(f"Before function call:\nsource = {source}\nevens = {evens}\nodds = {odds}")
source, evens, odds = func4_with_ret(source, evens, odds)
print(f"\nAfter function call:\nsource = {source}\nevens = {evens}\nodds = {odds}")
Output case 4:
Before function call:
source = [0, 1, 2, 3, 4, 5]
evens = []
odds = []
After function call:
source = []
evens = [0, 2, 4]
odds = [1, 3, 5]
Case 5: why no impact on the variables outside the function if I do not explicitly return from function call?
def func5_no_ret(_src, _dest1, _dest2):
_dest1 = [val for val in _src[0:len(_src):2]]
_dest2 = [val for val in _src[1:len(_src):2]]
_src = list()
source = list(range(6))
evens, odds = list(), list()
print(f"Before function call:\nsource = {source}\nevens = {evens}\nodds = {odds}")
func5_no_ret(source, evens, odds)
print(f"\nAfter function call:\nsource = {source}\nevens = {evens}\nodds = {odds}")
Output case 5:
Before function call:
source = [0, 1, 2, 3, 4, 5]
evens = []
odds = []
After function call:
source = [0, 1, 2, 3, 4, 5]
evens = []
odds = []
Thank you.
Your ultimate problem is confusing (in-place) mutation with rebinding (also referred to somewhat less precisely as "reassignment").
In all the cases where the change isn't visible outside the function, you rebound the name inside the function. When you do:
name = val
it does not matter what used to be in name; it's rebound to val, and the reference to the old object is thrown away. When it's the last reference, this leads to the object being cleaned up; in your case, the argument used to alias an object also bound to a name in the caller, but after rebinding, that aliasing association is lost.
Aside for C/C++ folks: Rebinding is like assigning to a pointer variable, e.g. int *px = pfoo; (initial binding), followed later by px = pbar; (rebinding), where both pfoo and pbar are themselves pointers to int. When the px = pbar; assignment occurs, it doesn't matter that px used to point to the same thing as pfoo, it points to something new now, and following it up with *px = 1; (mutation, not rebinding) only affects whatever pbar points to, leaving the target of pfoo unchanged.
By contrast, mutation doesn't break aliasing associations, so:
name[1] = val
does rebind name[1] itself, but it doesn't rebind name; it continues to refer to the same object as before, it just mutates that object in place, leaving all aliasing intact (so all names aliasing the same object see the result of the change).
For your specific case, you could change the "broken" functions from rebinding to aliasing by changing to slice assignment/deletion or other forms of in-place mutation, e.g.:
def func3(_in):
# _in = list() BAD, rebinds
_in.clear() # Good, method mutates in place
del _in[:] # Good, equivalent to clear
_in[:] = list() # Acceptable; needlessly creates empty list, but closest to original
# code, and has same effect
def func5_no_ret(_src, _dest1, _dest2):
# BAD, all rebinding to new lists, not changing contents of original lists
#_dest1 = [val for val in _src[0:len(_src):2]]
#_dest2 = [val for val in _src[1:len(_src):2]]
#_src = list()
# Acceptable (you should just use multiple return values, not modify caller arguments)
# this isn't C where multiple returns are a PITA
_dest1[:] = _src[::2] # Removed slice components where defaults equivalent
_dest2[:] = _src[1::2] # and dropped pointless listcomp; if _src might not be a list
# list(_src[::2]) is still better than no-op listcomp
_src.clear()
# Best (though clearing _src is still weird)
retval = _src[::2], _src[1::2]
_src.clear()
return retval
# Perhaps overly clever to avoid named temporary:
try:
return _src[::2], _src[1::2]
finally:
_src.clear()
While modeling an optimisation problem using pyomo I noticed a weird behaviour when using a list of Var or Param: I always get the following error ValueError: Evaluating the numeric value of parameter 'SimpleParam' before the Param has been constructed (there is currently no value to return).
The following code (minise 4*x+1 such that x >= 0) runs exactly as expected:
import pyomo.environ as pyo
from pyomo.opt import SolverFactory
def _obj(model):
return model.c*model.x + 1
model = pyo.ConcreteModel()
model.x = pyo.Var(domain=pyo.NonNegativeReals)
model.c = pyo.Param(initialize=lambda model: 4, domain=pyo.NonNegativeReals)
model.obj = pyo.Objective(rule=_obj, sense=pyo.minimize)
opt = SolverFactory('glpk')
opt.solve(model)
but as I set model.x and model.c in lists, the program crashes when creating the objective function:
import pyomo.environ as pyo
from pyomo.opt import SolverFactory
def _obj(model):
return model.c[0]*model.x[0] + 1
model = pyo.ConcreteModel()
model.x = [pyo.Var(domain=pyo.NonNegativeReals)]
model.c = [pyo.Param(initialize=lambda model: 4, domain=pyo.NonNegativeReals)]
model.obj = pyo.Objective(rule=_obj, sense=pyo.minimize)
opt = SolverFactory('glpk')
opt.solve(model)
What is causing this error? Is this a desired behaviour for a reason that I don't understand or is this a bug? Anyway, how can I use lists of Params and Vars in a problem? I know that I can theoretically flatten all of my parameters and variables into a single IndexedVar or IndexedParam and handle the new indices myself, but that would be tedious since the range of the 3rd and 4th indices of my x and c depend on the 1st and 2nd index, therefore it would be a lot clearer in my code if I could use lists.
More precisely: I have a code looking like this (though I am still interested in knowning why the MWE above does not work):
# I, J are lists of indices and N is a list of integer values
model.Vs = [pyo.RangeSet(N[i]) for i in range(len(N))]
model.xs = [[pyo.Var(model.Vs[i], model.Vs[j]) for j in J] for i in I]
model.cs = [[pyo.Param(model.Vs[i], model.Vs[j]) for j in J] for i in I]
def _obj(model):
sum(model.xs[i][j][k,ell] * model.xs[i][j][k,ell] \\
for i in I for j in J \\
for k in model.Vs[i] for ell in model.Vs[j])
model.obj = Objective(rule=_obj, sense=pyo.minimize)
model.constraints = [
[pyo.Constraint(model.Vs[i], model.Vs[j], rule=...) for j in J]
for i in I
]
opt = SolverFactory('glpk')
opt.solve(model)
Your minimal example
import pyomo.environ as pyo
from pyomo.opt import SolverFactory
def _obj(model):
return model.c[0]*model.x[0] + 1
model = pyo.ConcreteModel()
model.x = [pyo.Var(domain=pyo.NonNegativeReals)]
model.c = [pyo.Param(initialize=lambda model: 4, domain=pyo.NonNegativeReals)]
model.obj = pyo.Objective(rule=_obj, sense=pyo.minimize)
opt = SolverFactory('glpk')
opt.solve(model)
generates the following error:
ValueError: Evaluating the numeric value of parameter 'SimpleParam' before
the Param has been constructed (there is currently no value to return).
The reason is because you are not directly attaching the Var and Param you are generating to the model. A lot happens when you attach a Pyomo modeling component to a Block (ConcreteModel objects are instances of constructed Blocks):
The component is assigned a name (that matches the Block attribute name)
The component is inserted into the hierarchy (basically, pointers are set so that methods can walk up and down the model hierarchy)
The component is categorized so that writers, solvers, and transformations can find it later
(If the Block has already been constructed), the component is automatically constructed.
By placing the component in a list, you are effectively "hiding" its existence from Pyomo. The first error you get has to do with this last bullet (the Param hasn't been constructed). However, simply constructing the Param and Var as you build the list will be insufficient, as the other actions won't take place and you will just hit a different error later (the next error would be an obsure one when the LP writer comes across a Var in the Objective that it had not found when it first walked the model hierarchy).
Perhaps this will help. I'm not sure I can answer why your example fails other than to say that pyomo is a modeling language that passes a structured math problem to a solver and the sets need to be discretely defined, not in lists of objects. Maybe somebody else can pitch in and explain it more clearly.
In your modeling, it appears you want to construct some kind of ragged set for x[i,j] where the range of j can vary based on i. You typically want to make sets for both I and J in order to support various constraint constructs. Then you can make a subset of "valid" (i, j) tuples for whatever model component needs to be indexed by this ragged set. You can either use this subset as the basis of iteration or use it to check membership if you are constructing things on-the-fly.
Here is an example using your list N:
import pyomo.environ as pyo
N = [1, 4, 3]
m = pyo.ConcreteModel()
m.I = pyo.Set(initialize=range(len(N)))
m.J = pyo.Set(initialize=range(max(N)))
m.IJ = pyo.Set(within=m.I * m.J, initialize =
[(i, j) for i in range(len(N)) for j in range(N[i])])
m.x = pyo.Var(m.IJ, domain=pyo.NonNegativeReals)
def _obj(model):
return sum(m.x[t] for t in m.IJ)
m.obj = pyo.Objective(rule=_obj)
def constrain_x2(model):
return sum(m.x[2, j] for j in m.J if (2, j) in m.IJ) >=1
m.c1 = pyo.Constraint(rule=constrain_x2)
m.pprint()
Yields:
4 Set Declarations
I : Dim=0, Dimen=1, Size=3, Domain=None, Ordered=False, Bounds=(0, 2)
[0, 1, 2]
IJ : Dim=0, Dimen=2, Size=8, Domain=IJ_domain, Ordered=False, Bounds=None
[(0, 0), (1, 0), (1, 1), (1, 2), (1, 3), (2, 0), (2, 1), (2, 2)]
IJ_domain : Dim=0, Dimen=2, Size=12, Domain=None, Ordered=False, Bounds=None
Virtual
J : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=False, Bounds=(0, 3)
[0, 1, 2, 3]
1 Var Declarations
x : Size=8, Index=IJ
Key : Lower : Value : Upper : Fixed : Stale : Domain
(0, 0) : 0 : None : None : False : True : NonNegativeReals
(1, 0) : 0 : None : None : False : True : NonNegativeReals
(1, 1) : 0 : None : None : False : True : NonNegativeReals
(1, 2) : 0 : None : None : False : True : NonNegativeReals
(1, 3) : 0 : None : None : False : True : NonNegativeReals
(2, 0) : 0 : None : None : False : True : NonNegativeReals
(2, 1) : 0 : None : None : False : True : NonNegativeReals
(2, 2) : 0 : None : None : False : True : NonNegativeReals
1 Objective Declarations
obj : Size=1, Index=None, Active=True
Key : Active : Sense : Expression
None : True : minimize : x[0,0] + x[1,0] + x[1,1] + x[1,2] + x[1,3] + x[2,0] + x[2,1] + x[2,2]
1 Constraint Declarations
c1 : Size=1, Index=None, Active=True
Key : Lower : Body : Upper : Active
None : 1.0 : x[2,0] + x[2,1] + x[2,2] : +Inf : True
7 Declarations: I J IJ_domain IJ x obj c1
I am trying to unpack a list with fewer variables and get the following error.
human_prots looks like this:
['9606.ENSP00000482075_HomoSapiens\t10029.XP_007627869.1_CricetulusGriseus,10036.XP_005079789.1_MesocricetusAuratus,10042.XP_006987441.1_PeromyscusManiculatus9986.ENSOCUP00000025621_OryctolagusCuniculus', '']
with open(path_to_homolog_file,'r') as ortho_tab:
ortho_text = ortho_tab.read()
no_genes = []
ortho_list = []
ortho_text.strip()
human_prots = ortho_text.split('\n\n')
for horthos in human_prots:
[hprot,orthos,_] = horthos.split('\t')
[_,hprot] = hprot.split('.')
hprot_info = find_match(db_dict.get('HomoSapiens'),hprot)
if len(hprot_info) > 1 and warning == 'on':
print('WARNING! Multiple matches found for {}: {}'.format(hprot,hprot_info))
[hgene,_] = hprot_info[0].split('\t')
orthos = orthos.split(', ')
line 71, in <module>
[hprot,orthos,*_] = horthos.split('\t')
ValueError: not enough values to unpack (expected at least 2, got 1)
what am I doing wrong?
I have a string related to a programs output, now I need to convert the string into a dictionary. I have tried it by using dict() and zip() commands but I am not able to fetch the results.
This is the code I have so far:
string = "Eth1/1 vlan-1 typemode-eth status:access eth1/2 vlan-1 type-eth status:access"
list1=string.split(' ')
print(list1)
['Eth1/1', 'vlan-1', 'typemode-access']
and further than this I have no idea:
{'eth1/1': {'Speed': '10Gb', 'Vlan': 1, 'Type Mode': 'eth', 'status': 'access'}, 'eth1/2': {'Speed': '10Gb', 'Vlan': 1, 'Type Mode': 'eth', 'status': 'access'}}
From your result to get a value see the following example. See inline comments.
import re
result = {}
string = "Eth1/1 vlan-1 typemode-eth status:access eth1/2 vlan-1 type-eth status:access"
a = re.search('access', string) # this gives 2 positions for the word access.
list1 = [string[0:a[0]], string[[a[0]+1]:]] # two substrings. a[0] is used to get
# roughly the middle of the string where the spplitpoint is of both
# substrings. Using access as key word gives flexibility if there is a third
# substring as well.
result = dict(list1) # result should be same as result2.
# y1 z1
result2 = {'eth1/1': {'Speed': '10Gb', 'Vlan': 1, 'Type Mode': 'eth', 'status': 'access'},
'eth1/2': {'Speed': '10Gb', 'Vlan': 1, 'Type Mode': 'eth', 'status': 'access'}}
# y2 = eth1/2.
# y1 y2
x = result['eth1/1']['Speed'] # replace any word at y1 or z1 to fetch another result.
print ('Got x : %s' % x) # this prints '10Gb'.
Basically what you've created is nested dictionaries. So addressing y1 first is enabling to get data from that particular dictionary. after y1 calling for z1 is getting the value from that particular key inside your first nested dictionary. If you change the keywords at x you get different different values back (regardless that it looks the same in your example; ttry with different values to see the result). Enjoy!
Try this code below:
string = "Eth1/1 vlan-1 typemode-eth status:access eth1/2 vlan-1 type-eth status:access eth1/3 vlan-1 type-eth status:access"
strList = string.split(" ")
indexPos = []
for data in range(0,len(strList)):
if strList[data].lower()[0:3] == 'eth':
print('Found',data)
indexPos.append(data)
dataDict = dict()
for i in range(0,len(indexPos)):
stringDict = dict()
stringDict['Speed'] = '10Gb'
if i is not len(indexPos)-1:
string = strList[indexPos[i]:indexPos[i+1]]
else:
string = strList[indexPos[i]:]
for i in range(0,len(string)):
if i is not 0:
if i is not 3:
valueSplit = string[i].split('-')
else:
print(i)
valueSplit = string[i].split(':')
stringDict[valueSplit[0]] = valueSplit[1]
dataDict[string[0]] = stringDict
I have written this code according the pattern in code. Please let me know if it work for you.