pyomo: Connector for blocks not working - python-3.x

I'm trying to connect two blocks with the "Connector" class implemented in pyomo, using the following simple examplary code.
from pyomo.environ import *
m = ConcreteModel()
# Block 01
m.block_01 = Block()
m.block_01.flow = Var(within=NonNegativeReals, bounds=(2, 10))
m.block_01.OUT = Connector(initialize= {'flow': m.block_01.flow})
# Block 02
m.block_02 = Block()
m.block_02.flow = Var(within=NonNegativeReals)
m.block_02.IN = Connector(initialize= {'flow': m.block_02.flow})
m.con = Constraint(expr=m.block_01.OUT == m.block_02.IN)
def _obj(_m):
return _m.block_01.flow + _m.block_02.flow
m.obj = Objective(rule=_obj)
After "optimization" all variables take their lower bound values (m.block_01.flow = 2 and m.block_02.flow = 0). So the Connector seems not to transfer any data for the variables.
If I'm using:
m.con = Constraint(expr=m.block_01.flow == m.block_02.flow)
instead, it works. However this is not the idea of Connectors, right?
Any ideas about the reason for the problem?

Did you apply the expand_connectors transformation before sending your model to a solver?
TransformationFactory('core.expand_connectors').apply_to(m)

Related

python function with modular variable

hie,
I'm writing my first big python program (3.8) and I try to use a function for several uses (same work but with different targets from existing attributes)
I hope it's clear enough.
here the wanted Job :
it's inside a QT5 GUI (QApplication)
class GuiSuperQuizz(QWidget, QApplication):
...
...
def ajout_pts_blindtest(self, nbr):
x = nbr
x = str(x)
eval("team" + x).ajou_pts(int(self.point_blindtest))
eval("self.score_equip_" + x).setText(str(eval("team" + x).point)) # bug is here
eval("self.gest_score_equip_" + x).setText(str(eval("team" + x).point))
print(eval("team" + x).point)
self.continu[0] = False
self.en_pause[0] = False
self.records_scores()
The interpreter do not recognize the attribute "score_equip_1" and give me an error
AttributeError: 'GuiSuperQuizz' object has no attribute 'score_equip_1'
Yet, I know that attribute works well with this other function that work fine
def ajout_pts_rap_team1(self):
team1.ajou_pts(int(self.point_rap))
self.score_equip_1.setText(str(team1.point))
self.gest_score_equip_1.setText(str(team1.point))
print(team1.point)
self.continu[0] = False
self.en_pause[0] = False
self.aff_ligne4()
self.records_scores()
For not writing 4 functions to target 4 variables that are just incremented (it's a Quizz game with 4 players) I try try to concatenate in 1 function that arrange targets.
if I test the same logic on a very simple lines that works:
test1 = 456
def test(nbr):
x = nbr
x=str(x)
print(eval("test"+x))
test(1)
456
If anyone got some explanations ....

Issue producing a valid WIF key with Python 3.6 (Pythonista - iPadOS)

I am having some issues with some code. I have set about a project for creating Bitcoin wallets in an attempt to turn a hobby into a learning experience, whereby I can understand both Python and the Bitcoin protocol in more detail. I have posted here rather than in the Bitcoin site as the question is related to Python programming.
Below I have some code which I have created to turn a private key into a WIF key. I have written this out for clarity rather than the most optimal method of coding, so that I can see all the steps clearly and work on issues. This code was previously a series lines which I have now progressed into a class with functions.
I am following the example from this page: https://en.bitcoin.it/wiki/Wallet_import_format
Here is my current code:
import hashlib
import codecs
class wif():
def private_to_wif(private_key):
extended_key = wif.create_extended(private_key)
address = wif.create_wif_address(extended_key)
return address
def create_extended(private_key):
private_key1 = bytes.fromhex(private_key)
private_key2 = codecs.encode(private_key1, 'hex')
mainnet = b'80'
#testnet = b'ef'
#compressed = b'01'
extended_key = mainnet + private_key2
return extended_key
def create_wif_address(extended_key):
first_hash = hashlib.sha256(extended_key)
first_digest = first_hash.digest()
second_hash = hashlib.sha256(first_digest)
second_digest = second_hash.digest()
second_digest_hex = codecs.encode(second_digest, 'hex')
checksum = second_digest_hex[:8]
extended_key_chksm = (extended_key + checksum).decode('utf-8')
wif_address = base58(extended_key_chksm)
return wif_address
def base58(extended_key_chksm):
alphabet = '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz'
b58_string = ''
leading_zeros = len(extended_key_chksm) - len(extended_key_chksm.lstrip('0'))
address_int = int(extended_key_chksm, 16)
while address_int > 0:
digit = address_int % 58
digit_char = alphabet[digit]
b58_string = digit_char + b58_string
address_int //= 58
ones = leading_zeros // 2
for one in range(ones):
b58_string = '1' + b58_string
return b58_string
I then use a few lines of code to get this working, using the example private key from the above guide, as follows:
key = ‘0C28FCA386C7A227600B2FE50B7CAE11EC86D3BF1FBE471BE89827E19D72AA1D‘
address = wif.private_to_wif(key)
Print(address)
I should be getting the output: 5HueCGU8rMjxEXxiPuD5BDku4MkFqeZyd4dZ1jvhTVqvbTLvyTJ
Instead I’m getting:
5HueCGU8rMjxEXxiPuD5BDku4MkFqeZyd4dZ1jvhTVqvbWs6eYX
It’s only the last 6 characters that differ!
Any suggestions and help would be greatly appreciated.
Thank you in advance.
Connor
I have managed to find the solution by adding a missing encoding step.
I am posting for all those who run into a similar issue and can see the steps that brought resolution.
def create_wif_address(extended_key):
extended_key_dec = codecs.decode(extended_key, 'hex')
first_hash = hashlib.sha256(extended_key_dec)
first_digest = first_hash.digest()
second_hash = hashlib.sha256(first_digest)
second_digest = second_hash.digest()
second_digest_hex = codecs.encode(second_digest, 'hex')
checksum = second_digest_hex[:8]
extended_key_chksm = (extended_key + checksum).decode('utf-8')
wif_address = base58(extended_key_chksm)
return wif_address
So above I added in a step to the function, at the beginning, to decode the passed in variable from a hexadecimal byte string to raw bytes, which is what the hashing algorithms require it seems, and this produced the result I was hoping to achieve.
Connor

Running MPI python script in MPI azure ml pipeline

I'm trying to run distributed python job through azure ML pipelines using MPIStep pipeline class, by referring to the below example link - https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/pipeline-style-transfer/pipeline-style-transfer.ipynb
I tried implemented the same but even I change the node count parameter in MpiStep class, while running the script the it shows size (i.e comm.Get_size()) as 1 always. Can you please help me in what I'm missing here. Is there any specific setup required on the cluster?
Code snippets:
Pipeline code snippet:
model_dir = model_ds.path('./'+saved_model_blob+'/',data_reference_name='saved_model_path').as_mount()
label_dir = model_ds.path('./'+model_label_blob+'/',data_reference_name='model_label_blob').as_mount()
input_images = result_ds.path('./'+score_blob_name+'/',data_reference_name='Input_images').as_mount()
output_container = 'abc'
inti_container = 'xyz'
distributed_batch_score_step = MpiStep(
name="batch_scoring",
source_directory=SCRIPT_FOLDER,
script_name="batch_scoring_script_mpi.py",
arguments=["--dataset_path", input_images,
"--model_name", model_dir,
"--label_dir", label_dir,
"--intermediate_data_container", inti_container,
"--output_container", output_container],
compute_target=gpu_cluster,
inputs=[input_images, model_dir,label_dir],
pip_packages=["tensorflow","tensorflow-gpu==1.13.1","pillow","azure-keyvault","azure-storage-blob"],
conda_packages=["mesa-libgl-cos6-x86_64","mpi4py==3.0.2","opencv=3.4.2","scikit-learn=0.21.2"],
use_gpu=True,
allow_reuse = False,
node_count = nodecount_param,
process_count_per_node = 1
)
Python Script code snippet:
def run(input_dataset,comm):
rank = comm.Get_rank()
size = comm.Get_size()
print("Rank:" , rank)
print("Size:", size) # shows always 1, even the input node count is >1
print(MPI.Get_processor_name())
file_names = get_file_names(args.dataset_path)
sorted(file_names)
partition_size = len(file_names) // size
print("partition_size-->",partition_size)
partitioned_filenames = file_names[rank * partition_size: (rank + 1) * partition_size]
print("RANK {} - is processing {} images out of the total {}".format(rank, len(partitioned_filenames),
len(file_names)))
# call to Function 01
# call to Function 02
img_names = score_df['image_name'].unique()
output_batch = pd.DataFrame()
for i in img_names:
# call to Function 3
output_batch = output_batch.append(pp_output, ignore_index=True)
output_paths_list = comm.gather(output_batch, root=0)
print("RANK {} - number of pre-aggregated output files {}".format(rank, len(output_batch)))
print("saved in", currentDT + '\\' + 'data.csv')
if rank == 0:
print("RANK {} - number of aggregated output files {}".format(rank, len(output_paths_list)))
print("RANK {} - end".format(rank))
if __name__ == "__main__":
with tf.device('/GPU:0'):
init()
comm = MPI.COMM_WORLD
run(args.dataset_path,comm)
Got to know the issue is due to package version, earlier it is installed via conda with conda_packages=["mpi4py==3.0.2"], it worked after changing the install through pip - pip_packages=["mpi4py"]

Using multiple self-defined metrics in LightGBM

Given that we could use self-defined metric in LightGBM and use parameter 'feval' to call it during training.
And for given metric, we could define it in the parameter dict like metric:(l1, l2)
My question is that how call several self-defined metric at the same time? I cannot use feval=(my_metric1, my_metric2) to get the result
params = {}
params['learning_rate'] = 0.003
params['boosting_type'] = 'goss'
params['objective'] = 'multiclassova'
params['metric'] = ['multi_error', 'multi_logloss']
params['sub_feature'] = 0.8
params['num_leaves'] = 15
params['min_data'] = 600
params['tree_learner'] = 'voting'
params['bagging_freq'] = 3
params['num_class'] = 3
params['max_depth'] = -1
params['max_bin'] = 512
params['verbose'] = -1
params['is_unbalance'] = True
evals_result = {}
aa = lgb.train(params,
d_train,
valid_sets=[d_train, d_dev],
evals_result=evals_result,
num_boost_round=4500,
feature_name=f_names,
verbose_eval=10,
categorical_feature = f_names,
learning_rates=lambda iter: (1 / (1 + decay_rate * iter)) * params['learning_rate'])
Lets' discuss on the code I share here. d_train is my training set. d_dev is my validation set (I have a different test set.) evals_result will record our multi_error and multi_logloss per iteration as a list. verbose_eval = 10 will make LightGBM print multi_error and multi_logloss of both training set and validation set at every 10 iterations. If you want to plot multi_error and multi_logloss as a graph:
lgb.plot_metric(evals_result, metric='multi_error')
plt.show()
lgb.plot_metric(evals_result, metric='multi_logloss')
plt.show()
You can find other useful functions from LightGBM documentation. If you can't find what you need, go to XGBoost documentation, a simple trick. If there is something missing, please do not hesitate to ask more.

Assign Class attributes from list elements

I'm not sure if the title accurately describes what I'm trying to do. I have a Python3.x script that I wrote that will issue flood warning to my facebook page when the river near my home has reached it's lowest flood stage. Right now the script works, however it only reports data from one measuring station. I would like to be able to process the data from all of the stations in my county (total of 5), so I was thinking that maybe a class method may do the trick but I'm not sure how to implement it. I've been teaching myself Python since January and feel pretty comfortable with the language for the most part, and while I have a good idea of how to build a class object I'm not sure how my flow chart should look. Here is the code now:
#!/usr/bin/env python3
'''
Facebook Flood Warning Alert System - this script will post a notification to
to Facebook whenever the Sabine River # Hawkins reaches flood stage (22.3')
'''
import requests
import facebook
from lxml import html
graph = facebook.GraphAPI(access_token='My_Access_Token')
river_url = 'http://water.weather.gov/ahps2/river.php?wfo=SHV&wfoid=18715&riverid=203413&pt%5B%5D=147710&allpoints=143204%2C147710%2C141425%2C144668%2C141750%2C141658%2C141942%2C143491%2C144810%2C143165%2C145368&data%5B%5D=obs'
ref_url = 'http://water.weather.gov/ahps2/river.php?wfo=SHV&wfoid=18715&riverid=203413&pt%5B%5D=147710&allpoints=143204%2C147710%2C141425%2C144668%2C141750%2C141658%2C141942%2C143491%2C144810%2C143165%2C145368&data%5B%5D=all'
def checkflood():
r = requests.get(river_url)
tree = html.fromstring(r.content)
stage = ''.join(tree.xpath('//div[#class="stage_stage_flow"]//text()'))
warn = ''.join(tree.xpath('//div[#class="current_warns_statmnts_ads"]/text()'))
stage_l = stage.split()
level = float(stage_l[2])
#check if we're at flood level
if level < 22.5:
pass
elif level == 37:
major_diff = level - 23.0
major_r = ('The Sabine River near Hawkins, Tx has reached [Major Flood Stage]: #', stage_l[2], 'Ft. ', str(round(major_diff, 2)), ' Ft. \n Please click the link for more information.\n\n Current Warnings and Alerts:\n ', warn)
major_p = ''.join(major_r)
graph.put_object(parent_object='me', connection_name='feed', message = major_p, link = ref_url)
<--snip-->
checkflood()
Each station has different 5 different catagories for flood stage: Action, Flood, Moderate, Major, each different depths per station. So for Sabine river in Hawkins it will be Action - 22', Flood - 24', Moderate - 28', Major - 32'. For the other statinos those depths are different. So I know that I'll have to start out with something like:
class River:
def __init__(self, id, stage):
self.id = id #station ID
self.stage = stage #river level'
#staticmethod
def check_flood(stage):
if stage < 22.5:
pass
elif stage.....
but from there I'm not sure what to do. Where should it be added in(to?) the code, should I write a class to handle the Facebook postings as well, is this even something that needs a class method to handle, is there any way to clean this up for efficiency? I'm not looking for anyone to write this up for me, but some tips and pointers would sure be helpful. Thanks everyone!
EDIT Here is what I figured out and is working:
class River:
name = ""
stage = ""
action = ""
flood = ""
mod = ""
major = ""
warn = ""
def checkflood(self):
if float(self.stage) < float(self.action):
pass
elif float(self.stage) >= float(self.major):
<--snip-->
mineola = River()
mineola.name = stations[0]
mineola.stage = stages[0]
mineola.action = "13.5"
mineola.flood = "14.0"
mineola.mod = "18.0"
mineola.major = "21.0"
mineola.alert = warn[0]
hawkins = River()
hawkins.name = stations[1]
hawkins.stage = stages[1]
hawkins.action = "22.5"
hawkins.flood = "23.0"
hawkins.mod = "32.0"
hawkins.major = "37.0"
hawkins.alert = warn[1]
<--snip-->
So from here I'm tring to stick all the individual river blocks into one block. What I have tried so far is this:
class River:
... name = ""
... stage = ""
... def testcheck(self):
... return self.name, self.stage
...
>>> for n in range(num_river):
... stations[n] = River()
... stations[n].name = stations[n]
... stations[n].stage = stages[n]
...
>>> for n in range(num_river):
... stations[n].testcheck()
...
<__main__.River object at 0x7fbea469bc50> 4.13
<__main__.River object at 0x7fbea46b4748> 20.76
<__main__.River object at 0x7fbea46b4320> 22.13
<__main__.River object at 0x7fbea46b4898> 16.08
So this doesn't give me the printed results that I was expecting. How can I return the string instead of the object? Will I be able to define the Class variables in this manner or will I have to list them out individually? Thanks again!
After reading many, many, many articles and tutorials on class objects I was able to come up with a solution for creating the objects using list elements.
class River():
def __init__(self, river, stage, flood, action):
self.river = river
self.stage = stage
self.action = action
self.flood = flood
self.action = action
def alerts(self):
if float(self.stage < self.flood):
#alert = "The %s is below Flood Stage (%sFt) # %s Ft. \n" % (self.river, self.flood, self.stage)
pass
elif float(self.stage > self.flood):
alert = "The %s has reached Flood Stage(%sFt) # %sFt. Warnings: %s \n" % (self.river, self.flood, self.stage, self.action)
return alert
'''this is the function that I was trying to create
to build the class objects automagically'''
def riverlist():
river_list = []
for n in range(len(rivers)):
station = River(river[n], stages[n], floods[n], warns[n])
river_list.append(station)
return river_list
if __name__ == '__main__':
for x in riverlist():
print(x.alerts())

Resources