To download artifacts from a run, you need run id. I get the run id from the UI as shown below.
Run id from the UI
But when I set the run name parameter, run id is not visible in the UI. How to find the run Id of a particular run in MLflow ?
The run id in mlflow is a random stamp id generated. I had the same problem because I wrote an mlflow decorator, which needed access to the run id after the run was finished to set tags.
The question is what do you want to do after you have the run id? Then the approach would need extra infos.
If you only want to get access to your latest run:
Use the mlflow.list_run_infos() function and insert the experiment_id, which you can get by the mlflow.get_experiment_by_name function of mlflow. I guess you know your experiment's id. Here is the list_run_infos function
def list_run_infos(
self,
experiment_id: str,
run_view_type: int = ViewType.ACTIVE_ONLY,
max_results: int = SEARCH_MAX_RESULTS_DEFAULT,
order_by: Optional[List[str]] = None,
page_token: Optional[str] = None,
)
Then you should get a list of run objects. But, please read further:
In case you have multiple run objects in your experiment (that happens with several runs or even child runs from a parent run with Gridsearch and sklearn).
Loop through each! Runobject from the output of list__run_infos() and look at the end_time property of the Runobject. The endtime property is a UNIX timestamp. So even if you have a parent run or a single run, the highest UNIX timestamp in the end_time property will always be your last run (In case you didnt used several estimators in a loop in your experiment, that would require some refactoring). And by that you identify the appropriate RunObject.
Only then! you can access the Runtime Object's property: the run_id:
Here you can see the class of the run object from mlflow, keep in mind that you also need the exp_id again.
classmlflow.entities.RunInfo
(
run_uuid,
experiment_id,
user_id,
status,
start_time,
end_time,
lifecycle_stage,
artifact_uri=None,
run_id=None
)
In case you need the specific code:
last_parent_run = set()
exp_id = mlflow.get_experiment_by_name("your_exp_name"].experiment_id
for item in mlflow.list_run_infos(exp_id):
last_parent_run.add((item.__getattribute__("end_time"), item.__getattribute__("run_id")))
And then of course look for the maximum entry in your set
If you have any further questions just ask; i already tested my decorator with this and it works fine and keeps the main code of mlflow statements clean. Although its a little bit hacky, to access the run_id after the run.
I'm having issues with this Django project, when I try to run migrations got the following message:
django.core.exceptions.FieldDoesNotExist: InventoryImageHistory has no field named 'InventoryImageID'
this is the class for InventoryImageHistory
class InventoryImageHistory(models.Model):
ImageID = models.IntegerField(db_index=True, unique=True, null=False, primary_key=True)
history = ListField(DictField())
objects = models.DjongoManager()
these are the migration files:
migration file number 40:
class Migration(migrations.Migration):
operations = [
migrations.RenameField(
model_name='inventoryimagehistory',
old_name='InventoryImageID',
new_name='imageID',
),
De model definition is InventoryImageHistory, but for some reason, it keeps returning an error
Well, a few minutes after I posted my comment I seem to have fixed it. It was because I had initially put the RenameField in the previous (not yet committed to repo, but already applied) migration, and even though that one had been applied, Django seems to parse all of the dependency migrations logically and decided that the rename had already happened.
This message might have to do with the fact you faked/manipulated some migrations in that app recently.
What I do when this pops up is to:
Create another change in that models.py file (i.e.: add an extra field to the model).
Run the migration
Do the change I wanted to do in the first place that triggered the error and run the migration (hopefully successfully this time)
Undo the change I added on 1. and run migrations again.
I'm working on an existing Django 2.2 application comprising a custom app in conjunction with a Wagtail CMS, where I'm iteratively adding new wagtail page-types in separate user stories over time.
I want to be able to create a "master" or "index" migration that pre-builds each page-type in the database automatically when migrations are run (ours are performed in an Ansible task upon deployment). As far as I can tell, what I need requires:
The auto-built migration that modifies the DB schema for each page
A further migration that is always run last and which contains a dependencies attr - able to be updated with a single list-entry representing the new page's migration name, each time one is added.
I can already auto-build page-types using the following logic in a create() method called from migrations.RunPython() but at the moment, this same page-build logic needs to exist in each page's migration - I'd prefer it if this existed in a single migration (or an alternative procedure if one exists in DJango) that can always be run.
Ideally, the page_types list below could be replaced by just iterating over BasePage.__subclasses__(), (Where all page-types inherit from BasePage) meaning this "master" migration need never be altered again.
Note: if it helps any, the project is still in development, so any solution that is slightly controversial or strictly "dev-only" is acceptable - assuming it can be made acceptable and therefore less controversial by merging migrations later.
...
...
# Fetch the pre-created, root Page"
root_page = BasePage.objects.all().first()
page_types = [
ManageAccountPage,
EditUserDetailPage,
]
path_init = int('000100020003') # The last value for `path` from 0007_initialise_site_ttm.py
# Create, then add all child pages
for page_type in page_types:
title_raw = page_type.__name__.replace('Page', '')
page = page_type(
title=utils.convert_camel_to_human(title_raw),
slug=title_raw.lower(),
show_in_menus='t',
content_type=ContentType.objects.get_for_model(page_type),
path=path_init + 1,
depth=2
)
try:
root_page.add_child(instance=page)
except exceptions.ValidationError:
continue
...
...
What's the problem?
(See "What I've tried" below)
What I've tried:
A custom pin_curr_migration() method called from migrations.RunPython() that deletes the "master" migration's own record in django_migrations allowing it to be re-run. This however, results in errors where DJango complains about previously built pages already existing.
The Situation
I recently started working on a new project using nodejs. I have a background of using Python/Django and C#/.NET (not a huge fan of the latter). Node is awesome, but I must say I miss the ease of building models and automating migrations in Django. I am currently using the AdonisJS framework which leverages Knex. Knex is a powerful library, but the migrations all need to be manually built. Additionally, the AdonisJS ORM that manages the Models is independent of Knex (migration manager). You also do not define field attributes on the Models, which can have benifits for dynamically doing things in the front and back end. All things considered, there is a lot of room for human error, miscommunication and a boat load more typing required. I know the the hot thing these days is to keep it loose and fast, but for this specific project, I am looking for a bit more structure than loosely defined models.
Current State
What I have landed on is building a new Class called tableModel and a field class to define the fields within table model. I have already completed this and I am successfully writing the migration files leveraging mustache. I plan on also automatically writing the Models which I shouldn't have a problem with (fingers crossed).
The Problem
Here is where it gets a little tough and where I need help...I need to track what has been added or removed via migration so I can effectively write ups and downs as the tableModels change over time.
So let's say I add a "tableModel" which creates a migration to create table Foo with fields {id (bigint), user_id(int), name(string255)}
Later I want to add a field called description so I would simply add it to my "tableModel" and then run a build command which would build out the migration.
How do I check what has already been created though so I only do an up() for description?
Then I want to remove the name field so I mark it out in my "tableModel" and run a build migration command. How do I check what has been migrated that now needs to be added in to the down().
Edit: I would add a remove field to the up and the corresponding roll back to the down.
Bonus Round
Let's say I want to change user_id from an int to a bigint, because who makes a foreign key just an int? How do I check not just what needs to be added to the up and down, but also checks if I need to change a property on a field.
Edit: would just write the up. and a corresponding roll back to the down
The Big Question
Basically, how do I define dirty "tableModels" classes
Possible Solution?
I am thinking that maybe I should capture some type of registry or snapshot and then run the comparison when building the migrations and or models, then recapture/snapshot. If this is the route, should I store in a json file, write this to the DB itself, or is there another/better option.
If I create the tableModel instances as constants, could I actually write back to the JS file and capture the snapshot as an attribute? IF this is an option, is Node's file system the way to go and what's the best way to do this? Node keep suprising me so I wouldn't be baffled if any of these are an option.
Help!
If anyone has gone down this path before or knows of any tools I could leverage, I would greatly appreciate it and thank you in advance. Also, if I am headed in a completely wrong direction, then please let me know, I both handle and appreciate all types of feedback.
Example
Something to note, when I define the "tableModel" for a given migration or model, it is an instance of the class, I am not creating an extended class since this is not my orm.
class tableModel {
constructor(tableName, modelName = tableName, fields = []) {
this.tableName = tableName
this.modelName = modelName
this.fields = fields
}
// Bunch of other stuff
}
fooTableModel = new tableModel('fooTable', 'fooModel', fields = [
new tableField.stringField('title'),
new tableField.bigIntField('related_user_id'),
new tableField.textField('description','Testing Default',false,true)
]
)
which equates to:
tableModel {
tableName: 'fooTable',
modelName: 'fooModel',
fields:
[ stringField {
name: 'title',
type: 'string',
_unique: false,
allow_null: null,
fieldAttributes: {},
default_value: null },
bigIntField {
name: 'related_user_id',
type: 'bigInteger',
_unique: false,
allow_null: null,
fieldAttributes: {},
default_value: 0 },
textField {
name: 'description',
type: 'text',
_unique: false,
allow_null: true,
fieldAttributes: {},
default_value: 'Testing Default' } ]
You have the up and down notation mixed up. Those are for migrating the "latest" (runs the up function) and doing rollbacks (runs the down function). Up and down to not relate to dropping or adding table columns.
The migrations up is for any change, and the down is to reverse those changes. So if you wanted to drop a column from some table, you write the command in the up, then write the opposite in the down (you'd add it back in...), such that you can "rollback" and the change is effectively reversed. You have to be careful with such things though, as you can put yourself in a situation where you actually lose data.
Want to add a column? Write it in the up, and drop the column in the down.
One of the major points behind the migrations mechanism is to track the state of changes of your database, as time goes forward. So generally, if you created a table in some migration, then a day or so later you realize you need to drop/add columns, you normally don't go back and edit the existing migration, especially if the migration has already been run. You'd just write a new migration to drop/add your column.
Since you're using knex, there are a couple "knex" tables that get created. By default the one you're looking for is knex_migrations, unless someone specifically modified the settings to change the name of it. This table holds all the migrations that have run against your DB, per batch. From the CLI, assuming you have knex.js installed globally, you can run knex migrate:latest, and that will push all the migrations that exist in your directory to the target database, if they have not yet been run. It does this by way of examining that knex_migrations table. If you roll a change and don't like it, and assuming you've properly done the down function, you can invoke knex migrate:rollback to reverse the change. If there are 3 migration files that have NOT yet been run, invoking knex migrate:latest will run all 3 of those migration files under a new batch #, which is 1 higher than the most recent batch number. Conversely, if you invoke a knex migrate:rollback, it will find the highest batch number (there could be more than 1 migration in a batch...), and invoke the down function on all those files, effectively rollback those changes.
All that said, knex is a "query builder" tool. It's got a ton of helper functions to help build the sql for you. Personally, I find this to be a major distraction. Why spend hours on hours figuring out all the helper functions when I can just go crank out raw SQL and run that. Thus, that's what we've done in our system. we use knex.raw('') and write our own DDL and DML. It works great and does exactly what we need it to. We don't need to go figure out the magic of the query building.
The short answer is that knex will automatically know what has and has not been run for you (again, via that knex_migrations table it creates for you...).
Things can get weird though when it start involving git and different branches. I recommend that if you're writing migrations on some branch, and you need to go do other work, always remember to first perform a rollback of any migrations you've done in that branch BEFORE switching branches. Otherwise you will be in weird DB states that don't coincide with the application code.
I would personally just deal with updating models independently of writing migrations. For example, if I'm adding a description column to some table, then I probably want to manually update the ORM to reflect the change of the new db schema. Generally, I've found trying to use a tool that automagically does that for you (rather, if I change the orm, stuff happens to write all the underlying sql...) usually winds me up in a heap of trouble and I just spend more time trying to un-fudge stuff. But, that's just my 2 cents :)
Here is where it gets a little tough and where I need help...I need to track what has been added or removed via migration so I can effectively write ups and downs as the tableModels change over time.
You could store changes in a DB/txt file and those can act as snapshots. So when you want to rollback to a particular migration, you would find the changes (up/down) made for that mutation and adjust accordingly.
Later I want to add a field called description so I would simply add it to my "tableModel" and then run a build command which would build out the migration. How do I check what has already been created though so I only do an up() for description?
Here you either call the database itself directly and check what fields have already been created. If a field is already their and the attributes are the same, you can either ignore it or stop the transaction all together.
Bonus Round Let's say I want to change user_id from an int to a bigint, because who makes a foreign key just an int? How do I check not just what needs to be added to the up and down, but also checks if I need to change a property on a field.
Again, call the DB itself on the table in question. I know the SQL call would be:
describe [table_name];
After reading the end, I think you answered this yourself, but I think capturing these changes would work best in a NoSql database since you're using Node or PostGres with it's json field.
I am new in BPEL writing. I have realized the simple process below:
receive1
|
|
invoke1
|
|
receive2
|
|
invoke2
The problem is that the process correctly runs till to the "receive2" but when I invoke, via soapUI, the operation associated to the "receive2" nothing happens. I have read other posts about BPEL but nothing matching with this question. Below the real activities (I omitted the Assign ones) involved.
<bpel:receive name="receiveInput" partnerLink="client"
portType="tns:HealthMobility"
operation="initiate" variable="input"
createInstance="yes"/>
<bpel:invoke name="getTreatmentOptions"
partnerLink="treatmentProviderPL" operation="getTreatmentOptions"
inputVariable="getTreatmentOptionsReq" outputVariable="getTreatmentOptionsResp">
</bpel:invoke>
<bpel:receive name="bookMobility" partnerLink="client" operation="bookMobility"
variable="bookMobilityReq" portType="tns:HealthMobility"/>
<bpel:invoke name="getTripOptions" partnerLink="mobilityMultiProvidersPL"
operation="getTripOptions" inputVariable="getTripOptionsReq"
outputVariable="getTripOptionsResp"></bpel:invoke>
I have tried to make debugging simply by deleting the receive and initializing statically the input variable required by the getTriOptions invoke. In this case all works fine so it means, necessarly, that the process continue to wait on the receive also if I invoke bookMobility via SOAPUI. My question is: Why? I'm missing something?
Thanks
You need to define a correlation set for the second receive. Each message that is sent to the operation that is connected to the first receive activity will create a new process instance. This means you may have multiple instance running in parallel. When these instances have reached the second receive, they are waiting for the second message, but in your example, there are no means to distinguish, which message is targeted to which process instance. I assume that your BPEL engine also logged that it could not route the message to a target instance.
To solve this problem, you need to find an identifier in the payload of a message and initialize a correlation set with this value. Then, when using the same correlation set with the second receive, all messages that contain the same identifier will be routed to this particular process instance. For further information about correlation sets I recommend reading the BPEL primer, section 4.2.4.