My requirement is to create separate Mlflow custom models for training and prediction.
I want to create training model and use those training model in prediction model
You can use mlflow.pyfunc to create a custom model.
You can find additional details of mlflow.pyfunc on this link.
This link will also help you get a overall view on models
Related
I trained an Bert-based NER model using Pytorch framework by referring the below article.
https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/.
After training the model using this approach, I saved that model using torch.save() method. Now, I want to retrain the model with new dataset.
Can someone please help me on how to perform retraining/transfer learning as I'm new to NLP and transformers.
Thanks in advance.
First, you can read the doc on pytorch loading, the model that can be very helpful to retrain the save model in a new dataset. I will provide the example of loading a save model. That can be very helpful to you
Original Doc :- https://pytorch.org/tutorials/beginner/saving_loading_models.html
Example code :- https://pythonguides.com/pytorch-load-model/
This two link are very helpful to train the new dataset on save model
I would like to be able to register untrained models with MLFLow to use as prototypes for instantiating models for training. I need this because we need to train thousands of models of the same type. Is this possible?
You could do it by logging the untrained model as an artifact.
I was exploring turicreate ObjectDetector API. The documentation mentions it is a trained model. I wanted to know if I can use this trained model & detect the 1000 labels which was used to originally train this turi model. All the examples mention to train with our dataset, I do not want to train but wanted to use pre-trained model which can classify. Any help is appreciated.
Is your question about how to load and use a pre-trained model? Turi create API docs mentions a load_model method:
model.save('my_model_file')
loaded_model = tc.load_model('my_model_file')
EDIT: Yep, ObjectDetector exposes a save method that works well with load_model.
I am performing incremental training on a model I already trained in SageMaker. I want to add data to the existing classes as well as create new classes. The first model had 4 classes (num_classes = 4) but I want to keep those classes as well as add 3 additional classes.
The documentation says that the num_classes hyperparameter must be the same when doing incremental training. But if that is the case, that means I cannot add classes to my existing model, I would have to start from scratch each time I wanted to change the number of classes. Is this accurate? Or is there a way to update an existing model and change the number of classes it is trained on?
Here is the example notebook I am using for the incremental training job:
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-incremental-training-highlevel.ipynb
See num_classes parameter here
For incremental training with the SageMaker built-in Image Classification algorithm, the input hyperparameters of both the new model and the pre-trained model must have the same settings for the num_layers, image_shape and num_classes input parameters. Because these parameters define the network architecture.
Typically, for newly added classes you would create a new dataset with old and new classes and update the model weights on the new dataset. As far as I know, this is not possible with SageMaker Image Classification algorithm.
Refer to this to train a GAN model for MNIST dataset, I want to save a model and restore it for further prediction. After having some understanding of Saving and Importing a Tensorflow Model I am able to save and restore some variables of inputs and outputs but for this network I am able to save the model only after some specific iterations and not able to predict some output.
Did you refer to this guide? It explains very clearly how to load and save tensorflow models in all possible formats.
If you are new to ML, I'd recommend you give Keras a try first, which is much easier to use. See https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model, pretty much you can use:
model.save('my_model.h5')
to save your model to disk.
model = load_model('my_model.h5')
to load your model and make prediction