I am trying to replicate the hyperparameter tuning example reported at this link but I want to use scikit learn XGBoost instead of tensorflow in my training application.
I am able to run multiple trials in a single job, on for each of the hyperparameters combination. However, the Training output object returned by ML-Engine does not include the finalMetric field, reporting metric information (see the differences in the picture below).
What I get with the example of the link above:
Training output object with Tensorflow training app
What I get running my Training application with XGBoost:
Training output object with XGBoost training app
Is there a way for XGBoost to return training metrics to ML-Engine?
It seems that this process is automated for tensorflow, as specified in the documentation:
How Cloud ML Engine gets your metric
You may notice that there are no instructions in this documentation
for passing your hyperparameter metric to the Cloud ML Engine training
service. That's because the service monitors TensorFlow summary events
generated by your training application and retrieves the metric.
Is there a similar mechanism for XGBoost?
Now, I can always dump each metric results to a file at the end of each trial and then analyze them manually to select the best parameters. But, by doing so, am I loosing the automated mechanism offered by Cloud ML Engine, especially concerning the "ALGORITHM_UNSPECIFIED" hyperparameters search algorithm?
i.e.,
ALGORITHM_UNSPECIFIED: [...] applies Bayesian optimization to search
the space of possible hyperparameter values, resulting in the most
effective technique for your set of hyperparameters.
Hyperparameter tuning support of XGBoost was implemented in a different way. We created the cloudml-hypertune python package to help do it. We're still working on the public doc for it. At the meantime, you can refer to this staging sample to learn about how to use it.
Sara Robinson over at google put together a good post on how to do this. Rather than regurgitate and claim it as my own, I'll post this here for anyone else that comes across this post:
https://sararobinson.dev/2019/09/12/hyperparameter-tuning-xgboost.html
Related
I developed a designer to implement regression models in azure machine learning studio. I have taken the data set pill and then split the data set into train and test in prescribed manner. When I am trying to implement the evaluation metrics and run the pipeline, it was showing a warning and error in the moment I called the dataset for the operation. I am bit confused, with the same implementation, when i tried to run with linear regression and it worked as shown in the image. If the same approach is used to implement logistic regression it was showing some warning and error in building the evaluation metrics.
the above success is in linear regression. When it comes to logistic regression it was showing the warning and error in pipeline.
Any help is appreciated.
Creating a sample pipeline with designer with mathematical format.
We need to create a compute instance.
Assign the compute instance and click on create
Now the import data warning will be removed. In the same manner, we will be getting similar error in other pills too.
Create a mathematical format. If not needed for your case, try to remove that math operation and give the remaining.
Assign the column set. Select any option according to the requirement.
Finally, we can find the pills which have no warning or error.
I wonder how / if it is possible to run sklearn models / xgboost training for a large dataset.
If I use a dataframe that contains several giga-bytes, the machine crashes during the training.
Can you assist me please?
Scikit-learn documentation has an in-depth discussion about different strategies to scale models to bigger data.
Strategies include:
Streaming instances
Extracting features
Incremental learning (see also the partial_fit entry in the glossary)
I am using the object detector api for quite a while now so training models and use them for inference is all good. Unfortunately, when using TensorBoard to visualize metrics (such as mAP, AR, classification/localization loss) we only get to see those metrics on the validation set. I'd like to calculate the aforementioned metrics also during training so that we can compare train/validation metrics on Tensorboard.
edit: I've stumbled on this post which addresses the same concern how to check both training/eval performances in tensorflow object_detection
Anyone got a pointer on how to achieve this?
You can evaluate your model on the training data by adding the arguments --eval_training_data=True --sample_1_of_n_eval_on_train_examples=10 to the arguments of model_main.
By doing so, you instruct it to perform the evaluation on the training data, and you choose how much to dilute the training data sent to evaluation, since usually the amount of training data is very large.
The thing is that I don't think it's currently possible to evaluate both on training on validation data, but I don't think it's too bad, since usually evaluation on training data is only for sanity check, and not for actual continuous evaluation the model.
When defining the hyperparameter metric for Google Cloud ML I can use mean_squared_error, but should I be using val_mean_squared_error instead if I want it to be comparing the validation set accuracy? Or does it do it on its own?
This is the sample hptuning config:
trainingInput:
...
hyperparameters:
goal: MINIMIZE
hyperparameterMetricTag: ???mean_squared_error
And this is the fit invocation:
history = m.fit(train_x, train_y, epochs=epochs, batch_size=2048,
shuffle=False,
validation_data=(val_x, val_y),
verbose=verbose,
callbacks=callbacks)
Since I am passing my validation data and Keras, I am in doubt whether I should use val_mean_squared_error.
The answer is: if you (I) want Google Cloud ML hyperparameter tuning to use the VALIDATION metric instead of the training metric while using Keras, you need to specify val_mean_squared_error (or val_accuracy etc).
If you stick to accuracy or mean_squared_error you will bias Google Cloud ML tuning process to select overfitting models. To avoid the overfitting while searching for the parameters you should either create your own metric (as mentioned in a comment) or use the fit method with a validation set and us the val metrics.
I've updated the question to explicitly say I am using Keras, which automatically creates the val_mean_squared_error.
To get the answer I realized I could do a simple test: both with val_mean_squared_error and mean_squared_error while using Keras and invoking fit with the validation parameter set and compare the job results with the reported metrics.
I want to extract features using caffe and train those features using SVM. I have gone through this link: http://caffe.berkeleyvision.org/gathered/examples/feature_extraction.html. This links provides how we can extract features using caffenet. But I want to use Lenet architecture here. I am unable to change this line of command for Lenet:
./build/tools/extract_features.bin models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel examples/_temp/imagenet_val.prototxt fc7 examples/_temp/features 10 leveldb
And also, after extracting the features, how to train these features using SVM? I want to use python for this. For eg: If I get features from this code:
features = net.blobs['pool2'].data.copy()
Then, how can I train these features using SVM by defining my own classes?
You have two questions here:
Extracting features using LeNet
Training an SVM
Extracting features using LeNet
To extract the features from LeNet using the extract_features.bin script you need to have the model file (.caffemodel) and the model definition for testing (.prototxt).
The signature of extract_features.bin is here:
Usage: extract_features pretrained_net_param feature_extraction_proto_file extract_feature_blob_name1[,name2,...] save_feature_dataset_name1[,name2,...] num_mini_batches db_type [CPU/GPU] [DEVICE_ID=0]
So if you take as an example val prototxt file this one (https://github.com/BVLC/caffe/blob/master/models/bvlc_alexnet/train_val.prototxt), you can change it to the LeNet architecture and point it to your LMDB / LevelDB. That should get you most of the way there. Once you did that and get stuck, you can re-update your question or post a comment here so we can help.
Training SVM on top of features
I highly recommend using Python's scikit-learn for training an SVM from the features. It is super easy to get started, including reading in features saved from Caffe's format.
Very lagged reply, but should help.
Not 100% what you want, but I have used the VGG-16 net to extract face features using caffe and perform a accuracy test on a small subset of the LFW dataset. Exactly what you needed is in the code. The code creates classes for training and testing and pushes them into the SVM for classification.
https://github.com/wajihullahbaig/VGGFaceMatching