I am trying to load the FastText and save that as a model so that I can deploy that on production as the file size is 1.2 gb and wont be a good practice to use that on Prod.
Can anyone suggest an approach to save and load the model for production ("fasttext-wiki-news-subwords-300")
Loading the file using gensim.downloader api
You can use the library https://github.com/avidale/compress-fasttext, which is a wrapper around Gensim that can serve compressed versions of unsupervised FastText models.
The compressed versions can be orders of magnitude smaller (e.g. 20mb), with a tolerable loss in quality.
In order to have clarity over exactly what you're getting, in what format, I strongly recommend downloading things like sets of pretrained vectors from their original sources rather than the Gensim gensim.downloader convenience methods. (That API also, against most users' expectations & best packaging hygeine, will download & run arbitrary other code that's not part of Gensim's version-controlled source repository or its official PyPI package. See project issue #2283.)
For example, you could grab the raw vectors files direct from: https://fasttext.cc/docs/en/english-vectors.html
The tool from ~david-dale's answer looks interesting, for its radical compression, and if you can verify the compressed versions still work well for your purposes, it may be an ideal approach for memory-limited production deployments.
I would also consider:
A production machine with enough GB of RAM to load the full model may not be too costly, and with these sorts of vector-models, typical access patterns mean you essentially always want the full model in RAM, with no virtual-memory swapping at all. If your deployment is in a web server, there are some memory-mapping tricks possible that can help many processes share the same singly-loaded copy of the model (to avoid time- and memory-consumptive redundant reloads). See this answer for an approach that works with Word2Vec (though that may need some adaptation for FastText & recent Gensim versions).
If you don't need the Fasttext-specific subword-based synthesis, you can save the full-word vectors to a file in a simple format, then choose to only reload any small subset of the leading vectors (most common words) using the limit option of load_word2vec_format(). For exmaple:
# save only the word-vectors from a FastText model
ft_model.wv.save_word2vec_format('wvonly.txt', binary=False)
# ... then, later/elsewhere:
# load only 1st 50,000 word-vectors
wordvecs = KeyedVectors.load_word2vec_format('wvonly.txt', binary=False, limit=50000)
Related
I want to hand write a framework to perform inference of a given neural network. The network is so complicated, so to make sure my implementation is correct, I need to know how exactly the inference process is done on device.
I tried to use torchviz to visualize the network, but what I got seems to be the back propagation compute graph, which is really hard to understand.
Then I tried to convert the pytorch model to ONNX format, following the instruction enter link description here, but when I tried to visualize it, it seems that the original layers of the model had been seperated into very small operators.
I just want to get the result like this
How can I get this? Thanks!
Have you tried saving the model with torch.save (https://pytorch.org/tutorials/beginner/saving_loading_models.html) and opening it with Netron? The last view you showed is a view of the Netron app.
You can try also the package torchview, which provides several features (useful especially for large models). For instance you can set the display depth (depth in nested hierarchy of moduls).
It is also based on forward prop
github repo
Disclaimer: I am the author of the package
Note: The accepted format for tool is pytorch model
I'm using Stanza to get tokens, lemmas and tags from documents in multiple languages for the purposes of a language learning app. This means that I need to store and load many Stanza (default) models for different languages.
My main problem right now is that if I want to load all those models the memory requirement is too much for my resources. I currently deploy a web API running Stanza NLP on AWS. I want to keep my infrastructure costs at a minimum.
One possible solution is to load one model at a time when I need to run my script. I guess that means there will be some extra overhead each time in order to load the model in memory.
Another thing I tried is just to use the processors that I really need which decreases the memory footprint but not by that much.
I tried looking at open and closed issues on Github and Google but didn't find much.
What other possible solutions are out there?
The bottom line is a model for a language has to be in memory during execution, so by some means or another you need to make the model smaller or tolerate storing models on disk. I can offer some suggestions to make the models smaller, though be warned that making your model smaller will probably result in poorer accuracy.
You could examine the percentage breakdown of language requests, and store commonly requested languages in memory and only go to disk for rarer language requests.
The most immediate impact strategy for reducing model size is to shrink the vocabulary size. It is possible you could cut the vocabulary even smaller and still get similar accuracy. We have done some optimization on this front, but there may be more opportunity to cut model size.
You could experiment with smaller model size and word embeddings and may only get a small accuracy drop, we haven't really aggressively experimented with different model sizes to see how much accuracy you lose. This would mean retraining the model and just setting the embedding size and model size parameters smaller.
I don't know a lot about this, but there is a strategy of tagging a bunch of data with your big accurate model, and then training a smaller model to mimic the big model. I believe this is called "knowledge distillation".
In a similar direction, you could tag a bunch of data with Stanza, and then train a CoreNLP model (which I think would have a smaller memory footprint).
In summary, I think the easiest thing to do would be to retrain a model with a smaller vocabulary size. We I think it currently has 250,000 words, and cutting to 10,000 or 50,000 will reduce model size, but may not affect accuracy too badly.
Unfortunately I don't think there is a magical option you can select that will just solve this issue, you will have to retrain models and see what kind of accuracy you are willing to sacrifice for a lower memory footprint.
I have built and trained neural network in Pytorch and is ready for production to a website but how do I deploy it?
There is multiple ways to do it.
Yet first, I add a PS : I noticed that you were asking specifically about reinforcement learning after having posted my answer. Know that even though I have written this answer with a static neural network model in mind, I offer at the end of the post a solution to apply the ideas of this answer to reinforcement learning.
The different options :
From what I know, using PyTorch in production is not especially recommended for big scale production.
It is more common to convert the PyTorch model to the ONNX format (a format to make ai models interchangeable between frameworks). Here is a tutorial if you want to operate this way : https://github.com/onnx/tutorials/blob/master/tutorials/PytorchOnnxExport.ipynb .
Then run it using the ONNX runtime, Caffe2 (by Facebook) or with TensorFlow (by Google).
My answer is not going to explore those solutions (and i did not include tutorials to those options), because i recently did the same as you are trying to do (building a neural network architecture and wanting to deploy it, and also allowing users to train their neural network with the architecture), yet i did not converted my neural network for the following reasons :
ONNX is evolving quickly, yet is currently not supporting all the operations you can possibly do in a PyTorch model. So if you have a highly custom or specific neural network (like in my case), you might not be able to convert it to ONNX with ease. You might need to change your architecture, or maybe have to re-write a big part of it so that it can be converted to ONNX.
You will need to use one or two additional tools, where most tutorials are not going really deep, or not explaining the logic behind what they are doing.
Note that you might want to convert your neural network if you call your network billions or trillions of times a day, otherwise i think you can stick with PyTorch without issues even for production, and avoid the fallback of converting to ONNX.
First let's see how we can save a trained neural network, load it back trough the architecture of the network, and re-run the trained network.
Second how we can deploy a network to a website, and also how you can allow users to train their networks. It is likely not the best or most efficient way, yet it sure works.
Saving the network :
First, you clearly need to have imported pyTorch with "import torch". Inside your neural network file you should save the stateDict (basically a dictionary of the operations and weights of your network) of the network you want to re-use. You could for example only save the stateDict of the model with the smallest loss of your epoch.
# network is the variable containing your neural network class
network_stateDict = network.state_dict()
# Saving network stateDict to a variable
Then when you want to save the stateDict to a file that you can re-use later, use :
torch.save(network_stateDict, "folderPath/myStateDict.pt)
# Saving the stateDict variable to a file
# The pt extension is just a convention in the PyTorch community, ptr is also used a lot
Finally when you will want to re-use your trained network later on, you will need to :
network = myNetwork(1, 2, 3)
# Load the architecture of the network in a variable (use the same architecture
# and the same network parameters as the ones used to create the stateDict)
network.load_state_dict(torch.load(folderPath/myStateDict.pt))
# Loading the file containing the stateDict of the trained network into a format
# pyTorch can read with the torch.load function. Then load the stateDict inside the
# network architecture with the load_state_dict function, applied to your network
# object with network.load_state_dict .
network.eval()
# To make sure that the stateDict has correctly been loaded.
output = network(input_data)
# You should now be able to get output data from your
# trained network, by feeding it a single set of input data.
For more infos on saving models and the stateDicts : https://pytorch.org/tutorials/beginner/saving_loading_models.html
Deploying the network:
Now that we know how to save, restore and feed input data to a network, all that there is left to do is to deploy it so that this process is done trough the website.
You first need to get (likely from your user) the inputs that your neural network will use. I am not going to include any link, since there is so many different web frameworks.
You would then need to either use a framework (like Django) that allow you to do in Python the logic of :
import torch
network = myNetwork(1, 2, 3)
network.load_state_dict(torch.load(folderPath/myStateDict.pt))
network.eval()
input_data = data_fromMyUser
output = network(input_data)
Then you would collect the output to display it, or do whatever you want it.
If your framework is not giving you the ability to use Python, i think it would be a good idea to have a tiny Python script, to which you would give the input data, and which would return the output.
If you would like to give the possibility to the user to train networks, you should just give them the possibility to start the training of one, and then use torch.save on a stateDict object to save the stateDict to a file.
You or they could later use the trained networks (you should also need to create a little function to make sure that you do not override previous stateDict files).
How to apply it to reinforcement learning :
I did not deploy a reinforcement learning model, yet i can offer you some ideas and leads to explore to deploy one.
You could store and add the inputs that you get from your user to a file or a database, and write a little program, that say every 24 hours or every hour, re-run the neural network with the now bigger dataset.
You could then totally apply the suggestions in this answer, of running the network, saving the stateDict of the model and then changing the stateDict that your network is using in production.
This is a bit hacky, yet would allow you to save in a "static way" your trained networks, and still have them evolving and changing their stateDicts.
Conclusion
This is clearly not the most mass-scale production approach that you could employ, yet it is in my opinion the easiest to put in place.
You also know that the output that you will get, will be the actual output of your neural network, without any distorsions or errors in the values.
Have a great day !
save the trained model however you want (HD5 or with pickle)
write the program to handle in production by loading the trained model
deploy the program on distributed system for real time computation like on Apache storm, Flink, Alink, Apache Samoa etc..
if you feel you need to retrain the model depending on feedback then retrain the model on different cluster or parallel environment and observe the model accuracy if looks good then move the model to production (initial days you need to retrain multiple times and it will decrease time goes on if your model is designed in a good way)
Now I have 10GB of data set to train the model in sklearn, but my computer only has 8GB of memory, so I have other ways to go besides incremental classifier.
I think sklearn can be used for larger data if the technique is right. If your chosen algorithms support partial_fit or an online learning approach then you're on track. The chunk_size may influence your success
This link may be useful( Working with big data in python and numpy, not enough ram, how to save partial results on the disc?)
Another thing you can do is to randomly pick whether or not to keep a row in your csv file...and save the result to a .npy file so it loads quicker. That way you get a sampling of your data that will allow you to start playing with it with all algorithms...and deal with the bigger data issue along the way(or not at all! sometimes a sample with a good approach is good enough depending on what you want).
I would like to apply fast online dimensionality reduction techniques such as (online/mini-batch) Dictionary Learning on big text corpora.
My input data naturally do not fit in the memory (this is why i want to use an online algorithm) so i am looking for an implementation that can iterate over a file rather than loading everything in memory.
Is it possible to do this with sklearn ? are there alternatives ?
Thanks
register
For some algorithms supporting partial_fit, it would be possible to write an outer loop in a script to do out-of-core, large scale text classification. However there are some missing elements: a dataset reader that iterates over the data on the disk as folders of flat files or a SQL database server, or NoSQL store or a Solr index with stored fields for instance. We also lack an online text vectorizer.
Here is a sample integration template to explain how it would fit together.
import numpy as np
from sklearn.linear_model import Perceptron
from mymodule import SomeTextDocumentVectorizer
from mymodule import DataSetReader
dataset_reader = DataSetReader('/path/to/raw/data')
expected_classes = dataset_reader.get_all_classes() # need to know the possible classes ahead of time
feature_extractor = SomeTextDocumentVectorizer()
classifier = Perceptron()
dataset_reader = DataSetReader('/path/to/raw/data')
for i, (documents, labels) in enumerate(dataset_reader.iter_chunks()):
vectors = feature_extractor.transform(documents)
classifier.partial_fit(vectors, labels, classes=expected_classes)
if i % 100 == 0:
# dump model to be able to monitor quality and later analyse convergence externally
joblib.dump(classifier, 'model_%04d.pkl' % i)
The dataset reader class is application specific and will probably never make it into scikit-learn (except maybe for a folder of flat text files or CSV files that would not require to add a new dependency to the library).
The text vectorizer part is more problematic. The current vectorizer does not have a partial_fit method because of the way we build the in-memory vocabulary (a python dict that is trimmed depending on max_df and min_df). We could maybe build one using an external store and drop the max_df and min_df features.
Alternatively we could build an HashingTextVectorizer that would use the hashing trick to drop the dictionary requirements. None of those exist at the moment (although we already have some building blocks such as a murmurhash wrapper and a pull request for hashing features).
In the mean time I would advise you to have a look at Vowpal Wabbit and maybe those python bindings.
Edit: The sklearn.feature_extraction.FeatureHasher class has been merged into the master branch of scikit-learn and will be available in the next release (0.13). Have a look at the documentation on feature extraction.
Edit 2: 0.13 is now released with both FeatureHasher and HashingVectorizerthat can directly deal with text data.
Edit 3: there is now an example on out-of-core learning with the Reuters dataset in the official example gallery of the project.
Since Sklearn 0.13 there is indeed an implementation of the HashingVectorizer.
EDIT: Here is a full-fledged example of such an application
Basically, this example demonstrates that you can learn (e.g. classify text) on data that cannot fit in the computer's main memory (but rather on disk / network / ...).
In addition to Vowpal Wabbit, gensim might be interesting as well - it too features online Latent Dirichlet Allocation.