I remember that in TensorFlow, there are some operators can only be used for forward operations, and no backward operations being supported/implemented for back-propagation.
My question is, are there any these kind of operators in Pytorch? If yes, could you please give some references? Thanks in advance.
Related
I am sorry I am asking this question, it seems so stupid, but I have no other way to look for it. If you have some parallel documentation that easily compares this kind of questions, please let me know.
Good people of SO, the SparseCategoricalCrossentropy is equivalent to it in TF2.0 : https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy
It's SparseCategoricalCrossentropy and not CategoricalCrossentropy, because torch's cross entropy does not use one hot encoding as CategoricalCrossentropy in Tensorflow does.
I have read many tutorials on how to use PyTorch to make a regression over a data set, using, for instance, a model composed of several linear layers and the MSE loss.
Well, imagine that I know the function F depends on a variable x and some unknown parameters (p_j: j=0,..., P-1) with P relatively small, but the function is a composition of special function. So, my problem is the classical minimization knowing the data {x_i,y_i}_i<=N
Min_{ {p_j} } Sum_i (F(x_i;{p_j}) - y_i)^2
I would like to know if I can use the PyTorch optimizers and if yes how I can do it?
Thanks.
In fact, PyTorch experts answer that the function to minimized must be expressed in terms of torch.tensors to let the minimizers computing the gradients. So, it is not possible in my case.
I have been reading some Keras codes on Seq2Seq and come to wonder whether the teacher forcing is used or not during its training? I couldn't find any parameters? I might have missed though.
Is the teacher forcing on by default in Keras? Or Keras doesn't support it natively yet?
Thank you in advance!
Has anyone tried using a python probabilistic programming library with Spark? Or does anyone have a good idea of what it would take?
I have a feeling Edward would be simplest because there are already tools connecting Tensorflow and Spark, but still hazy about what low-level code changes would be required.
I know distributed MCMC is still an area of active research (see MC-Stan on Spark?), so is this even reasonable to implement? Thanks!
You can use Tensorflow connectors with Edward since it is based on Tensorflow, one of the main drawbacks of MCMC is very computational intensive, you may try Variational inference for your Bayesian models it approximates the target distribution. (this also applies to Pyro and PyMC3 I believe), you can also work with Tensorflow distributed tensorflow distributed
I also recommend you to use/try a library called "Dask
"https://dask.pydata.org/en/latest/Dask, you can scale your model from your workstation to a cluster it also has Tensorflow connectors.
Hope this helps
I've seen people run Pyro+PyTorch in PySpark, but the use case was CPU-only and did not involve distributed training.
I am new to Pytorch and I have got into some trouble.
I want to build a rank-model to judge the similarity of question and its answers (including right answers and wrong answers). And I use LSTM as an encoder.
There are two LSTMs in my model and they share the weights. So my model’s input is two sequences (question and answer). But if I use batch, reordering will disrupt the correspondence between questions and answers.
What should I do?
Maybe a starting point could be to use similar RNN wrapper as here https://github.com/facebookresearch/DrQA/blob/master/drqa/reader/layers.py#L20
You can encode question and asnwer separately (this module will pack and unpack internally and will take care of sorting) and then you can work on the encoded representations
I have uploaded some useful torch utilities for NLP. Some might be useful for pack_padded_sequence: https://github.com/wjpjet/torch_utilities/blob/master/torch_util.py