Related
I'm learning ML, and I have an innocent question regarding the epsilon value used in tf.keras.
What is exactly K.epsilon()? Is it truly the machine epsilon commonly used in other programming languages?
If K.epsilon() is truly that concept, then why keras is not changing its value after I explicitly asked it to use double precision numbers?
Is it enough to use K.set_floatx('float64') if I want to use double precision numbers in the whole code? Or do I need to declare that for each variable I'm using?
Please, keep in mind that I'm used to program in FORTRAN, so it is natural to me to declare the precision for each variable, and stick to over the whole code and its dependencies.
Consider the following code
import tensorflow as tf
from tensorflow.keras import backend as K
eps = K.epsilon()
print(eps)
The output is 1e-07, which leads me to conclude that tf.keras is using simple precision numbers (float32). Searching on the internet, I've found that we can change that using
K.set_floatx('float64')
Now I want to know the new value of epsilon
import tensorflow as tf
from tensorflow.keras import backend as K
K.set_floatx('float64')
eps = K.epsilon()
print(eps)
but the output is again 1e-07!
That explains the questions. Please tell me if I'm misguided or something.
I have a model where the likelihood involves computing the sum of all the terms in the matrix
P = U exp(tD) U^-1
Where
UDU^-1 = Q
and Q is my matrix of parameters.If I wanted to use NUTS in PyMC3, NUTS would have to be able to compute the derivative of all the elements in P with respect to each of the elements in Q. Is this possible using the symbolic differentiator in Theano that PyMC3 uses?
PyMC3 uses Theano for computation and autodiff. Theano has very good support for tensor algebra (of which matrix algebra is a subset) so I think that your model should be supported.
In the Java version of LIBLINEAR there is a class called 'SolverType' in which one can choose type of the loss function to which they want to optimize the function. For example 'SolverType.L2LOSS_SVM_DUAL'. Is there any way to define a user-defined loss function?
The short answer is no.
The "loss function" defines the optimization problem, in fact this parameter changes (in particular) this model to
linear regression
logistic regression
support vector machine
While first two are quite similar, third requires completely different machinery to solve it, much more complex methods. In particular one can define very arbitrary functions, which fall into "linear models" category, which are unsolvable (are solvable by very complex techniques).
On the other hand, if the function is very simple, ie. it is a differentiable function, without any bounds (optimization is performed on the whole parameters space) then (assuming you know analytical form of the derivatives) you can plug it in into any steepest descent algorithm implementation (there are dozens of such solvers avaliable).
SVM is formulated as a QP problem.
minimize ||w|| w.r.t
y * (w'x) >= 1 for all (x, y) in the training dataset
This is the dual form of the problem and the objective is to minimize the L2 norm of the weight w.
If you change the objective ||w|| then it is no longer SVM. However, you can change the weight of training examples. You can find a tutorial here:
http://scikit-learn.org/stable/modules/svm.html#unbalanced-problems
Intel MKL library offers optimized set of threaded functions, but for case of iterative sparse solver (ISS), the preconditioned conjugate gradient method does not seem to be straightforward to be threaded.
To be more precise, using preconditioning techniques such as incomplete Cholesky factorization or ILU, at some point sparse triangular solvers are required, but corresponding MKL function to perform triangular solving mkl_cspblas_?csrtrsv is not threaded. My question is if there are any sparse solver libraries to beat the current version of MKL (which is not fully threaded) on multi-core processors?
I don't know if it beats MKL - it is a research project after all - but there is a mixed-mode OpemMP+MPI branch of PETSc available. In my group's experimentation with the SpMV implementation it scaled well for simple matrices, but not for the more complex ones we deal with regularly. You may have better luck, depending on your problem.
I'm using the SVM classifier in the machine learning scikit-learn package for python.
My features are integers. When I call the fit function, I get the user warning "Scaler assumes floating point values as input, got int32", the SVM returns its prediction, I calculate the confusion matrix (I have 2 classes) and the prediction accuracy.
I've tried to avoid the user warning, so I saved the features as floats. Indeed, the warning disappeared, but I got a completely different confusion matrix and prediction accuracy (surprisingly much less accurate)
Does someone know why it happens? What is preferable, should I send the features as float or integers?
Thanks!
You should convert them as floats but the way to do it depends on what the integer features actually represent.
What is the meaning of your integers? Are they category membership indicators (for instance: 1 == sport, 2 == business, 3 == media, 4 == people...) or numerical measures with an order relationship (3 is larger than 2 that is in turn is larger than 1). You cannot say that "people" is larger than "media" for instance. It is meaningless and would confuse the machine learning algorithm to give it this assumption.
Categorical features should hence be transformed to explode each feature as several boolean features (with value 0.0 or 1.0) for each possible category. Have a look at the DictVectorizer class in scikit-learn to better understand what I mean by categorical features.
If there are numerical values just convert them as floats and maybe use the Scaler to have them loosely in the range [-1, 1]. If they span several order of magnitudes (e.g. counts of word occurrences) then taking the logarithm of the counts might yield better results. More documentation on feature preprocessing and examples in this section of the documentation: http://scikit-learn.org/stable/modules/preprocessing.html
Edit: also read this guide that has many more details for features representation and preprocessing: http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf