Mixture prior with Turing.jl - statistics
I am trying to write a model with a mixture prior on one of the parameters. However I run into TypeError: in typeassert, expected Float64, got a value of type ForwardDiff.Dual{Nothing, Float64, 2}
In Stan it's possible to write this type of prior by incrementing the target with the logsumexp of each of the mixture densities. Is that the way to handle this in Turing.jl also?
A simplified version of my model and inference code:
using Random
using Turing
Random.seed!(42);
x = randn(2000);
#model function model(x)
s ~ InverseGamma(2, 3)
mean = [0, 0]
sd = [0.0309, 0.3479]
mix = [0.9736, 0.0264]
k = 2
m ~ MixtureModel(
Normal,
[(mean[j], sd[j]) for j = 1:k],
mix
)
for i = 1:length(x)
x[i] ~ Normal(m, sqrt(s))
end
end;
m = model(x);
samples_nuts = sample(m, NUTS(200, 0.65), 10_000);
The start of the stack trace is
ERROR: TypeError: in typeassert, expected Float64, got a value of type ForwardDiff.Dual{Nothing, Float64, 2}
Stacktrace:
[1] setindex!(A::Vector{Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Turing.Core.var"#f#7"{DynamicPPL.TypedVarInfo{NamedTuple{(:s, :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s, Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{MixtureModel{Univariate, Continuous, Normal, Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Model{var"#7#10", (:x,), (), (), Tuple{Vector{Float64}}, Tuple{}}, DynamicPPL.Sampler{NUTS{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.DiagEuclideanMetric}}, DynamicPPL.DefaultContext}, Float64}, Float64, 2}, i1::Int64)
# Base ./array.jl:903
[2] _mixlogpdf1(d::MixtureModel{Univariate, Continuous, Normal, Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Turing.Core.var"#f#7"{DynamicPPL.TypedVarInfo{NamedTuple{(:s, :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s, Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{MixtureModel{Univariate, Continuous, Normal, Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Model{var"#7#10", (:x,), (), (), Tuple{Vector{Float64}}, Tuple{}}, DynamicPPL.Sampler{NUTS{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.DiagEuclideanMetric}}, DynamicPPL.DefaultContext}, Float64}, Float64, 2})
# Distributions ~/.julia/packages/Distributions/HjzA0/src/mixtures/mixturemodel.jl:358
[3] logpdf_with_trans(d::MixtureModel{Univariate, Continuous, Normal, Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Turing.Core.var"#f#7"{DynamicPPL.TypedVarInfo{NamedTuple{(:s, :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s, Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{MixtureModel{Univariate, Continuous, Normal, Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Model{var"#7#10", (:x,), (), (), Tuple{Vector{Float64}}, Tuple{}}, DynamicPPL.Sampler{NUTS{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.DiagEuclideanMetric}}, DynamicPPL.DefaultContext}, Float64}, Float64, 2}, transform::Bool)
# Bijectors ~/.julia/packages/Bijectors/3suua/src/Bijectors.jl:0
[4] assume
# ~/.julia/packages/Turing/uAz5c/src/inference/hmc.jl:529 [inlined]
[5] _tilde
# ~/.julia/packages/DynamicPPL/SgzCy/src/context_implementations.jl:62 [inlined]
pkg status
[523fee87] CodecBzip2 v0.7.2
[1624bea9] ConjugatePriors v0.4.0
[b964fa9f] LaTeXStrings v1.3.0
[91a5bcdd] Plots v1.25.5
[df47a6cb] RData v0.8.3
[f3b207a7] StatsPlots v0.14.30
[fce5fe82] Turing v0.15.1
Related
Utilizing Scikit-learn with Python3.11 path in Julia
I'm trying to perform some benchmarking in clustering by various frameworks, But in the case of porting Scikit-learn from python to julia, I can't make it even work. Here is the code: using PyCall Train = rand(Float64, 1611, 10) py""" def Silhouette_py(Train, k): from sklearn.metrics import silhouette_score from sklearn.cluster import KMeans model = KMeans(n_clusters=k) return silhouette_score(Train, model.labels_) """ function test(Train, k) py"Silhouette_py"(Train, k) end The following code leads to an error: julia> test(Train, 3) ERROR: PyError ($(Expr(:escape, :(ccall(#= C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:43 =# #pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'AttributeError'> AttributeError("'KMeans' object has no attribute 'labels_'") File "C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyeval.jl", line 5, in Silouhette_py const _namespaces = Dict{Module,PyDict{String,PyObject,true}}() ^^^^^^^^^^^^^ Stacktrace: [1] pyerr_check # C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:62 [inlined] [2] pyerr_check # C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:66 [inlined] [3] _handle_error(msg::String) # PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:83 [4] macro expansion # C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:97 [inlined] [5] #107 # C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:43 [inlined] [6] disable_sigint # .\c.jl:473 [inlined] [7] __pycall! # C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:42 [inlined] [8] _pycall!(ret::PyObject, o::PyObject, args::Tuple{Matrix{Float64}, Int64}, nargs::Int64, kw::Ptr{Nothing}) # PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:29 [9] _pycall!(ret::PyObject, o::PyObject, args::Tuple{Matrix{Float64}, Int64}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) # PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:11 [10] (::PyObject)(::Matrix{Float64}, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) # PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:86 [11] (::PyObject)(::Matrix{Float64}, ::Vararg{Any}) # PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:86 [12] t(Train::Matrix{Float64}, k::Int64) # Main .\REPL[12]:2 [13] top-level scope # REPL[20]:1 The libpython and related stuff configuration: julia> PyCall.libpython "C:\\Users\\Shayan\\AppData\\Local\\Programs\\Python\\Python311\\python311.dll" julia> PyCall.pyversion v"3.11.0" julia> PyCall.current_python() "C:\\Users\\Shayan\\AppData\\Local\\Programs\\Python\\Python311\\python.exe" Further tests But if I say: julia> sk = pyimport("sklearn") julia> model = sk.cluster.KMeans(3) PyObject KMeans(n_clusters=3) julia> model.fit(Train) sys:1: ConvergenceWarning: Number of distinct clusters (1) found smaller than n_clusters (3). Possibly due to duplicate points in X. PyObject KMeans(n_clusters=3) julia> model.labels_ 1611-element Vector{Int32}: 0 0 0 0 0 0 ⋮ But I need it to work in a function. As you can see, it doesn't throw AttributeError("'KMeans' object has no attribute 'labels_'") anymore in this case.
It seems this would work: KMeans = pyimport("sklearn.cluster").KMeans silhouette_score = pyimport("sklearn.metric").silhouette_score Train = rand(Float64, 1611, 10); function test(Train, k) model = KMeans(k) model.fit(Train) return silhouette_score(Train, model.labels_) end julia> test(Train, 3) 0.7885442174636309
How to fix 'Mul' Op has type float32 that does not match type int64?
model.fit([training_pairs[:, 0], training_pairs[:, 1]],training_labels,batch_size=64, epochs=10) Epoch 1/10 model.fit([training_pairs[:, 0], training_pairs[:, 1]],training_labels,batch_size=64, epochs=10) TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type int64 of argument 'x'. print(training_labels.dtype) dtype('int64') print(training_pairs.dtype) dtype('float32') As I am not quite familiar with the stackoverflow tool,I have put my notebook link also. I am using tensorflow version 2.4. Much appreciate for your help. https://colab.research.google.com/drive/1z4ubLpnvPk6RWGCviCW6z_xMZ5t64gUP?usp=sharing
As shown in error, you need to change the label data type as below which is 'int64': training_labels = tf.cast(training_labels, tf.float32) training_labels.dtype OR training_labels=training_labels.astype('float32') training_labels.dtype
sqrt_vml_cpu not implemented for 'Long'
a = torch.full([2, 2], 9) b = a.sqrt() print(b) b = a.rsqrt() print(b) RuntimeError: sqrt_vml_cpu not implemented for 'Long' a is torch.LongTensor, but sqrt and rsqrt do not suppor Long, what should I do?
I couldn't reproduce your code because, when I run it, I get the following error (PyTorch 1.6): RuntimeError: Providing a bool or integral fill value without setting the optional `dtype` or `out` arguments is currently unsupported. In PyTorch 1.7, when `dtype` and `out` are not set a bool fill value will return a tensor of torch.bool dtype, and an integral fill value will return a tensor of torch.long dtype. However, I think your problem occurs because PyTorch does not support square root operation for Long/torch.int64. You should use another data type. This should do it: a = torch.full([2, 2], 9, dtype=torch.float32) b = a.sqrt() b = a.rsqrt()
Tensorflow function RAM usage keeps rising
I have a very simple tensorflow-based function that takes a tensor of shape (1, 6, 64, 64, 64, 1) and returns a tensor of shape (1, 6, 3) containing the centre of mass of each (64, 64, 64) volume in the original tensor. I works without any problems, but every time my loop (see below) goes into the next iteration, the RAM used in my pc increases. This limits me to about 500 samples before I ran completely out. I assume I'm missing something somewhere but I'm not experienced enough to know where. code: import tensorflow as tf import pickle import scipy.io import scipy.ndimage import sys from os import listdir from os.path import isfile, join import numpy as np def get_raw_centroids(lm_vol): # Find centres of mass for each landmark lm_vol *= tf.cast(tf.greater(lm_vol, 0.75), tf.float64) batch_size, lm_size, vol_size = lm_vol.shape[:3] xx, yy, zz = tf.meshgrid(tf.range(vol_size), tf.range( vol_size), tf.range(vol_size), indexing='ij') coords = tf.stack([tf.reshape(xx, (-1,)), tf.reshape(yy, (-1,)), tf.reshape(zz, (-1,))], axis=-1) coords = tf.cast(coords, tf.float64) volumes_flat = tf.reshape(lm_vol, [-1, int(lm_size), int(vol_size * vol_size * vol_size), 1]) total_mass = tf.reduce_sum(volumes_flat, axis=2) raw_centroids = tf.reduce_sum(volumes_flat * coords, axis=2) / total_mass return raw_centroids path = '/home/mosahle/Avg_vol_tf/' lm_data_path = path + 'MAT_data_volumes/' files = [f for f in listdir(lm_data_path) if isfile(join(lm_data_path, f))] files.sort() for i in range(10): sess = tf.Session() print("File {} of {}".format(i, len(files))) """ Load file """ dir = lm_data_path + files[i] lm_vol = scipy.io.loadmat(dir)['datavol'] lm_vol = tf.convert_to_tensor(lm_vol, dtype=tf.float64) lm_vol are the (1, 6, 64, 64, 64, 1) arrays. They're just numpy arrays and are converted into tensors. """ Get similarity matrix """ pts_raw = get_raw_centroids(lm_vol) print(sess.run(pts_raw)) sess.close() I've tried putting the tf.Session() outside the loop as well but it makes no difference.
The issue in the above code is you are creating multiple graphs inside the loop, when you calling the function get_raw_centroids. Lets consider a simpler example: def get_raw_centroids(lm_vol): raw_centroids = lm_vol * 2 return raw_centroids for i in range(10): sess = tf.Session() lm_vol = tf.constant(3) pts_raw = get_raw_centroids(lm_vol) print(sess.run(pts_raw)) print('****Graph: ***\n') print([x for x in tf.get_default_graph().get_operations()]) sess.close() The output of the above code is: #6 #****Graph: *** #[<tf.Operation 'Const' type=Const>, #<tf.Operation 'mul/y' type=Const>, #<tf.Operation 'mul' type=Mul>] #6 #****Graph: *** #[<tf.Operation 'Const' type=Const>, # <tf.Operation 'mul/y' type=Const>, # <tf.Operation 'mul' type=Mul>, # <tf.Operation 'Const_1' type=Const>, # <tf.Operation 'mul_1/y' type=Const>, # <tf.Operation 'mul_1' type=Mul>] #6 #****Graph: *** #[<tf.Operation 'Const' type=Const>, #<tf.Operation 'mul/y' type=Const>, #<tf.Operation 'mul' type=Mul>, #<tf.Operation 'Const_1' type=Const>, #<tf.Operation 'mul_1/y' type=Const>, #<tf.Operation 'mul_1' type=Mul>, #<tf.Operation 'Const_2' type=Const>, #<tf.Operation 'mul_2/y' type=Const>, #<tf.Operation 'mul_2' type=Mul>] ... So each loop adds a new graph with new variables along with the old graph. The correct way to handle the above code is the following: # Create a placeholder for the input lm_vol = tf.placeholder(dtype=tf.float32) pts_raw = get_raw_centroids(lm_vol) # Session for i in range(10): # numpy input lm_vol_np = 3 # pass the input to the placeholder and get the output of the graph print(sess.run(pts_raw, {lm_vol: lm_vol_np})) print('****Graph: ***\n') print([x for x in tf.get_default_graph().get_operations()]) sess.close() The output of the code will be: #6.0 #****Graph: *** #[<tf.Operation 'Placeholder' type=Placeholder>, #<tf.Operation 'mul/y' type=Const>, #<tf.Operation 'mul' type=Mul>] #6.0 #****Graph: *** #[<tf.Operation 'Placeholder' type=Placeholder>, #<tf.Operation 'mul/y' type=Const>, #<tf.Operation 'mul' type=Mul>] #6.0 #****Graph: *** #[<tf.Operation 'Placeholder' type=Placeholder>, #<tf.Operation 'mul/y' type=Const>, #<tf.Operation 'mul' type=Mul>]
tensorflow map_fn on signle value Tensor
Is it possible to run map_fn on a tensor with a single value? The following works: import tensorflow as tf a = tf.constant(1.0, shape=[3]) tf.map_fn(lambda x: x+1, a) #output: [2.0, 2.0, 2.0] However this does not: import tensorflow as tf b = tf.constant(1.0) tf.map_fn(lambda x: x+1, b) #expected output: 2.0 Is it possible at all? What am I doing wrong? Any hints will be greatly appreciated!
Well, I see you accepted an answer, which correctly states that tf.map_fn() is applying a function to elements of a tensor, and a scalar tensor has no elements. But it's not impossible to do this for a scalar tensor, you just have to tf.reshape() it before and after, like this code (tested): import tensorflow as tf b = tf.constant(1.0) if () == b.get_shape(): c = tf.reshape( tf.map_fn(lambda x: x+1, tf.reshape( b, ( 1, ) ) ), () ) else: c = tf.map_fn(lambda x: x+1, b) #expected output: 2.0 with tf.Session() as sess: print( sess.run( c ) ) will output: 2.0 as desired. This way you can factor this into an agnostic function that can take both scalar and non-scalar tensors as argument.
No, this is not possible. As you probably saw it throws an error: ValueError: elems must be a 1+ dimensional Tensor, not a scalar The point of map_fn is to apply a function to each element of a tensor, so it makes no sense to use this for a scalar (single-element) tensor. As to "what you are doing wrong": This is difficult to say without knowing what you're trying to achieve.