Mixture prior with Turing.jl - statistics

I am trying to write a model with a mixture prior on one of the parameters. However I run into TypeError: in typeassert, expected Float64, got a value of type ForwardDiff.Dual{Nothing, Float64, 2}
In Stan it's possible to write this type of prior by incrementing the target with the logsumexp of each of the mixture densities. Is that the way to handle this in Turing.jl also?
A simplified version of my model and inference code:
using Random
using Turing
Random.seed!(42);
x = randn(2000);
#model function model(x)
s ~ InverseGamma(2, 3)
mean = [0, 0]
sd = [0.0309, 0.3479]
mix = [0.9736, 0.0264]
k = 2
m ~ MixtureModel(
Normal,
[(mean[j], sd[j]) for j = 1:k],
mix
)
for i = 1:length(x)
x[i] ~ Normal(m, sqrt(s))
end
end;
m = model(x);
samples_nuts = sample(m, NUTS(200, 0.65), 10_000);
The start of the stack trace is
ERROR: TypeError: in typeassert, expected Float64, got a value of type ForwardDiff.Dual{Nothing, Float64, 2}
Stacktrace:
[1] setindex!(A::Vector{Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Turing.Core.var"#f#7"{DynamicPPL.TypedVarInfo{NamedTuple{(:s, :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s, Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{MixtureModel{Univariate, Continuous, Normal, Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Model{var"#7#10", (:x,), (), (), Tuple{Vector{Float64}}, Tuple{}}, DynamicPPL.Sampler{NUTS{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.DiagEuclideanMetric}}, DynamicPPL.DefaultContext}, Float64}, Float64, 2}, i1::Int64)
# Base ./array.jl:903
[2] _mixlogpdf1(d::MixtureModel{Univariate, Continuous, Normal, Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Turing.Core.var"#f#7"{DynamicPPL.TypedVarInfo{NamedTuple{(:s, :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s, Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{MixtureModel{Univariate, Continuous, Normal, Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Model{var"#7#10", (:x,), (), (), Tuple{Vector{Float64}}, Tuple{}}, DynamicPPL.Sampler{NUTS{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.DiagEuclideanMetric}}, DynamicPPL.DefaultContext}, Float64}, Float64, 2})
# Distributions ~/.julia/packages/Distributions/HjzA0/src/mixtures/mixturemodel.jl:358
[3] logpdf_with_trans(d::MixtureModel{Univariate, Continuous, Normal, Float64}, x::ForwardDiff.Dual{ForwardDiff.Tag{Turing.Core.var"#f#7"{DynamicPPL.TypedVarInfo{NamedTuple{(:s, :m), Tuple{DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:s, Tuple{}}, Int64}, Vector{InverseGamma{Float64}}, Vector{AbstractPPL.VarName{:s, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}, DynamicPPL.Metadata{Dict{AbstractPPL.VarName{:m, Tuple{}}, Int64}, Vector{MixtureModel{Univariate, Continuous, Normal, Float64}}, Vector{AbstractPPL.VarName{:m, Tuple{}}}, Vector{Float64}, Vector{Set{DynamicPPL.Selector}}}}}, Float64}, DynamicPPL.Model{var"#7#10", (:x,), (), (), Tuple{Vector{Float64}}, Tuple{}}, DynamicPPL.Sampler{NUTS{Turing.Core.ForwardDiffAD{40}, (), AdvancedHMC.DiagEuclideanMetric}}, DynamicPPL.DefaultContext}, Float64}, Float64, 2}, transform::Bool)
# Bijectors ~/.julia/packages/Bijectors/3suua/src/Bijectors.jl:0
[4] assume
# ~/.julia/packages/Turing/uAz5c/src/inference/hmc.jl:529 [inlined]
[5] _tilde
# ~/.julia/packages/DynamicPPL/SgzCy/src/context_implementations.jl:62 [inlined]
pkg status
[523fee87] CodecBzip2 v0.7.2
[1624bea9] ConjugatePriors v0.4.0
[b964fa9f] LaTeXStrings v1.3.0
[91a5bcdd] Plots v1.25.5
[df47a6cb] RData v0.8.3
[f3b207a7] StatsPlots v0.14.30
[fce5fe82] Turing v0.15.1

Related

Utilizing Scikit-learn with Python3.11 path in Julia

I'm trying to perform some benchmarking in clustering by various frameworks, But in the case of porting Scikit-learn from python to julia, I can't make it even work. Here is the code:
using PyCall
Train = rand(Float64, 1611, 10)
py"""
def Silhouette_py(Train, k):
from sklearn.metrics import silhouette_score
from sklearn.cluster import KMeans
model = KMeans(n_clusters=k)
return silhouette_score(Train, model.labels_)
"""
function test(Train, k)
py"Silhouette_py"(Train, k)
end
The following code leads to an error:
julia> test(Train, 3)
ERROR: PyError ($(Expr(:escape, :(ccall(#= C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:43 =# #pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'AttributeError'>
AttributeError("'KMeans' object has no attribute 'labels_'")
File "C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyeval.jl", line 5, in Silouhette_py
const _namespaces = Dict{Module,PyDict{String,PyObject,true}}()
^^^^^^^^^^^^^
Stacktrace:
[1] pyerr_check
# C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:62 [inlined]
[2] pyerr_check
# C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:66 [inlined]
[3] _handle_error(msg::String)
# PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:83
[4] macro expansion
# C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\exception.jl:97 [inlined]
[5] #107
# C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:43 [inlined]
[6] disable_sigint
# .\c.jl:473 [inlined]
[7] __pycall!
# C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:42 [inlined]
[8] _pycall!(ret::PyObject, o::PyObject, args::Tuple{Matrix{Float64}, Int64}, nargs::Int64, kw::Ptr{Nothing})
# PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:29
[9] _pycall!(ret::PyObject, o::PyObject, args::Tuple{Matrix{Float64}, Int64}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
# PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:11
[10] (::PyObject)(::Matrix{Float64}, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(),
Tuple{}}})
# PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:86
[11] (::PyObject)(::Matrix{Float64}, ::Vararg{Any})
# PyCall C:\Users\Shayan\.julia\packages\PyCall\ygXW2\src\pyfncall.jl:86
[12] t(Train::Matrix{Float64}, k::Int64)
# Main .\REPL[12]:2
[13] top-level scope
# REPL[20]:1
The libpython and related stuff configuration:
julia> PyCall.libpython
"C:\\Users\\Shayan\\AppData\\Local\\Programs\\Python\\Python311\\python311.dll"
julia> PyCall.pyversion
v"3.11.0"
julia> PyCall.current_python()
"C:\\Users\\Shayan\\AppData\\Local\\Programs\\Python\\Python311\\python.exe"
Further tests
But if I say:
julia> sk = pyimport("sklearn")
julia> model = sk.cluster.KMeans(3)
PyObject KMeans(n_clusters=3)
julia> model.fit(Train)
sys:1: ConvergenceWarning: Number of distinct clusters (1) found smaller than n_clusters (3). Possibly due to duplicate points in X.
PyObject KMeans(n_clusters=3)
julia> model.labels_
1611-element Vector{Int32}:
0
0
0
0
0
0
⋮
But I need it to work in a function. As you can see, it doesn't throw AttributeError("'KMeans' object has no attribute 'labels_'") anymore in this case.
It seems this would work:
KMeans = pyimport("sklearn.cluster").KMeans
silhouette_score = pyimport("sklearn.metric").silhouette_score
Train = rand(Float64, 1611, 10);
function test(Train, k)
model = KMeans(k)
model.fit(Train)
return silhouette_score(Train, model.labels_)
end
julia> test(Train, 3)
0.7885442174636309

How to fix 'Mul' Op has type float32 that does not match type int64?

model.fit([training_pairs[:, 0], training_pairs[:, 1]],training_labels,batch_size=64, epochs=10)
Epoch 1/10
model.fit([training_pairs[:, 0], training_pairs[:, 1]],training_labels,batch_size=64, epochs=10)
TypeError: Input 'y' of 'Mul' Op has type float32 that does not match
type int64 of argument 'x'.
print(training_labels.dtype)
dtype('int64')
print(training_pairs.dtype)
dtype('float32')
As I am not quite familiar with the stackoverflow tool,I have put my notebook link also. I am using tensorflow version 2.4. Much appreciate for your help.
https://colab.research.google.com/drive/1z4ubLpnvPk6RWGCviCW6z_xMZ5t64gUP?usp=sharing
As shown in error, you need to change the label data type as below which is 'int64':
training_labels = tf.cast(training_labels, tf.float32)
training_labels.dtype
OR
training_labels=training_labels.astype('float32')
training_labels.dtype

sqrt_vml_cpu not implemented for 'Long'

a = torch.full([2, 2], 9)
b = a.sqrt()
print(b)
b = a.rsqrt()
print(b)
RuntimeError: sqrt_vml_cpu not implemented for 'Long'
a is torch.LongTensor, but sqrt and rsqrt do not suppor Long, what should I do?
I couldn't reproduce your code because, when I run it, I get the following error (PyTorch 1.6):
RuntimeError: Providing a bool or integral fill value without setting the
optional `dtype` or `out` arguments is currently unsupported. In PyTorch 1.7,
when `dtype` and `out` are not set a bool fill value will return a tensor of
torch.bool dtype, and an integral fill value will return a tensor of
torch.long dtype.
However, I think your problem occurs because PyTorch does not support square root operation for Long/torch.int64. You should use another data type. This should do it:
a = torch.full([2, 2], 9, dtype=torch.float32)
b = a.sqrt()
b = a.rsqrt()

Tensorflow function RAM usage keeps rising

I have a very simple tensorflow-based function that takes a tensor of shape (1, 6, 64, 64, 64, 1) and returns a tensor of shape (1, 6, 3) containing the centre of mass of each (64, 64, 64) volume in the original tensor. I works without any problems, but every time my loop (see below) goes into the next iteration, the RAM used in my pc increases. This limits me to about 500 samples before I ran completely out. I assume I'm missing something somewhere but I'm not experienced enough to know where.
code:
import tensorflow as tf
import pickle
import scipy.io
import scipy.ndimage
import sys
from os import listdir
from os.path import isfile, join
import numpy as np
def get_raw_centroids(lm_vol):
# Find centres of mass for each landmark
lm_vol *= tf.cast(tf.greater(lm_vol, 0.75), tf.float64)
batch_size, lm_size, vol_size = lm_vol.shape[:3]
xx, yy, zz = tf.meshgrid(tf.range(vol_size), tf.range(
vol_size), tf.range(vol_size), indexing='ij')
coords = tf.stack([tf.reshape(xx, (-1,)), tf.reshape(yy, (-1,)),
tf.reshape(zz, (-1,))], axis=-1)
coords = tf.cast(coords, tf.float64)
volumes_flat = tf.reshape(lm_vol, [-1, int(lm_size), int(vol_size * vol_size * vol_size), 1])
total_mass = tf.reduce_sum(volumes_flat, axis=2)
raw_centroids = tf.reduce_sum(volumes_flat * coords, axis=2) / total_mass
return raw_centroids
path = '/home/mosahle/Avg_vol_tf/'
lm_data_path = path + 'MAT_data_volumes/'
files = [f for f in listdir(lm_data_path) if isfile(join(lm_data_path, f))]
files.sort()
for i in range(10):
sess = tf.Session()
print("File {} of {}".format(i, len(files)))
"""
Load file
"""
dir = lm_data_path + files[i]
lm_vol = scipy.io.loadmat(dir)['datavol']
lm_vol = tf.convert_to_tensor(lm_vol, dtype=tf.float64)
lm_vol are the (1, 6, 64, 64, 64, 1) arrays. They're just numpy arrays and are converted into tensors.
"""
Get similarity matrix
"""
pts_raw = get_raw_centroids(lm_vol)
print(sess.run(pts_raw))
sess.close()
I've tried putting the tf.Session() outside the loop as well but it makes no difference.
The issue in the above code is you are creating multiple graphs inside the loop, when you calling the function get_raw_centroids.
Lets consider a simpler example:
def get_raw_centroids(lm_vol):
raw_centroids = lm_vol * 2
return raw_centroids
for i in range(10):
sess = tf.Session()
lm_vol = tf.constant(3)
pts_raw = get_raw_centroids(lm_vol)
print(sess.run(pts_raw))
print('****Graph: ***\n')
print([x for x in tf.get_default_graph().get_operations()])
sess.close()
The output of the above code is:
#6
#****Graph: ***
#[<tf.Operation 'Const' type=Const>,
#<tf.Operation 'mul/y' type=Const>,
#<tf.Operation 'mul' type=Mul>]
#6
#****Graph: ***
#[<tf.Operation 'Const' type=Const>,
# <tf.Operation 'mul/y' type=Const>,
# <tf.Operation 'mul' type=Mul>,
# <tf.Operation 'Const_1' type=Const>,
# <tf.Operation 'mul_1/y' type=Const>,
# <tf.Operation 'mul_1' type=Mul>]
#6
#****Graph: ***
#[<tf.Operation 'Const' type=Const>,
#<tf.Operation 'mul/y' type=Const>,
#<tf.Operation 'mul' type=Mul>,
#<tf.Operation 'Const_1' type=Const>,
#<tf.Operation 'mul_1/y' type=Const>,
#<tf.Operation 'mul_1' type=Mul>,
#<tf.Operation 'Const_2' type=Const>,
#<tf.Operation 'mul_2/y' type=Const>,
#<tf.Operation 'mul_2' type=Mul>]
...
So each loop adds a new graph with new variables along with the old graph.
The correct way to handle the above code is the following:
# Create a placeholder for the input
lm_vol = tf.placeholder(dtype=tf.float32)
pts_raw = get_raw_centroids(lm_vol)
# Session
for i in range(10):
# numpy input
lm_vol_np = 3
# pass the input to the placeholder and get the output of the graph
print(sess.run(pts_raw, {lm_vol: lm_vol_np}))
print('****Graph: ***\n')
print([x for x in tf.get_default_graph().get_operations()])
sess.close()
The output of the code will be:
#6.0
#****Graph: ***
#[<tf.Operation 'Placeholder' type=Placeholder>,
#<tf.Operation 'mul/y' type=Const>,
#<tf.Operation 'mul' type=Mul>]
#6.0
#****Graph: ***
#[<tf.Operation 'Placeholder' type=Placeholder>,
#<tf.Operation 'mul/y' type=Const>,
#<tf.Operation 'mul' type=Mul>]
#6.0
#****Graph: ***
#[<tf.Operation 'Placeholder' type=Placeholder>,
#<tf.Operation 'mul/y' type=Const>,
#<tf.Operation 'mul' type=Mul>]

tensorflow map_fn on signle value Tensor

Is it possible to run map_fn on a tensor with a single value?
The following works:
import tensorflow as tf
a = tf.constant(1.0, shape=[3])
tf.map_fn(lambda x: x+1, a)
#output: [2.0, 2.0, 2.0]
However this does not:
import tensorflow as tf
b = tf.constant(1.0)
tf.map_fn(lambda x: x+1, b)
#expected output: 2.0
Is it possible at all?
What am I doing wrong?
Any hints will be greatly appreciated!
Well, I see you accepted an answer, which correctly states that tf.map_fn() is applying a function to elements of a tensor, and a scalar tensor has no elements. But it's not impossible to do this for a scalar tensor, you just have to tf.reshape() it before and after, like this code (tested):
import tensorflow as tf
b = tf.constant(1.0)
if () == b.get_shape():
c = tf.reshape( tf.map_fn(lambda x: x+1, tf.reshape( b, ( 1, ) ) ), () )
else:
c = tf.map_fn(lambda x: x+1, b)
#expected output: 2.0
with tf.Session() as sess:
print( sess.run( c ) )
will output:
2.0
as desired.
This way you can factor this into an agnostic function that can take both scalar and non-scalar tensors as argument.
No, this is not possible. As you probably saw it throws an error:
ValueError: elems must be a 1+ dimensional Tensor, not a scalar
The point of map_fn is to apply a function to each element of a tensor, so it makes no sense to use this for a scalar (single-element) tensor.
As to "what you are doing wrong": This is difficult to say without knowing what you're trying to achieve.

Resources