Tensor Utils

This module contains util functions to facilitate working with tensorflow.Tensor objects.

pysgmcmc.tensor_utils.vectorize(tensor)

Turn any matrix into a long vector by expanding it.

Tranforms [[a, b], [c, d]] into [a, b, c, d].

For vector inputs, this simply returns a copy of the vector.

For reference see also vec-operator in:

https://hec.unil.ch/docs/files/23/100/handout1.pdf#page=2

Parameters:tensor (tensorflow.Variable object or tensorflow.Tensor object) – Input tensor to vectorize. Returns:tensor_vectorized – Vectorized result for input tensor. Return type:tensorflow.Variable object or tensorflow.Tensor object

Examples

A tensorflow.Variable can be vectorized: (NOTE: the returned vectorized variable must be initialized before using it in tensorflow computations.)

>>> import tensorflow as tf
>>> v1 = tf.Variable([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
>>> v1_vectorized = vectorize(v1)
>>> session = tf.Session()
>>> session.run(tf.global_variables_initializer())
>>> session.run(v1_vectorized)
array([[ 1.],
       [ 2.],
       [ 3.],
       [ 4.],
       [ 5.],
       [ 6.]], dtype=float32)

A normal tensorflow.Tensor can be vectorized:

>>> import tensorflow as tf
>>> t1 = tf.constant([[12.0, 14.0, -3.0], [4.0, 3.0, 1.0], [9.0, 2.0, 4.0]])
>>> t1_vectorized = vectorize(t1)
>>> session = tf.Session()
>>> session.run(t1_vectorized)
array([[ 12.],
       [ 14.],
       [ -3.],
       [  4.],
       [  3.],
       [  1.],
       [  9.],
       [  2.],
       [  4.]], dtype=float32)

Input of other types will raise a ValueError:

>>> import tensorflow as tf
>>> session = tf.Session()
>>> v = vectorize([1.0])
Traceback (most recent call last):
 ...
ValueError: Unsupported input to tensor_utils.vectorize: [1.0] is not a tensorflow.Tensor subclass

pysgmcmc.tensor_utils.unvectorize(tensor, original_shape)

Reshape previously vectorized tensor back to its original_shape.

Essentially the inverse transformation as the one performed by tensor_utils.vectorize.

Parameters:

• tensor (tensorflow.Variable object or tensorflow.Tensor object) – Input tensor to unvectorize.
• original_shape (tensorflow.Shape) – Original shape of tensor prior to its vectorization.

Returns:

tensor_unvectorized – Tensor with the same values as tensor but reshaped back to shape original_shape.

Return type:

tensorflow.Tensor

Examples

Function unvectorize undoes the work done by vectorize:

>>> import tensorflow as tf
>>> import numpy as np
>>> t1 = tf.constant([[12.0, 14.0, -3.0], [4.0, 3.0, 1.0], [9.0, 2.0, 4.0]])
>>> t2 = unvectorize(vectorize(t1), original_shape=t1.shape)
>>> session = tf.Session()
>>> t1_array, t2_array = session.run([t1, t2])
>>> np.allclose(t1_array, t2_array)
True

It will also work for tensorflow.Variable objects, but will return tensorflow.Tensor as unvectorized output.

>>> import tensorflow as tf
>>> import numpy as np
>>> v = tf.Variable([[0.0, 1.0], [2.0, 0.0]])
>>> session = tf.Session()
>>> session.run(tf.global_variables_initializer())
>>> t = unvectorize(vectorize(v.initialized_value()), original_shape=v.shape)
>>> v_array, t_array = session.run([v, t])
>>> np.allclose(t_array, v_array)
True

pysgmcmc.tensor_utils.median(tensor)

Return the median (middle value) of data in tensor.

When the number of data points is odd, return the middle data point. When the number of data points is even, the median is interpolated by taking the average of the two middle values.

Parameters:tensor (tensorflow.Tensor) – Input tensor (may be multidimensional) for which the median should be computed. Returns:median_tensor – Scalar tensor whose value is the median of the input tensor. Return type:tensorflow.Tensor

Examples

When the number of data points is odd, return the middle data point:

>>> import tensorflow as tf
>>> t = tf.constant([1, 3, 5], dtype=tf.float64)
>>> with tf.Session() as session: session.run(median(t))
3.0

When the number of data points is even, the median is interpolated by taking the average of the two middle values:

>>> import tensorflow as tf
>>> t = tf.constant([1, 3, 5, 7], dtype=tf.float64)
>>> with tf.Session() as session: session.run(median(t))
4.0

pysgmcmc.tensor_utils.safe_divide(x, y, small_constant=1e-16, name=None)

tf.divide(x, y) after adding a small appropriate constant to y

in a smart way so that we can avoid division-by-zero artefacts.

Parameters:

• x (tensorflow.Tensor) – Left-side operand of tensorflow.divide
• y (tensorflow.Tensor) – Right-side operand of tensorflow.divide
• small_constant (tensorflow.Tensor) – Small constant tensor to add to/subtract from y before computing x / y to avoid division-by-zero.
• name (string or NoneType, optional) – Name of the resulting node in a tensorflow.Graph. Defaults to None.

Returns:

division_result – Result of division tf.divide(x, y) after applying clipping to y.

Return type:

tensorflow.Tensor

Examples

Will safely avoid divisions-by-zero under normal circumstances:

>>> import tensorflow as tf
>>> import numpy as np
>>> session = tf.Session()
>>> x = tf.constant(1.0)
>>> nan_tensor = tf.divide(x, 0.0)  # will produce "inf" due to division-by-zero
>>> np.isinf(nan_tensor.eval(session=session))
True
>>> z = safe_divide(x, 0., small_constant=1e-16)  # will avoid "inf" due to division-by-zero by clipping
>>> np.isinf(z.eval(session=session))
False

To see that simply adding a constant may fail, but this implementation handles those corner cases correctly, consider this example:

>>> import tensorflow as tf
>>> import numpy as np
>>> x, y = tf.constant(1.0), tf.constant(-1e-16)
>>> small_constant = tf.constant(1e-16)
>>> v1 = x / (y + small_constant)  # without sign
>>> v2 = safe_divide(x, y, small_constant=small_constant) # with sign
>>> val1, val2 = session.run([v1, v2])
>>> np.isinf(val1) # simply adding without considering the sign can still yield "inf"
True
>>> np.isinf(val2)  # our version behaves appropriately
False

pysgmcmc.tensor_utils.safe_sqrt(x, clip_value_min=0.0, clip_value_max=inf, name=None)

Computes tf.sqrt(x) after clipping tensor x using

tf.clip_by_value(x, clip_value_min, clip_value_max) to avoid square root (e.g. of negative values) artefacts.

Parameters:

• x (tensorflow.Tensor or tensorflow.SparseTensor) – Operand of tensorflow.sqrt.
• clip_value_min (0-D (scalar) tensorflow.Tensor, optional) – The minimum value to clip by. Defaults to 0
• clip_value_max (0-D (scalar) tensorflow.Tensor, optional) – The maximum value to clip by. Defaults to float(“inf”)
• name (string or NoneType, optional) – Name of the resulting node in a tensorflow.Graph. Defaults to None.

Returns:

sqrt_result – Result of square root tf.sqrt(x) after applying clipping to x.

Return type:

tensorflow.Tensor

Examples

Will safely avoid square root of negative values:

>>> import tensorflow as tf
>>> import numpy as np
>>> x = tf.constant(-1e-16)
>>> z = tf.sqrt(x)  # fails, results in 'nan'
>>> z_safe = safe_sqrt(x)  # works, results in '0'
>>> session = tf.Session()
>>> z_val, z_safe_val = session.run([z, z_safe])
>>> np.isnan(z_val)  # ordinary tensorflow computation gives 'nan'
True
>>> np.isnan(z_safe_val) # `safe_sqrt` produces '0'.
False
>>> z_safe_val
0.0

pysgmcmc.tensor_utils.pdist(tensor, metric='euclidean')

Pairwise distances between observations in n-dimensional space. Ported from scipy.spatial.distance.pdist @2f5aa264724099c03772ed784e7a947d2bea8398 for cherry-picked distance metrics.

Parameters:

• tensor (tensorflow.Tensor) –
• metric (string, optional) – Pairwise metric to apply. Defaults to “euclidean”.

Returns:

Y – Returns a condensed distance matrix Y as tensorflow.Tensor. For each \(i\) and \(j\) (where \(i<j<m\)), where m is the number of original observations. The metric dist(u=X[i], v=X[j]) is computed and stored in entry j of subtensor Y[j].

Return type:

tensorflow.Tensor

Examples

Gives equivalent results to scipy.spatial.distance.pdist but uses tensorflow.Tensor objects:

>>> import tensorflow as tf
>>> import numpy as np
>>> from scipy.spatial.distance import pdist as pdist_scipy
>>> input_scipy = np.array([[ 0.77228064,  0.09543156], [ 0.3918973 ,  0.96806584], [ 0.66008144,  0.22163063]])
>>> result_scipy = pdist_scipy(input_scipy, metric="euclidean")
>>> session = tf.Session()
>>> input_tensorflow = tf.constant(input_scipy)
>>> result_tensorflow = session.run(pdist(input_tensorflow, metric="euclidean"))
>>> np.allclose(result_scipy, result_tensorflow)
True

Will raise a NotImplementedError for unsupported metric choices:

>>> import tensorflow as tf
>>> import numpy as np
>>> input_scipy = np.array([[ 0.77228064,  0.09543156], [ 0.3918973 ,  0.96806584], [ 0.66008144,  0.22163063]])
>>> session = tf.Session()
>>> input_tensorflow = tf.constant(input_scipy)
>>> session.run(pdist(input_tensorflow, metric="lengthy_metric"))
Traceback (most recent call last):
 ...
NotImplementedError: tensor_utils.pdist: Metric 'lengthy_metric' currently not supported!

Like scipy.spatial.distance.pdist, we fail for input that is not 2-d:

>>> import tensorflow as tf
>>> import numpy as np
>>> input_scipy = np.random.rand(2, 2, 1)
>>> session = tf.Session()
>>> input_tensorflow = tf.constant(input_scipy)
>>> session.run(pdist(input_tensorflow, metric="lengthy_metric"))
Traceback (most recent call last):
 ...
ValueError: tensor_utils.pdist: A 2-d tensor must be passed.

pysgmcmc.tensor_utils.squareform(tensor)

Converts a vector-form distance vector to a square-form distance matrix. Ported from scipy.spatial.distance.squareform @2f5aa264724099c03772ed784e7a947d2bea8398, but supports only 1-d (vector) input.

Parameters:tensor (tensorflow.Tensor) – Returns:redundant_distance_tensor Return type:tensorflow.Tensor

Examples

May be used in conjunction with tensor_utils.pdist to obtain a redundant distance matrix:

>>> import tensorflow as tf
>>> import numpy as np
>>> from scipy.spatial.distance import pdist as scipy_pdist, squareform as scipy_squareform
>>> original_input = np.random.rand(2, 4)
>>> tf_redundant_distance_tensor = squareform(pdist(tf.constant(original_input)))
>>> scipy_redundant_distance_matrix = scipy_squareform(scipy_pdist(original_input))
>>> session = tf.Session()
>>> tf_redundant_distance_matrix = session.run(tf_redundant_distance_tensor)
>>> np.allclose(tf_redundant_distance_matrix, scipy_redundant_distance_matrix)
True

Contrary to scipy.spatial.squareform, conversion of 2D input to a condensed distance vector is not supported:

>>> import numpy as np
>>> import tensorflow as tf
>>> illegal_input = tf.constant(np.random.rand(4, 4))
>>> squareform(illegal_input)
Traceback (most recent call last):
 ...
NotImplementedError: tensor_utils.squareform: Only 1-d (vector) input is supported!

pysgmcmc.tensor_utils.uninitialized_params(params, session)

Return the list containing all tensorflow.Variable objects present in iterable params that are not yet initialized.

Parameters:

• params (list of tensorflow.Variable objects) – List of parameters to check for initialization.
• session (tf.Session) – Session used to determine which parameters are uninitialized.

Returns:

params_uninitialized – All tensorflow.Variable objects in params that were not yet initialized in the current graph.

Return type:

list of tensorflow.Variable objects