.. currentmodule:: tensor

.. _libdoc_basic_tensor:

===========================
Basic Tensor Functionality
===========================

.. testsetup::

   import theano.tensor as T
   from theano.tensor import scalar, iscalar, TensorType, dmatrix, ivector
   from theano.tensor import set_subtensor, inc_subtensor, batched_dot
   from theano import shared
   import numpy

Theano supports any kind of Python object, but its focus is support for
symbolic matrix expressions.  When you type,

>>> x = T.fmatrix()

the ``x`` is a :class:`TensorVariable` instance.
The ``T.fmatrix`` object itself is an instance of :class:`TensorType`.
Theano knows what type of variable ``x`` is because ``x.type``
points back to ``T.fmatrix``.

This chapter explains the various ways of creating tensor variables,
the attributes and methods of :class:`TensorVariable` and :class:`TensorType`,
and various basic symbolic math and arithmetic that Theano supports for
tensor variables.

.. _libdoc_tensor_creation:

Creation
========

Theano provides a list of predefined tensor types that can be used
to create a tensor variables.  Variables can be named to facilitate debugging,
and all of these constructors accept an optional ``name`` argument.
For example, the following each produce a TensorVariable instance that stands
for a 0-dimensional ndarray of integers with the name ``'myvar'``:

>>> x = scalar('myvar', dtype='int32')
>>> x = iscalar('myvar')
>>> x = TensorType(dtype='int32', broadcastable=())('myvar')

Constructors with optional dtype
----------------------------------------

These are the simplest and often-preferred methods for creating symbolic
variables in your code.  By default, they produce floating-point variables
(with dtype determined by config.floatX, see :attr:`floatX`) so if you use
these constructors it is easy to switch your code between different levels of
floating-point precision.

.. function:: scalar(name=None, dtype=config.floatX)

    Return a Variable for a 0-dimensional ndarray

.. function:: vector(name=None, dtype=config.floatX)

    Return a Variable for a 1-dimensional ndarray

.. function:: row(name=None, dtype=config.floatX)

    Return a Variable for a 2-dimensional ndarray
    in which the number of rows is guaranteed to be 1.

.. function:: col(name=None, dtype=config.floatX)

    Return a Variable for a 2-dimensional ndarray
    in which the number of columns is guaranteed to be 1.

.. function:: matrix(name=None, dtype=config.floatX)

    Return a Variable for a 2-dimensional ndarray

.. function:: tensor3(name=None, dtype=config.floatX)

    Return a Variable for a 3-dimensional ndarray

.. function:: tensor4(name=None, dtype=config.floatX)

    Return a Variable for a 4-dimensional ndarray

.. #COMMENT
    Each of the types described above can be constructed by two methods:
    a singular version (e.g., :ref:`dmatrix <libdoc_tensor_creation>`)
    and a plural version (:ref:`dmatrices <libdoc_tensor_creation>`).
    When called, the singular version takes a single
    argument which is the name of the *Variable* we want to make and it
    makes a single Variable of that type. The plural version can either take


All Fully-Typed Constructors
----------------------------

The following TensorType instances are provided in the theano.tensor module.
They are all callable, and accept an optional ``name`` argument.  So for example:

.. testcode:: constructors

   from theano.tensor import *

   x = dmatrix()        # creates one Variable with no name
   x = dmatrix('x')     # creates one Variable with name 'x'
   xyz = dmatrix('xyz') # creates one Variable with name 'xyz'

.. #COMMENT
    table generated by
    $ python Theano/doc/generate_dtype_tensor_table.py

============ =========== ==== =========== =================================
Constructor  dtype       ndim shape       broadcastable
============ =========== ==== =========== =================================
bscalar      int8        0    ()          ()
bvector      int8        1    (?,)        (False,)
brow         int8        2    (1,?)       (True, False)
bcol         int8        2    (?,1)       (False, True)
bmatrix      int8        2    (?,?)       (False, False)
btensor3     int8        3    (?,?,?)     (False, False, False)
btensor4     int8        4    (?,?,?,?)   (False, False, False, False)
wscalar      int16       0    ()          ()
wvector      int16       1    (?,)        (False,)
wrow         int16       2    (1,?)       (True, False)
wcol         int16       2    (?,1)       (False, True)
wmatrix      int16       2    (?,?)       (False, False)
wtensor3     int16       3    (?,?,?)     (False, False, False)
wtensor4     int16       4    (?,?,?,?)   (False, False, False, False)
iscalar      int32       0    ()          ()
ivector      int32       1    (?,)        (False,)
irow         int32       2    (1,?)       (True, False)
icol         int32       2    (?,1)       (False, True)
imatrix      int32       2    (?,?)       (False, False)
itensor3     int32       3    (?,?,?)     (False, False, False)
itensor4     int32       4    (?,?,?,?)   (False, False, False, False)
lscalar      int64       0    ()          ()
lvector      int64       1    (?,)        (False,)
lrow         int64       2    (1,?)       (True, False)
lcol         int64       2    (?,1)       (False, True)
lmatrix      int64       2    (?,?)       (False, False)
ltensor3     int64       3    (?,?,?)     (False, False, False)
ltensor4     int64       4    (?,?,?,?)   (False, False, False, False)
dscalar      float64     0    ()          ()
dvector      float64     1    (?,)        (False,)
drow         float64     2    (1,?)       (True, False)
dcol         float64     2    (?,1)       (False, True)
dmatrix      float64     2    (?,?)       (False, False)
dtensor3     float64     3    (?,?,?)     (False, False, False)
dtensor4     float64     4    (?,?,?,?)   (False, False, False, False)
fscalar      float32     0    ()          ()
fvector      float32     1    (?,)        (False,)
frow         float32     2    (1,?)       (True, False)
fcol         float32     2    (?,1)       (False, True)
fmatrix      float32     2    (?,?)       (False, False)
ftensor3     float32     3    (?,?,?)     (False, False, False)
ftensor4     float32     4    (?,?,?,?)   (False, False, False, False)
cscalar      complex64   0    ()          ()
cvector      complex64   1    (?,)        (False,)
crow         complex64   2    (1,?)       (True, False)
ccol         complex64   2    (?,1)       (False, True)
cmatrix      complex64   2    (?,?)       (False, False)
ctensor3     complex64   3    (?,?,?)     (False, False, False)
ctensor4     complex64   4    (?,?,?,?)   (False, False, False, False)
zscalar      complex128  0    ()          ()
zvector      complex128  1    (?,)        (False,)
zrow         complex128  2    (1,?)       (True, False)
zcol         complex128  2    (?,1)       (False, True)
zmatrix      complex128  2    (?,?)       (False, False)
ztensor3     complex128  3    (?,?,?)     (False, False, False)
ztensor4     complex128  4    (?,?,?,?)   (False, False, False, False)
============ =========== ==== =========== =================================

Plural Constructors
--------------------------

There are several constructors that can produce multiple variables at once.
These are not frequently used in practice, but often used in tutorial examples to save space!

.. function:: iscalars, lscalars, fscalars, dscalars

    Return one or more scalar variables.

.. function:: ivectors, lvectors, fvectors, dvectors

    Return one or more vector variables.

.. function:: irows, lrows, frows, drows

    Return one or more row variables.

.. function:: icols, lcols, fcols, dcols

    Return one or more col variables.

.. function:: imatrices, lmatrices, fmatrices, dmatrices

    Return one or more matrix variables.

Each of these plural constructors accepts
an integer or several strings. If an integer is provided, the method
will return that many Variables and if strings are provided, it will
create one Variable for each string, using the string as the Variable's
name. For example:

.. testcode:: constructors

   from theano.tensor import *

   x, y, z = dmatrices(3) # creates three matrix Variables with no names
   x, y, z = dmatrices('x', 'y', 'z') # creates three matrix Variables named 'x', 'y' and 'z'


Custom tensor types
-------------------

If you would like to construct a tensor variable with a non-standard
broadcasting pattern, or a larger number of dimensions you'll need to create
your own :class:`TensorType` instance.  You create such an instance by passing
the dtype and broadcasting pattern to the constructor.  For example, you
can create your own 5-dimensional tensor type

>>> dtensor5 = TensorType('float64', (False,)*5)
>>> x = dtensor5()
>>> z = dtensor5('z')

You can also redefine some of the provided types and they will interact
correctly:

>>> my_dmatrix = TensorType('float64', (False,)*2)
>>> x = my_dmatrix()       # allocate a matrix variable
>>> my_dmatrix == dmatrix
True

See :class:`TensorType` for more information about creating new types of
Tensor.


Converting from Python Objects
-------------------------------

Another way of creating a TensorVariable (a TensorSharedVariable to be
precise) is by calling :func:`shared()`

.. testcode::

    x = shared(numpy.random.randn(3,4))

This will return a :term:`shared variable <shared variable>` whose ``.value`` is
a numpy ndarray.  The number of dimensions and dtype of the Variable are
inferred from the ndarray argument.  The argument to `shared` *will not be
copied*, and subsequent changes will be reflected in ``x.value``.

For additional information, see the :func:`shared() <shared.shared>` documentation.

.. _libdoc_tensor_autocasting:

Finally, when you use a numpy ndarry or a Python number together with
:class:`TensorVariable` instances in arithmetic expressions, the result is a
:class:`TensorVariable`. What happens to the ndarray or the number?
Theano requires that the inputs to all expressions be Variable instances, so
Theano automatically wraps them in a :class:`TensorConstant`.

.. note::

    Theano makes a copy of any ndarray that you use in an expression, so
    subsequent
    changes to that ndarray will not have any effect on the Theano expression.

For numpy ndarrays the dtype is given, but the broadcastable pattern must be
inferred.  The TensorConstant is given a type with a matching dtype,
and a broadcastable pattern with a ``True`` for every shape dimension that is 1.

For python numbers, the broadcastable pattern is ``()`` but the dtype must be
inferred.  Python integers are stored in the smallest dtype that can hold
them, so small constants like ``1`` are stored in a ``bscalar``.
Likewise, Python floats are stored in an fscalar if fscalar suffices to hold
them perfectly, but a dscalar otherwise.

.. note::

    When config.floatX==float32 (see :mod:`config`), then Python floats
    are stored instead as single-precision floats.

    For fine control of this rounding policy, see
    theano.tensor.basic.autocast_float.

.. function:: as_tensor_variable(x, name=None, ndim=None)

    Turn an argument `x` into a TensorVariable or TensorConstant.

    Many tensor Ops run their arguments through this function as
    pre-processing.  It passes through TensorVariable instances, and tries to
    wrap other objects into TensorConstant.

    When `x` is a Python number, the dtype is inferred as described above.

    When `x` is a `list` or `tuple` it is passed through numpy.asarray

    If the `ndim` argument is not None, it must be an integer and the output
    will be broadcasted if necessary in order to have this many dimensions.

    :rtype: :class:`TensorVariable` or :class:`TensorConstant`


TensorType and TensorVariable
=============================

.. class:: TensorType(Type)

    The Type class used to mark Variables that stand for `numpy.ndarray`
    values (`numpy.memmap`, which is a subclass of `numpy.ndarray`, is also allowed).
    Recalling to the tutorial, the purple box in
    :ref:`the tutorial's graph-structure figure <tutorial-graphfigure>` is an instance of this class.

    .. attribute:: broadcastable

        A tuple of True/False values, one for each dimension.  True in
        position 'i' indicates that at evaluation-time, the ndarray will have
        size 1 in that 'i'-th dimension.  Such a dimension is called a
        *broadcastable dimension* (see :ref:`libdoc_tensor_broadcastable`).

        The broadcastable pattern indicates both the number of dimensions and
        whether a particular dimension must have length 1.

        Here is a table mapping some `broadcastable` patterns to what they
        mean:

        ===================== =================================
        pattern               interpretation
        ===================== =================================
        []                    scalar
        [True]                1D scalar (vector of length 1)
        [True, True]          2D scalar (1x1 matrix)
        [False]               vector
        [False, False]        matrix
        [False] * n           nD tensor
        [True, False]         row (1xN matrix)
        [False, True]         column (Mx1 matrix)
        [False, True, False]  A Mx1xP tensor (a)
        [True, False, False]  A 1xNxP tensor (b)
        [False, False, False] A MxNxP tensor (pattern of a + b)
        ===================== =================================

        For dimensions in which broadcasting is False, the length of this
        dimension can be 1 or more.  For dimensions in which broadcasting is True,
        the length of this dimension must be 1.

        When two arguments to an element-wise operation (like addition or
        subtraction) have a different
        number of dimensions, the broadcastable
        pattern is *expanded to the left*, by padding with ``True``. For example,
        a vector's pattern, ``[False]``, could be expanded to ``[True, False]``, and
        would behave like a row (1xN matrix). In the same way, a matrix (``[False,
        False]``) would behave like a 1xNxP tensor (``[True, False, False]``).

        If we wanted to create a type representing a matrix that would
        broadcast over the middle dimension of a 3-dimensional tensor when
        adding them together, we would define it like this:

        >>> middle_broadcaster = TensorType('complex64', [False, True, False])

    .. attribute:: ndim

        The number of dimensions that a Variable's value will have at
        evaluation-time.  This must be known when we are building the
        expression graph.

    .. attribute:: dtype

        A string indicating
        the numerical type of the ndarray for which a Variable of this Type
        is standing.

        .. _dtype_list:

        The dtype attribute of a TensorType instance can be any of the
        following strings.

        ================= =================== =================
        dtype             domain              bits
        ================= =================== =================
        ``'int8'``        signed integer      8
        ``'int16'``       signed integer      16
        ``'int32'``       signed integer      32
        ``'int64'``       signed integer      64
        ``'uint8'``       unsigned integer    8
        ``'uint16'``      unsigned integer    16
        ``'uint32'``      unsigned integer    32
        ``'uint64'``      unsigned integer    64
        ``'float32'``     floating point      32
        ``'float64'``     floating point      64
        ``'complex64'``   complex             64 (two float32)
        ``'complex128'``  complex             128 (two float64)
        ================= =================== =================

    .. method:: __init__(self, dtype, broadcastable)

        If you wish to use a type of tensor which is not already available
        (for example, a 5D tensor) you can build an appropriate type by instantiating
        :class:`TensorType`.


TensorVariable
----------------

.. class:: TensorVariable(Variable, _tensor_py_operators)

    The result of symbolic operations typically have this type.

    See :class:`_tensor_py_operators` for most of the attributes and methods
    you'll want to call.

.. class:: TensorConstant(Variable, _tensor_py_operators)

    Python and numpy numbers are wrapped in this type.

    See :class:`_tensor_py_operators` for most of the attributes and methods
    you'll want to call.

.. class:: TensorSharedVariable(Variable, _tensor_py_operators)

    This type is returned by :func:`shared` when the value to share is a numpy
    ndarray.

    See :class:`_tensor_py_operators` for most of the attributes and methods
    you'll want to call.


.. class:: _tensor_py_operators(object)

    This mix-in class adds convenient attributes, methods, and support
    to TensorVariable, TensorConstant and TensorSharedVariable for
    Python operators (see :ref:`tensor_operator_support`).

    .. attribute:: type

        A reference to the :class:`TensorType` instance describing the sort of
        values that might be associated with this variable.

    .. attribute:: ndim

        The number of dimensions of this tensor.  Aliased to
        :attr:`TensorType.ndim`.

    .. attribute:: dtype

        The numeric type of this tensor. Aliased to
        :attr:`TensorType.dtype`.

    .. method:: reshape(shape, ndim=None)

        Returns a view of this tensor that has been reshaped as in
        numpy.reshape.  If the shape is a Variable argument, then you might
        need to use the optional `ndim` parameter to declare how many elements
        the shape has, and therefore how many dimensions the reshaped Variable
        will have.

        See :func:`reshape`.

    .. method:: dimshuffle(*pattern)

        Returns a view of this tensor with permuted dimensions.  Typically the
        pattern will include the integers 0, 1, ... ndim-1, and any number of
        'x' characters in dimensions where this tensor should be broadcasted.

        A few examples of patterns and their effect:

            * ('x') -> make a 0d (scalar) into a 1d vector
            * (0, 1) -> identity for 2d vectors
            * (1, 0) -> inverts the first and second dimensions
            * ('x', 0) -> make a row out of a 1d vector (N to 1xN)
            * (0, 'x') -> make a column out of a 1d vector (N to Nx1)
            * (2, 0, 1) -> AxBxC to CxAxB
            * (0, 'x', 1) -> AxB to Ax1xB
            * (1, 'x', 0) -> AxB to Bx1xA

    .. method:: flatten(ndim=1)

        Returns a view of this tensor with `ndim` dimensions, whose shape for the first
        `ndim-1` dimensions will be the same as `self`, and shape in the
        remaining dimension will be expanded to fit in all the data from self.

        See :func:`flatten`.

    .. method:: ravel()

        return self.flatten(). For NumPy compatibility.

    .. attribute:: T

        Transpose of this tensor.

        >>> x = T.zmatrix()
        >>> y = 3+.2j * x.T

        .. note::

            In numpy and in Theano, the transpose of a vector is exactly the
            same vector!  Use `reshape` or `dimshuffle` to turn your vector
            into a row or column matrix.

    .. method:: {any,all}(axis=None, keepdims=False)
    .. method:: {sum,prod,mean}(axis=None, dtype=None, keepdims=False, acc_dtype=None)
    .. method:: {var,std,min,max,argmin,argmax}(axis=None, keepdims=False),
    .. method:: diagonal(offset=0, axis1=0, axis2=1)
    .. method:: astype(dtype)
    .. method:: take(indices, axis=None, mode='raise')
    .. method:: copy()
    .. method:: norm(L, axis=None)
    .. method:: nonzero(self, return_matrix=False)
    .. method:: nonzero_values(self)
    .. method:: sort(self, axis=-1, kind='quicksort', order=None)
    .. method:: argsort(self, axis=-1, kind='quicksort', order=None)
    .. method:: clip(self, a_min, a_max)
    .. method:: conf()
    .. method:: repeat(repeats, axis=None)
    .. method:: round(mode="half_away_from_zero")
    .. method:: trace()
    .. method:: get_scalar_constant_value()
    .. method:: zeros_like(model, dtype=None)

       All the above methods are equivalent to NumPy for Theano on the current tensor.

    .. method:: __{abs,neg,lt,le,gt,ge,invert,and,or,add,sub,mul,div,truediv,floordiv}__

       Those elemwise operation are supported via Python syntax.

Shaping and Shuffling
=====================

To re-order the dimensions of a variable, to insert or remove broadcastable
dimensions, see :meth:`_tensor_py_operators.dimshuffle`.

.. function:: shape(x)

    Returns an lvector representing the shape of `x`.

.. function:: reshape(x, newshape, ndim=None)

    :type x: any TensorVariable (or compatible)
    :param x: variable to be reshaped

    :type newshape: lvector (or compatible)
    :param newshape: the new shape for `x`

    :param ndim: optional - the length that `newshape`'s value will have.
        If this is ``None``, then `reshape()` will infer it from `newshape`.

    :rtype: variable with x's dtype, but ndim dimensions

    .. note::

        This function can infer the length of a symbolic newshape in some
        cases, but if it cannot and you do not provide the `ndim`, then this
        function will raise an Exception.


.. function:: shape_padleft(x, n_ones=1)

    Reshape `x` by left padding the shape with `n_ones` 1s. Note that all
    this new dimension will be broadcastable. To make them non-broadcastable
    see the :func:`unbroadcast`.

    :param x: variable to be reshaped
    :type x: any TensorVariable (or compatible)

    :type n_ones: int
    :type n_ones: number of dimension to be added to `x`



.. function:: shape_padright(x, n_ones=1)

    Reshape `x` by right padding the shape with `n_ones` 1s. Note that all
    this new dimension will be broadcastable. To make them non-broadcastable
    see the :func:`unbroadcast`.

    :param x: variable to be reshaped
    :type x: any TensorVariable (or compatible)

    :type n_ones: int
    :type n_ones: number of dimension to be added to `x`

.. function:: unbroadcast(x, *axes)

    Make `x` impossible to broadcast in the specified axes `axes`. For
    example, `unbroadcast(x, 0)` will make the first dimension of `x`
    unbroadcastable.

.. function:: addbroadcast(x, *axes)

    Make `x` broadcastable in the specified axes `axes`. For
    example, `addbroadcast(x, 0)` will make the first dimension of `x`
    broadcastable. When performing the function, if the length of `x`
    along that dimension is not 1, a ``ValueError`` will be raised.

.. function:: patternbroadcast(x, broadcastable)

    Change `x` broadcastable pattern to
    `broadcastable`. `broadcastable` must be iterable. For example,
    `patternbroadcast(x, (True, False))` will make the first dimension
    of `x` broadcastable and the second dimension not broadcastable,
    so x will now be a `row`.

.. function:: flatten(x, outdim=1)

    Similar to :func:`reshape`, but the shape is inferred from the shape of `x`.

    :param x: variable to be flattened
    :type x: any TensorVariable (or compatible)

    :type outdim: int
    :param outdim: the number of dimensions in the returned variable

    :rtype: variable with same dtype as `x` and `outdim` dimensions
    :returns: variable with the same shape as `x` in the leading `outdim-1`
        dimensions, but with all remaining dimensions of `x` collapsed into
        the last dimension.

    For example, if we flatten a tensor of shape (2, 3, 4, 5) with flatten(x,
    outdim=2), then we'll have the same (2-1=1) leading dimensions (2,), and the
    remaining dimensions are collapsed.  So the output in this example would
    have shape (2, 60).


.. function:: tile(x, reps, ndim=None)

    Construct an array by repeating the input `x` according to `reps`
    pattern.

    Tiles its input according to `reps`. The length of `reps` is the
    number of dimension of `x` and contains the number of times to
    tile `x` in each dimension.

    :see: `numpy.tile
        <http://docs.scipy.org/doc/numpy/reference/generated/numpy.tile.html>`_
        documentation for examples.

    :see: :func:`theano.tensor.extra_ops.repeat
        <theano.tensor.extra_ops.repeat>`

    :note: Currently, `reps` must be a constant, `x.ndim` and
        `len(reps)` must be equal and, if specified, `ndim` must be
        equal to both.


Creating Tensor
===============


.. function:: zeros_like(x)

    :param x: tensor that has same shape as output

    Returns a tensor filled with 0s that has same shape as `x`.

.. function:: ones_like(x)

    :param x: tensor that has same shape as output

    Returns a tensor filled with 1s that has same shape as `x`.

.. function:: fill(a,b)

    :param a: tensor that has same shape as output
    :param b: theano scalar or value with which you want to fill the output

    Create a matrix by filling the shape of `a` with `b`

.. function:: alloc(value, *shape)

    :param value: a value with which to fill the output
    :param shape: the dimensions of the returned array
    :returns: an N-dimensional tensor initialized by `value` and having the specified shape.

.. function:: eye(n, m=None, k=0, dtype=theano.config.floatX)

    :param n: number of rows in output (value or theano scalar)
    :param m: number of columns in output (value or theano scalar)
    :param k: Index of the diagonal: 0 refers to the main diagonal,
              a positive value refers to an upper diagonal, and a
              negative value to a lower diagonal. It can be a theano
              scalar.
    :returns: An array where all elements are equal to zero, except for the `k`-th
              diagonal, whose values are equal to one.

.. function:: identity_like(x)

    :param x: tensor
    :returns: A tensor of same shape as `x` that is filled with 0s everywhere
              except for the main diagonal, whose values are equal to one. The output
              will have same dtype as `x`.

.. function:: stack(*tensors)

    Return a Tensor representing for the arguments all stacked up into a single Tensor.
    (of 1 rank greater).

    :param tensors: one or more tensors of the same rank
    :returns: A tensor such that rval[0] == tensors[0], rval[1] == tensors[1], etc.

    >>> x0 = T.scalar()
    >>> x1 = T.scalar()
    >>> x2 = T.scalar()
    >>> x = T.stack(x0, x1, x2)
    >>> x.ndim # x is a vector of length 3.
    1

.. function:: concatenate(tensor_list, axis=0)

    :type tensor_list: a list or tuple of Tensors that all have the same shape in the axes
                        *not* specified by the `axis` argument.
    :param tensor_list: one or more Tensors to be concatenated together into one.
    :type axis: literal or symbolic integer
    :param axis: Tensors will be joined along this axis, so they may have different
        ``shape[axis]``

    >>> x0 = T.fmatrix()
    >>> x1 = T.ftensor3()
    >>> x2 = T.fvector()
    >>> x = T.concatenate([x0, x1[0], T.shape_padright(x2)], axis=1)
    >>> x.ndim
    2

.. function:: stacklists(tensor_list)

    :type tensor_list: an iterable that contains either tensors or other
        iterables of the same type as `tensor_list` (in other words, this
        is a tree whose leaves are tensors).
    :param tensor_list: tensors to be stacked together.

    Recursively stack lists of tensors to maintain similar structure.

    This function can create a tensor from a shaped list of scalars:

    >>> from theano.tensor import stacklists, scalars, matrices
    >>> from theano import function
    >>> a, b, c, d = scalars('abcd')
    >>> X = stacklists([[a, b], [c, d]])
    >>> f = function([a, b, c, d], X)
    >>> f(1, 2, 3, 4)
    array([[ 1.,  2.],
           [ 3.,  4.]])

    We can also stack arbitrarily shaped tensors. Here we stack matrices into
    a 2 by 2 grid:

    >>> from numpy import ones
    >>> a, b, c, d = matrices('abcd')
    >>> X = stacklists([[a, b], [c, d]])
    >>> f = function([a, b, c, d], X)
    >>> x = ones((4, 4), 'float32')
    >>> f(x, x, x, x).shape
    (2, 2, 4, 4)

Reductions
==========


.. function:: max(x, axis=None, keepdims=False)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to compute the maximum
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: maximum of *x* along *axis*

    axis can be:
     * *None* - in which case the maximum is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: argmax(x, axis=None, keepdims=False)

    :Parameter: *x* - symbolic Tensor (or compatible)
    :Parameter: *axis* - axis along which to compute the index of the maximum
    :Parameter: *keepdims* - (boolean) If this is set to True, the axis which is reduced is
		left in the result as a dimension with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: the index of the maximum value along a given axis

    if axis=None, Theano 0.5rc1 or later: argmax over the flattened tensor (like numpy)
                  older: then axis is assumed to be ndim(x)-1

.. function:: max_and_argmax(x, axis=None, keepdims=False)

    :Parameter: *x* - symbolic Tensor (or compatible)
    :Parameter: *axis* - axis along which to compute the maximum and its index
    :Parameter: *keepdims* - (boolean) If this is set to True, the axis which is reduced is
		left in the result as a dimension with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: the maxium value along a given axis and its index.

    if axis=None, Theano 0.5rc1 or later: max_and_argmax over the flattened tensor (like numpy)
                  older: then axis is assumed to be ndim(x)-1

.. function:: min(x, axis=None, keepdims=False)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to compute the minimum
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: minimum of *x* along *axis*

    axis can be:
     * *None* - in which case the minimum is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: argmin(x, axis=None, keepdims=False)

    :Parameter: *x* - symbolic Tensor (or compatible)
    :Parameter: *axis* - axis along which to compute the index of the minimum
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: the index of the minimum value along a given axis

    if axis=None, Theano 0.5rc1 or later: argmin over the flattened tensor (like numpy)
                  older: then axis is assumed to be ndim(x)-1

.. function:: sum(x, axis=None, dtype=None, keepdims=False, acc_dtype=None)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to compute the sum
    :Parameter: *dtype* - The dtype of the returned tensor.
        If None, then we use the default dtype which is the same as
        the input tensor's dtype except when:

        - the input dtype is a signed integer of precision < 64 bit, in
          which case we use int64
        - the input dtype is an unsigned integer of precision < 64 bit, in
          which case we use uint64

        This default dtype does _not_ depend on the value of "acc_dtype".

    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.

    :Parameter: *acc_dtype* -  The dtype of the internal accumulator.
        If None (default), we use the dtype in the list below,
        or the input dtype if its precision is higher:

        - for int dtypes, we use at least int64;
        - for uint dtypes, we use at least uint64;
        - for float dtypes, we use at least float64;
        - for complex dtypes, we use at least complex128.

    :Returns: sum of *x* along *axis*

    axis can be:
     * *None* - in which case the sum is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: prod(x, axis=None, dtype=None, keepdims=False, acc_dtype=None, no_zeros_in_input=False)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to compute the product
    :Parameter: *dtype* - The dtype of the returned tensor.
        If None, then we use the default dtype which is the same as
        the input tensor's dtype except when:

        - the input dtype is a signed integer of precision < 64 bit, in
          which case we use int64
        - the input dtype is an unsigned integer of precision < 64 bit, in
          which case we use uint64

        This default dtype does _not_ depend on the value of "acc_dtype".

    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.

    :Parameter: *acc_dtype* -  The dtype of the internal accumulator.
        If None (default), we use the dtype in the list below,
        or the input dtype if its precision is higher:

        - for int dtypes, we use at least int64;
        - for uint dtypes, we use at least uint64;
        - for float dtypes, we use at least float64;
        - for complex dtypes, we use at least complex128.

    :Parameter: *no_zeros_in_input* - The grad of prod is complicated
         as we need to handle 3 different cases: without zeros in the
         input reduced group, with 1 zero or with more zeros.

	 This could slow you down, but more importantly, we currently
	 don't support the second derivative of the 3 cases. So you
	 cannot take the second derivative of the default prod().

	 To remove the handling of the special cases of 0 and so get
	 some small speed up and allow second derivative set
	 ``no_zeros_in_inputs`` to ``True``. It defaults to ``False``.

	 **It is the user responsibility to make sure there are no zeros
	 in the inputs. If there are, the grad will be wrong.**

    :Returns: product of every term in *x* along *axis*

    axis can be:
     * *None* - in which case the sum is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: mean(x, axis=None, dtype=None, keepdims=False, acc_dtype=None)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to compute the mean
    :Parameter: *dtype* - The dtype to cast the result of the inner summation into.
        For instance, by default, a sum of a float32 tensor will be
        done in float64 (acc_dtype would be float64 by default),
        but that result will be casted back in float32.
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Parameter: *acc_dtype* -  The dtype of the internal accumulator of the
        inner summation. This will not necessarily be the dtype of the
        output (in particular if it is a discrete (int/uint) dtype, the
        output will be in a float type).  If None, then we use the same
        rules as :func:`sum()`.
    :Returns: mean value of *x* along *axis*

    axis can be:
     * *None* - in which case the mean is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: var(x, axis=None, keepdims=False)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to compute the variance
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: variance of *x* along *axis*

    axis can be:
     * *None* - in which case the variance is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: std(x, axis=None, keepdims=False)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to compute the standard deviation
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: variance of *x* along *axis*

    axis can be:
     * *None* - in which case the standard deviation is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: all(x, axis=None, keepdims=False)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to apply 'bitwise and'
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: bitwise and of *x* along *axis*

    axis can be:
     * *None* - in which case the 'bitwise and' is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: any(x, axis=None, keepdims=False)

    :Parameter: *x* -  symbolic Tensor (or compatible)
    :Parameter: *axis* - axis or axes along which to apply bitwise or
    :Parameter: *keepdims* - (boolean) If this is set to True, the axes which are reduced are
		left in the result as dimensions with size one. With this option, the result
		will broadcast correctly against the original tensor.
    :Returns: bitwise or of *x* along *axis*

    axis can be:
     * *None* - in which case the 'bitwise or' is computed along all axes (like numpy)
     * an *int* - computed along this axis
     * a *list of ints* - computed along these axes

.. function:: ptp(x, axis = None)

    Range of values (maximum - minimum) along an axis.
    The name of the function comes from the acronym for peak to peak.

    :Parameter: *x* Input tensor.

    :Parameter: *axis* Axis along which to find the peaks. By default,
                flatten the array.

    :Returns: A new array holding the result.


Indexing
========

Like NumPy, Theano distinguishes between *basic* and *advanced* indexing.
Theano fully supports basic indexing
(see `NumPy's indexing  <http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html>`_).

`Integer advanced indexing
<http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#integer>`_
will be supported in 0.6rc4 (or the development version). We do not
support boolean masks, as Theano does not have a boolean type (we use
int8 for the output of logic operators).

.. testsetup:: indexing

   import theano
   import numpy as np

NumPy with a mask:

.. doctest:: indexing

   >>> n = np.arange(9).reshape(3,3)
   >>> n[n > 4]
   array([5, 6, 7, 8])

Theano indexing with a "mask" (incorrect approach):

.. doctest:: indexing

   >>> t = theano.tensor.arange(9).reshape((3,3))
   >>> t[t > 4].eval()  # an array with shape (3, 3, 3)
   array([[[0, 1, 2],
           [0, 1, 2],
           [0, 1, 2]],
   <BLANKLINE>
          [[0, 1, 2],
           [0, 1, 2],
           [3, 4, 5]],
   <BLANKLINE>
          [[3, 4, 5],
           [3, 4, 5],
           [3, 4, 5]]], dtype=int8)

Getting a Theano result like NumPy:

.. doctest:: indexing

   >>> t[(t > 4).nonzero()].eval()
   array([5, 6, 7, 8], dtype=int8)

The gradient of Advanced indexing needs in many cases NumPy
1.8. It is not released yet as of April 30th, 2013. You can use NumPy
development version to have this feature now.


Index-assignment is *not* supported.  If you want to do something like ``a[5]
= b`` or ``a[5]+=b``, see :func:`theano.tensor.set_subtensor` and :func:`theano.tensor.inc_subtensor` below.

.. autofunction:: theano.tensor.set_subtensor

.. autofunction:: theano.tensor.inc_subtensor

.. _tensor_operator_support:

Operator Support
================

Many Python operators are supported.

>>> a, b = T.itensor3(), T.itensor3() # example inputs

Arithmetic
--------------

.. doctest::
   :options: +SKIP

   >>> a + 3      # T.add(a, 3) -> itensor3
   >>> 3 - a      # T.sub(3, a)
   >>> a * 3.5    # T.mul(a, 3.5) -> ftensor3 or dtensor3 (depending on casting)
   >>> 2.2 / a    # T.truediv(2.2, a)
   >>> 2.2 // a   # T.intdiv(2.2, a)
   >>> 2.2**a     # T.pow(2.2, a)
   >>> b % a      # T.mod(b, a)

Bitwise
-------------

.. doctest::
   :options: +SKIP

   >>> a & b      # T.and_(a,b)    bitwise and (alias T.bitwise_and)
   >>> a ^ 1      # T.xor(a,1)     bitwise xor (alias T.bitwise_xor)
   >>> a | b      # T.or_(a,b)     bitwise or (alias T.bitwise_or)
   >>> ~a         # T.invert(a)    bitwise invert (alias T.bitwise_not)

Inplace
-------------

In-place operators are *not* supported.  Theano's graph-optimizations
will determine which intermediate values to use for in-place
computations.  If you would like to update the value of a
:term:`shared variable`, consider using the ``updates`` argument to
:func:`theano.function`.

.. _libdoc_tensor_elementwise:

Elementwise
===========

Casting
-------

.. function:: cast(x, dtype)

    Cast any tensor `x` to a Tensor of the same shape, but with a different
    numerical type `dtype`.

    This is not a reinterpret cast, but a coersion cast, similar to
    ``numpy.asarray(x, dtype=dtype)``.

    .. testcode:: cast

        import theano.tensor as T
        x = T.matrix()
        x_as_int = T.cast(x, 'int32')

    Attempting to casting a complex value to a real value is ambiguous and
    will raise an exception.  Use `real()`, `imag()`, `abs()`, or `angle()`.

.. function:: real(x)

    Return the real (not imaginary) components of Tensor x.
    For non-complex `x` this function returns x.

.. function:: imag(x)

    Return the imaginary components of Tensor x.
    For non-complex `x` this function returns zeros_like(x).


Comparisons
------------

The six usual equality and inequality operators share the same interface.
  :Parameter:  *a* - symbolic Tensor (or compatible)
  :Parameter:  *b* - symbolic Tensor (or compatible)
  :Return type: symbolic Tensor
  :Returns: a symbolic tensor representing the application of the logical elementwise operator.

  .. note::

    Theano has no boolean dtype.  Instead, all boolean tensors are represented
    in ``'int8'``.

  Here is an example with the less-than operator.

  .. testcode:: oper

    import theano.tensor as T
    x,y = T.dmatrices('x','y')
    z = T.le(x,y)

.. function:: lt(a, b)

    Returns a symbolic ``'int8'`` tensor representing the result of logical less-than (a<b).

    Also available using syntax ``a < b``

.. function:: gt(a, b)

    Returns a symbolic ``'int8'`` tensor representing the result of logical greater-than (a>b).

    Also available using syntax ``a > b``

.. function:: le(a, b)

    Returns a variable representing the result of logical less than or equal (a<=b).

    Also available using syntax ``a <= b``

.. function:: ge(a, b)

    Returns a variable representing the result of logical greater or equal than (a>=b).

    Also available using syntax ``a >= b``

.. function:: eq(a, b)

    Returns a variable representing the result of logical equality (a==b).

.. function:: neq(a, b)

    Returns a variable representing the result of logical inequality (a!=b).

.. function:: isnan(a)

    Returns a variable representing the comparison of ``a`` elements with nan.

    This is equivalent to ``numpy.isnan``.

.. function:: isinf(a)

    Returns a variable representing the comparison of ``a`` elements
    with inf or -inf.

    This is equivalent to ``numpy.isinf``.

Condition
---------

.. function:: switch(cond, ift, iff)

    Returns a variable representing a switch between ift (iftrue) and iff (iffalse)
     based on the condition cond. This is the theano equivalent of numpy.where.

      :Parameter:  *cond* - symbolic Tensor (or compatible)
      :Parameter:  *ift* - symbolic Tensor (or compatible)
      :Parameter:  *iff* - symbolic Tensor (or compatible)
      :Return type: symbolic Tensor

    .. testcode:: switch

      import theano.tensor as T
      a,b = T.dmatrices('a','b')
      x,y = T.dmatrices('x','y')
      z = T.switch(T.lt(a,b), x, y)

.. function:: where(cond, ift, iff)

   Alias for `switch`. where is the numpy name.

.. function:: clip(x, min, max)

    Return a variable representing x, but with all elements greater than
    `max` clipped to `max` and all elements less than `min` clipped to `min`.

    Normal broadcasting rules apply to each of `x`, `min`, and `max`.

Bit-wise
--------


The bitwise operators possess this interface:
    :Parameter:  *a* - symbolic Tensor of integer type.
    :Parameter:  *b* - symbolic Tensor of integer type.

    .. note::

        The bitwise operators must have an integer type as input.

        The bit-wise not (invert) takes only one parameter.

    :Return type: symbolic Tensor with corresponding dtype.

.. function:: and_(a, b)

    Returns a variable representing the result of the bitwise and.

.. function:: or_(a, b)

    Returns a variable representing the result of the bitwise or.

.. function:: xor(a, b)

    Returns a variable representing the result of the bitwise xor.

.. function:: invert(a)

    Returns a variable representing the result of the bitwise not.

.. function:: bitwise_and(a, b)

   Alias for `and_`. bitwise_and is the numpy name.

.. function:: bitwise_or(a, b)

   Alias for `or_`. bitwise_or is the numpy name.

.. function:: bitwise_xor(a, b)

   Alias for `xor_`. bitwise_xor is the numpy name.

.. function:: bitwise_not(a, b)

   Alias for invert. invert is the numpy name.

Here is an example using the bit-wise ``and_`` via the ``&`` operator:

.. testcode:: bitwise

    import theano.tensor as T
    x,y = T.imatrices('x','y')
    z = x & y


Mathematical
------------

.. function:: abs_(a)

    Returns a variable representingthe absolute of a, ie ``|a|``.

    .. note:: Can also be accessed with ``abs(a)``.

.. function:: angle(a)

    Returns a variable representing angular component of complex-valued Tensor `a`.

.. function:: exp(a)

    Returns a variable representing the exponential of a, ie e^a.

.. function:: maximum(a, b)

   Returns a variable representing the maximum element by element of a and b

.. function:: minimum(a, b)

   Returns a variable representing the minimum element by element of a and b

.. function:: neg(a)

    Returns a variable representing the negation of `a` (also ``-a``).

.. function:: inv(a)

    Returns a variable representing the inverse of a, ie 1.0/a. Also called reciprocal.

.. function:: log(a), log2(a), log10(a)

    Returns a variable representing the base e, 2 or 10 logarithm of a.

.. function:: sgn(a)

    Returns a variable representing the sign of a.

.. function:: ceil(a)

    Returns a variable representing the ceiling of a (for example ceil(2.1) is 3).

.. function:: floor(a)

    Returns a variable representing the floor of a (for example floor(2.9) is 2).

.. function:: round(a, mode="half_away_from_zero")

    Returns a variable representing the rounding of a in the same dtype as a. Implemented rounding mode are half_away_from_zero and half_to_even.

.. function:: iround(a, mode="half_away_from_zero")

    Short hand for cast(round(a, mode),'int64').

.. function:: sqr(a)

    Returns a variable representing the square of a, ie a^2.

.. function:: sqrt(a)

    Returns a variable representing the of a, ie a^0.5.

.. function:: cos(a), sin(a), tan(a)

    Returns a variable representing the trigonometric functions of a (cosine, sine and tangent).

.. function:: cosh(a), sinh(a), tanh(a)

    Returns a variable representing the hyperbolic trigonometric functions of a (hyperbolic cosine, sine and tangent).

.. function:: erf(a), erfc(a)

    Returns a variable representing the error function or the complementary error function. `wikipedia <http://en.wikipedia.org/wiki/Error_function>`__

.. function:: erfinv(a), erfcinv(a)

    Returns a variable representing the inverse error function or the inverse complementary error function. `wikipedia <http://en.wikipedia.org/wiki/Error_function#Inverse_functions>`__

.. function:: gamma(a)

   Returns a variable representing the gamma function.

.. function:: gammaln(a)

   Returns a variable representing the logarithm of the gamma function.

.. function:: psi(a)

   Returns a variable representing the derivative of the logarithm of
   the gamma function (also called the digamma function).

.. function:: chi2sf(a, df)

   Returns a variable representing the survival function (1-cdf —
   sometimes more accurate).

   C code is provided in the Theano_lgpl repository.
   This makes it faster.

   https://github.com/Theano/Theano_lgpl.git


.. _libdoc_tensor_broadcastable:

Broadcasting in Theano vs. Numpy
--------------------------------

Broadcasting is a mechanism which allows tensors with
different numbers of dimensions to be added or multiplied
together by (virtually) replicating the smaller tensor along
the dimensions that it is lacking.

Broadcasting is the mechanism by which a scalar
may be added to a matrix, a vector to a matrix or a scalar to
a vector.

.. figure:: bcast.png

Broadcasting a row matrix. T and F respectively stand for
True and False and indicate along which dimensions we allow
broadcasting.

If the second argument were a vector, its shape would be
``(2,)`` and its broadcastable pattern ``(F,)``. They would
be automatically expanded to the **left** to match the
dimensions of the matrix (adding ``1`` to the shape and ``T``
to the pattern), resulting in ``(1, 2)`` and ``(T, F)``.
It would then behave just like the example above.


Unlike numpy which does broadcasting dynamically, Theano needs
to know, for any operation which supports broadcasting, which
dimensions will need to be broadcasted. When applicable, this
information is given in the :ref:`type` of a *Variable*.

See also:

* `SciPy documentation about numpy's broadcasting <http://www.scipy.org/EricsBroadcastingDoc>`_

* `OnLamp article about numpy's broadcasting <http://www.onlamp.com/pub/a/python/2000/09/27/numerically.html>`_





Linear Algebra
==============

.. function:: dot(X, Y)

    :param X: left term
    :param Y: right term
    :type X: symbolic matrix or vector
    :type Y: symbolic matrix or vector
    :rtype: symbolic matrix or vector
    :return: the inner product of `X` and `Y`.

.. function:: outer(X, Y)

    :param X: left term
    :param Y: right term
    :type X: symbolic vector
    :type Y: symbolic vector
    :rtype: symbolic matrix

    :return: vector-vector outer product

.. function:: tensordot(a, b, axes=2)

    Given two tensors a and b,tensordot computes a generalized dot product over
    the provided axes. Theano's implementation reduces all expressions to
    matrix or vector dot products and is based on code from Tijmen Tieleman's
    gnumpy (http://www.cs.toronto.edu/~tijmen/gnumpy.html).

    :param a: the first tensor variable
    :type a: symbolic tensor

    :param b: the second tensor variable
    :type b: symbolic tensor

    :param axes: an integer or array. If an integer, the number of axes
                 to sum over. If an array, it must have two array
                 elements containing the axes to sum over in each tensor.

                 Note that the default value of 2 is not guaranteed to work
                 for all values of a and b, and an error will be raised if
                 that is the case. The reason for keeping the default is to
                 maintain the same signature as numpy's tensordot function
                 (and np.tensordot raises analogous errors for non-compatible
                 inputs).

                 If an integer i, it is converted to an array containing
                 the last i dimensions of the first tensor and the first
                 i dimensions of the second tensor:

                     axes = [range(a.ndim - i, b.ndim), range(i)]

                 If an array, its two elements must contain compatible axes
                 of the two tensors. For example, [[1, 2], [2, 0]] means sum
                 over the 2nd and 3rd axes of a and the 3rd and 1st axes of b.
                 (Remember axes are zero-indexed!) The 2nd axis of a and the
                 3rd axis of b must have the same shape; the same is true for
                 the 3rd axis of a and the 1st axis of b.
    :type axes: int or array-like of length 2

    :returns: a tensor with shape equal to the concatenation of a's shape
              (less any dimensions that were summed over) and b's shape
              (less any dimensions that were summed over).
    :rtype: symbolic tensor

    It may be helpful to consider an example to see what tensordot does.
    Theano's implementation is identical to NumPy's. Here a has shape (2, 3, 4)
    and b has shape (5, 6, 4, 3). The axes to sum over are [[1, 2], [3, 2]] --
    note that a.shape[1] == b.shape[3] and a.shape[2] == b.shape[2]; these axes
    are compatible. The resulting tensor will have shape (2, 5, 6) -- the
    dimensions that are not being summed:

    .. testcode:: tensordot

        import numpy as np

        a = np.random.random((2,3,4))
        b = np.random.random((5,6,4,3))

        #tensordot
        c = np.tensordot(a, b, [[1,2],[3,2]])

        #loop replicating tensordot
        a0, a1, a2 = a.shape
        b0, b1, _, _ = b.shape
        cloop = np.zeros((a0,b0,b1))

        #loop over non-summed indices -- these exist
        #in the tensor product.
        for i in range(a0):
            for j in range(b0):
                for k in range(b1):
                    #loop over summed indices -- these don't exist
                    #in the tensor product.
                    for l in range(a1):
                        for m in range(a2):
                            cloop[i,j,k] += a[i,l,m] * b[j,k,m,l]

        assert np.allclose(c, cloop)

    This specific implementation avoids a loop by transposing a and b such that
    the summed axes of a are last and the summed axes of b are first. The
    resulting arrays are reshaped to 2 dimensions (or left as vectors, if
    appropriate) and a matrix or vector dot product is taken. The result is
    reshaped back to the required output dimensions.

    In an extreme case, no axes may be specified. The resulting tensor
    will have shape equal to the concatenation of the shapes of a and b:

    .. doctest:: tensordot

        >>> c = np.tensordot(a, b, 0)
        >>> a.shape
        (2, 3, 4)
        >>> b.shape
        (5, 6, 4, 3)
        >>> print(c.shape)
        (2, 3, 4, 5, 6, 4, 3)

    :note: See the documentation of `numpy.tensordot <http://docs.scipy.org/doc/numpy/reference/generated/numpy.tensordot.html>`_ for more examples.

.. function:: batched_dot(X, Y)

    :param x: A Tensor with sizes e.g.: for  3D (dim1, dim3, dim2)
    :param y: A Tensor with sizes e.g.: for 3D (dim1, dim2, dim4)

    This function computes the dot product between the two tensors, by iterating
    over the first dimension using scan.
    Returns a tensor of size e.g. if it is 3D: (dim1, dim3, dim4)
    Example:

    >>> first = T.tensor3('first')
    >>> second = T.tensor3('second')
    >>> result = batched_dot(first, second)

    :note:  This is a subset of numpy.einsum, but we do not provide it for now.
        But numpy einsum is slower than dot or tensordot:
        http://mail.scipy.org/pipermail/numpy-discussion/2012-October/064259.html

    :param X: left term
    :param Y: right term
    :type X: symbolic tensor
    :type Y: symbolic tensor

    :return: tensor of products

.. function:: batched_tensordot(X, Y, axes=2)

    :param x: A Tensor with sizes e.g.: for 3D (dim1, dim3, dim2)
    :param y: A Tensor with sizes e.g.: for 3D (dim1, dim2, dim4)
    :param axes: an integer or array. If an integer, the number of axes
                 to sum over. If an array, it must have two array
                 elements containing the axes to sum over in each tensor.

                 If an integer i, it is converted to an array containing
                 the last i dimensions of the first tensor and the first
                 i dimensions of the second tensor (excluding the first
                 (batch) dimension)::

                     axes = [range(a.ndim - i, b.ndim), range(1,i+1)]

                 If an array, its two elements must contain compatible axes
                 of the two tensors. For example, [[1, 2], [2, 4]] means sum
                 over the 2nd and 3rd axes of a and the 3rd and 5th axes of b.
                 (Remember axes are zero-indexed!) The 2nd axis of a and the
                 3rd axis of b must have the same shape; the same is true for
                 the 3rd axis of a and the 5th axis of b.
    :type axes: int or array-like of length 2

    :returns: a tensor with shape equal to the concatenation of a's shape
              (less any dimensions that were summed over) and b's shape
              (less first dimension and any dimensions that were summed over).
    :rtype: tensor of tensordots

    A hybrid of batch_dot and tensordot, this function computes the
    tensordot product between the two tensors, by iterating over the
    first dimension using scan to perform a sequence of tensordots.

    :note: See :func:`tensordot` and :func:`batched_dot` for
        supplementary documentation.



Gradient / Differentiation
==========================

.. function:: grad(cost, wrt, g_cost=None, consider_constant=None, warn_type=False)

    Return symbolic gradients for one or more variables with respect to some
    cost.

    For more information about how automatic differentiation works in Theano,
    see :mod:`gradient`. For information on how to implement the gradient of
    a certain Op, see :func:`grad`.

    :type cost: 0-d tensor variable
    :type wrt: tensor variable or list of tensor variables
    :type g_cost: same as type of `cost`
    :type consider_constant: list of variables
    :type warn_type: bool

    :param cost: a scalar with respect to which we are differentiating
    :param wrt: term[s] for which we want gradients
    :param g_cost: the gradient on the cost
    :param consider_constant: variables whose gradients will be held at 0.
    :param warn_type: True will trigger warnings via the logging module when
       the gradient on an expression has a different type than the original
       expression

    :rtype: variable or list of variables (matching `wrt`)
    :returns: gradients of the cost with respect to each of the `wrt` terms

.. function:: subgraph_grad(wrt, end, start=None, cost=None, details=False)

    With respect to `wrt`, computes gradients of cost and/or from existing
    `start` gradients, up to the `end` variables of a symbolic digraph.
    In other words, computes gradients for a subgraph of the
    symbolic theano function. Ignores all disconnected inputs.

    This can be useful when one needs to perform the gradient descent
    iteratively (e.g. one layer at a time in an MLP), or when a particular
    operation is not differentiable in theano (e.g. stochastic sampling
    from a multinomial). In the latter case, the gradient of the
    non-differentiable process could be approximated by user-defined
    formula, which could be calculated using the gradients of a cost
    with respect to samples (0s and 1s). These gradients are obtained
    by performing a subgraph_grad from the `cost` or previously known gradients
    (`start`) up to the outputs of the stochastic process (`end`).
    A dictionary mapping gradients obtained from the user-defined
    differentiation of the process, to variables, could then be fed into
    another subgraph_grad as `start` with any other `cost` (e.g. weight decay).

    In an MLP, we could use subgraph_grad to iteratively backpropagate:

    .. testcode:: subgraph_grad

       import theano
       import numpy as np
       x, t = theano.tensor.fvector('x'), theano.tensor.fvector('t')
       w1 = theano.shared(np.random.randn(3,4))
       w2 = theano.shared(np.random.randn(4,2))
       a1 = theano.tensor.tanh(theano.tensor.dot(x,w1))
       a2 = theano.tensor.tanh(theano.tensor.dot(a1,w2))
       cost2 = theano.tensor.sqr(a2 - t).sum()
       cost2 += theano.tensor.sqr(w2.sum())
       cost1 = theano.tensor.sqr(w1.sum())

       params = [[w2],[w1]]
       costs = [cost2,cost1]
       grad_ends = [[a1], [x]]

       next_grad = None
       param_grads = []
       for i in xrange(2):
           param_grad, next_grad = theano.subgraph_grad(
               wrt=params[i], end=grad_ends[i],
               start=next_grad, cost=costs[i]
           )
           next_grad = dict(zip(grad_ends[i], next_grad))
           param_grads.extend(param_grad)

    :type wrt: list of variables
    :param wrt:
      Gradients are computed with respect to `wrt`.

    :type end: list of variables
    :param end:
      Theano variables at which to end gradient descent (they are
      considered constant in theano.grad).  For convenience, the
      gradients with respect to these variables are also returned.

    :type start: dictionary of variables
    :param start:
      If not None, a dictionary mapping variables to their
      gradients. This is useful when the gradient on some variables
      are known. These are used to compute the gradients backwards up
      to the variables in `end` (they are used as known_grad in
      theano.grad).

    :type cost: scalar (0-dimensional) variable
    :param cost:

      Additional costs for which to compute the gradients.  For
      example, these could be weight decay, an l1 constraint, MSE,
      NLL, etc. May optionally be None if start is provided.

      .. warning::

        If the gradients of `cost` with respect to any of the `start`
        variables is already part of the `start` dictionary, then it
        may be counted twice with respect to `wrt` and `end`.

    :type details: bool
    :param details:

      When True, additionally returns the list of gradients from
      `start` and of `cost`, respectively, with respect to `wrt` (not
      `end`).

    :rtype: Tuple of 2 or 4 Lists of Variables

    :return: Returns lists of gradients with respect to `wrt` and `end`,
            respectively.

    .. versionadded:: 0.6.1

.. _R_op_list:


List of Implemented R op
========================


See the :ref:`gradient tutorial <tutcomputinggrads>` for the R op documentation.

list of ops that support R-op:
 * with test [Most is tensor/tests/test_rop.py]
    * SpecifyShape
    * MaxAndArgmax
    * Subtensor
    * IncSubtensor set_subtensor too
    * Alloc
    * Dot
    * Elemwise
    * Sum
    * Softmax
    * Shape
    * Join
    * Rebroadcast
    * Reshape
    * Flatten
    * DimShuffle
    * Scan [In scan_module/tests/test_scan.test_rop]

 * without test
    * Split
    * ARange
    * ScalarFromTensor
    * AdvancedSubtensor1
    * AdvancedIncSubtensor1
    * AdvancedIncSubtensor

Partial list of ops without support for R-op:

 * All sparse ops
 * All linear algebra ops.
 * PermuteRowElements
 * Tile
 * AdvancedSubtensor
 * TensorDot
 * Outer
 * Prod
 * MulwithoutZeros
 * ProdWithoutZeros
 * CAReduce(for max,... done for MaxAndArgmax op)
 * MaxAndArgmax(only for matrix on axis 0 or 1)

