.. _theano_random:

=============================
Implementing some specific Op
=============================

This page guide on the implementation of some specify type of Op:
scalar operation and random distribution.

This is useful as for scalar, it is easy to add just the scalar
operation have it being reused with the elemwise/reduction code
generated and optimization.

For the random number, it explain the different implementation
strategy.


.. scalar_ops:
Scalar Operation
================

There is those 2 PR that add `GammaLn and Psi
<https://github.com/Theano/Theano/pull/686/>`_ and `Gamma
<https://github.com/Theano/Theano/pull/826/>`_ scalar op.

Take care
`Fix to grad() methods <https://github.com/Theano/Theano/commit/002872ad97919b97eaf58e095044e3c3067668e4>`_ and `impl() methods related to SciPy <https://github.com/Theano/Theano/commit/08d16c0aa6681fc53d8d0f40342551eb47ff536e>`_


.. _scipy_ops:
SciPy
=====

We can wrap SciPy functions in Theano. But SciPy is an optional dependency.
Here is some code that allows the Op to be optional:

.. code-block:: python

    try:
        import scipy.linalg
        imported_scipy = True
    except ImportError:
        # some ops (e.g. Cholesky, Solve, A_Xinv_b) won't work
        imported_scipy = False

    class SomeOp(Op):
        ...
        def make_node(self, x):
            assert imported_scipy, (
            "SciPy not available. SciPy is needed for the SomeOp op.")
            ...

    from nose.plugins.skip import SkipTest
    class test_SomeOp(utt.InferShapeTester):
        ...
        def test_infer_shape(self):
            if not imported_scipy:
                raise SkipTest("SciPy needed for the SomeOp op.")
            ...

.. sparse_ops:
Sparse
======

There is few differences if you want to make an op that use
:ref:`sparse <tutsparse>` inputs or outputs. In particular, in the
``make_node()`` function, you call
``theano.sparse.as_sparse_variable(x)`` on sparse input variable
instead of ``as_tensor_variable(x)``.

Another difference is that you need to use SparseVariable and
SparseType instead of TensorVariable and TensorType.

Don't forget that we support only sparse matrix (so only 2 dimensions)
and they don't support broadcast operation by default as scipy sparse
matrix (but a few op do it when called manually). Also, we support 2
formats for sparse type: ``csr`` and ``csr``. So in ``make_mode()``,
you create outputs variables like this:

.. code-block:: python

    out_format = inputs[0].format  # or 'csr' or 'csc' if the output format is fixed
    SparseType(dtype=inputs[0].dtype, format=out_format).make_variable()

See the sparse :class:`theano.sparse.basic.Cast` op `code
<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L753>`_
for a good example for a sparse op with python code.

.. note::

   From the definition of CSR and CSC format, CSR column indices are
   not necessarily sorted. Likewise for CSC row indices. Use
   :class:`EnsureSortedIndices
   <theano.sparse.basic.EnsureSortedIndices>` if your code don't
   support it.

   Also, there can be explicit zeros in your inputs. Use
   :class:`Remove0 <theano.sparse.basic.Remove0>` or ``remove0`` to
   make sure they aren't present in your input if you don't support
   that.

   To remove explicit zeros and make sure indices are sorted, use
   :func:`clean <theano.sparse.basic.clean>`.

Sparse Gradient
---------------

There is 2 types of :ref:`gradients <tutsparse_gradient>` : ``normal``
gradient and ``structured`` gradient. Please document what your op
implement in its docstring. It is important that the user know it and
it is not always easy to infer from the code. Also make clear witch
inputs/outputs are sparse and witch ones are dense.

Sparse c code
-------------

Theano don't have a native c code interface for sparse matrix. The
reason is simple, we use the scipy sparse matrix object and they don't
have a c object. So we use a simple trick: a sparse matrix is made of
4 fields that are vector: data, indices, indptr and shape. So to make
an op with c code that have sparse variables as inputs, we make an op
that take as input the needed fields of those sparse variables.

You can extract the 4 fields with
:func:`theano.sparse.basic.csm_properties`. You can use
:func:`theano.sparse.basic.csm_data`,
:func:`theano.sparse.basic.csm_indices`,
:func:`theano.sparse.basic.csm_indptr` and
:func:`theano.sparse.basic.csm_shape` to extract the individual
fields.

You can look at the `AddSD
<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L1704>`_
sparse op for an example with c code. It implement the addition of a
sparse matrix with a dense matrix.

Sparse Tests
------------

You can reuse the test system for tensor variable. To generate the
needed sparse variable and data, you can use
:func:`theano.sparse.tests.test_basic.sparse_random_inputs`. It take
take many paramters including parameters for the format (csr or csc), the shape, the
dtype, to have explicit 0 and to have unsorted indices.

.. random_ops:
Random distribution
===================

We have 3 base random number generators. One that wrap NumPy random
generator, one that implement MRG31k3p and one that wrap CURAND.

The fastest, but less developed is CURAND. It work only on CUDA enable
GPUs. It don't work on the CPU and it have less random distribution.

The recommended and 2nd faster is MRG. It work on the GPU and CPU and
have more distribution.

The slowest is our wrapper on NumPy random generator.

We explain and guide on 3 possibles implementations of new
distribution here::

1) Extend our wrapper to NumPy random function.
   See this `PR <https://github.com/Theano/Theano/pull/1607>`_ as an example.

2) Extend MRG implementation by reusing existing Theano Op. Look into
   the ``theano/sandbox/rng_mrg.py`` file and grep for all code about
   binomal(). This distribution use the output of the uniform
   distribution and convert it to a binomial distribution with
   existing Theano op. The test go in
   ``theano/sandbox/test_rng_mrg.py``

3) Extend MRG implementation with a new Op that take an uniform as
   input. Look in the ``theano/sandbox/{rng_mrg,multinomial}.py`` file
   and its test in ``theano/sandbox/test_multinomal.py``. This is
   recommended when current Theano ops aren't well suited to modify
   the uniform to the target distribution. This can happen in
   particular is there is a loop or complicated condition.

.. note::

    In all cases, you must reuse the same interface as NumPy for compatibility.
