.. _libdoc_gpuarray_dnn:

===========================================
:mod:`theano.gpuarray.dnn` -- cuDNN
===========================================

.. moduleauthor:: LISA

`cuDNN <https://developer.nvidia.com/cuDNN>`_ is an NVIDIA library
with functionality used by deep neural networks. It provides optimized
versions of some operations like the convolution. cuDNN is not
currently installed with CUDA. You must download and install it
yourself.

To install it, decompress the downloaded file and make the ``*.h`` and
``*.so*`` files available to the compilation environment.
There are at least three possible ways of doing so:

- The easiest is to include them in your CUDA installation. Copy the
  ``*.h`` files to ``CUDA_ROOT/include`` and the ``*.so*`` files to
  ``CUDA_ROOT/lib64`` (by default, ``CUDA_ROOT`` is ``/usr/local/cuda``
  on Linux).
- Alternatively, on Linux, you can set the environment variables
  ``LD_LIBRARY_PATH``, ``LIBRARY_PATH`` and ``CPATH`` to the directory
  extracted from the download. If needed, separate multiple directories
  with ``:`` as in the ``PATH`` environment variable.

  example::

      export LD_LIBRARY_PATH=/home/user/path_to_CUDNN_folder/lib64:$LD_LIBRARY_PATH
      export CPATH=/home/user/path_to_CUDNN_folder/include:$CPATH
      export LIBRARY_PATH=/home/user/path_to_CUDNN_folder/lib64:$LD_LIBRARY_PATH

- And as a third way, also on Linux, you can copy the ``*.h`` files
  to ``/usr/include`` and the ``*.so*`` files to ``/lib64``.

By default, Theano will detect if it can use cuDNN. If so, it will use
it.  If not, Theano optimizations will not introduce cuDNN ops. So
Theano will still work if the user did not introduce them manually.

To get an error if Theano can not use cuDNN, use this Theano flag:
``optimizer_including=cudnn``.

.. note::

   cuDNN v5.1 is supported in Theano master version. So it dropped cuDNN v3 support.
   Theano 0.8.0 and 0.8.1 support only cuDNN v3 and v4.
   Theano 0.8.2 will support only v4 and v5.

.. note::

   Starting in cuDNN v3, multiple convolution implementations are offered and
   it is possible to use heuristics to automatically choose a convolution
   implementation well suited to the parameters of the convolution.

   The Theano flag ``dnn.conv.algo_fwd`` allows to specify the cuDNN
   convolution implementation that Theano should use for forward convolutions.
   Possible values include :

   * ``small`` (default) : use a convolution implementation with small memory
     usage
   * ``none`` : use a slower implementation with minimal memory usage
   * ``large`` : use a sometimes faster implementation with large memory usage
   * ``fft`` : use the Fast Fourier Transform implementation of convolution
     (very high memory usage)
   * ``guess_once`` : the first time a convolution is executed, the
     implementation to use is chosen according to cuDNN's heuristics and reused
     for every subsequent execution of the convolution.
   * ``guess_on_shape_change`` : like ``guess_once`` but a new convolution
     implementation selected every time the shapes of the inputs and kernels
     don't match the shapes from the last execution.
   * ``time_once`` : the first time a convolution is executed, every convolution
     implementation offered by cuDNN is executed and timed. The fastest is
     reused for every subsequent execution of the convolution.
   * ``time_on_shape_change`` : like ``time_once`` but a new convolution
     implementation selected every time the shapes of the inputs and kernels
     don't match the shapes from the last execution.

   The Theano flag ``dnn.conv.algo_bwd`` allows to specify the cuDNN
   convolution implementation that Theano should use for gradient convolutions.
   Possible values include :

   * ``none`` (default) : use the default non-deterministic convolution
     implementation
   * ``deterministic`` : use a slower but deterministic implementation
   * ``fft`` : use the Fast Fourier Transform implementation of convolution
     (very high memory usage)
   * ``guess_once`` : the first time a convolution is executed, the
     implementation to use is chosen according to cuDNN's heuristics and reused
     for every subsequent execution of the convolution.
   * ``guess_on_shape_change`` : like ``guess_once`` but a new convolution
     implementation selected every time the shapes of the inputs and kernels
     don't match the shapes from the last execution.
   * ``time_once`` : the first time a convolution is executed, every convolution
     implementation offered by cuDNN is executed and timed. The fastest is
     reused for every subsequent execution of the convolution.
   * ``time_on_shape_change`` : like ``time_once`` but a new convolution
     implementation selected every time the shapes of the inputs and kernels
     don't match the shapes from the last execution.

   ``guess_*`` and ``time_*`` flag values take into account the amount of
   available memory when selecting an implementation. This means that slower
   implementations might be selected if not enough memory is available for the
   faster implementations.

.. note::

    Normally you should not call GPU Ops directly, but the CPU interface
    currently does not allow all options supported by cuDNN ops. So it is
    possible that you will need to call them manually.

.. note::

    The documentation of CUDNN tells that, for the 2 following operations, the
    reproducibility is not guaranteed with the default implementation:
    `cudnnConvolutionBackwardFilter` and `cudnnConvolutionBackwardData`.
    Those correspond to the gradient wrt the weights and the gradient wrt the
    input of the convolution. They are also used sometimes in the forward
    pass, when they give a speed up.

    The Theano flag ``dnn.conv.algo_bwd`` can be use to force the use of a
    slower but deterministic convolution implementation.

.. note::

    There is a problem we do not understand yet when cudnn paths are
    used with symbolic links. So avoid using that.

.. note::

    cudnn.so* must be readable and executable by everybody.
    cudnn.h must be readable by everybody.


- Convolution:
    - :func:`theano.gpuarray.dnn.dnn_conv`, :func:`theano.gpuarray.dnn.dnn_conv3d`.
    - :func:`theano.gpuarray.dnn.dnn_gradweight`, :func:`theano.gpuarray.dnn.dnn_gradweight3d`.
    - :func:`theano.gpuarray.dnn.dnn_gradinput`, :func:`theano.gpuarray.dnn.dnn_gradinput3d`.
- Pooling:
    - :func:`theano.gpuarray.dnn.dnn_pool`.
- Batch Normalization:
    - :func:`theano.gpuarray.dnn.dnn_batch_normalization_train`
    - :func:`theano.gpuarray.dnn.dnn_batch_normalization_test`.
- RNN:
    - :class:`theano.gpuarray.dnn.RNNBlock`
- Softmax:
    - You can manually use the op :class:`GpuDnnSoftmax
      <theano.gpuarray.dnn.GpuDnnSoftmax>` to use its extra feature.
- Spatial Transformer:
    - :func:`theano.gpuarray.dnn.dnn_spatialtf`.


cuDNN RNN Example
=================

This is a code example of using the cuDNN RNN functionality.  We
present the code with some commentary in between to explain some
peculiarities.

The terminology here assumes that you are familiar with RNN structure.

.. code-block:: python

    dtype = 'float32'
    input_dim = 32
    hidden_dim = 16
    batch_size = 2
    depth = 3
    timesteps = 5

To clarify the rest of the code we define some variables to hold sizes.

.. code-block:: python

    X = T.tensor3('X')
    Y = T.tensor3('Y')
    h0 = T.tensor3('h0')

We also define some Theano variables to work with.  Here `X` is input,
`Y` is output (as in expected output) and `h0` is the initial state
for the recurrent inputs.

.. code-block:: python

    rnnb = dnn.RNNBlock(dtype, hidden_dim, depth, 'gru')

This defines an RNNBlock.  This is a departure from usual Theano
operations in that it has the structure of a layer more than a
separate operation.  This is constrained by the underlying API.

.. code-block:: python

    psize = rnnb.get_param_size([batch_size, input_dim])
    params_cudnn = gpuarray_shared_constructor(
        np.zeros((psize,), dtype=theano.config.floatX))

Here we allocate space for the trainable parameters of the RNN.  The
first function tells us how many elements we will need to store the
parameters.  This space if for all the parameters of all the layers
inside the RNN and the layout is opaque.

.. code-block:: python

   layer = 0
    = rnnb.split_params(params_cudnn, layer,
                                  [batch_size, input_dim])

If you need to access the parameters individually, you can call
split_params on your shared variable to get all the parameters for a
single layer. The order and number of returned items depends on the
type of RNN.

rnn_relu, rnn_tanh
  input, recurrent

gru
  input reset, input update, input newmem, recurrent reset, recurrent
  update, recurrent newmem

lstm
  input input gate, input forget gate, input newmem gate, input output
  gate, recurrent input gate, recurrent update gate, recurrent newmem
  gate, recurrent output gate

All of these elements are composed of a weights and bias (matrix and
vector).

.. code-block:: python

    y, hy = rnnb.apply(params_cudnn, X, h0)

This is more akin to an op in Theano in that it will apply the RNN
operation to a set of symbolic inputs and return symbolic outputs.
`y` is the output, `hy` is the final state for the recurrent inputs.

After this, the gradient works as usual so you can treat the returned
symbolic outputs as normal Theano symbolic variables.


List of Implemented Operations
==============================

.. automodule:: theano.gpuarray.dnn
   :members:

