Merge pull request #5615 from abergeron/f16_doc

doc/developer/compat.txt

-===================================
-Compatibility with Python versions.
-===================================
-
-Theano is compatible with Python versions >= 2.6 including 3.x series.
-This guide provides coding guidelines on how to maintain
-comnpatibility.
-
-
-Installation (2to3 and setup.py)
---------------------------------
-
-
-Python code compatibility
--------------------------
-Ideally, Python 3-incompatible changes should be caught by the Travis build,
-but here are some guidelines for the major things to be aware of.
-
-* When catching an exception and giving a name to it, use the new-style
-  syntax:
-
-    .. code-block:: python
-
-        except Exception as e:
-           ...
-
-* No tuple-unpacking in the argument list:
-
-    .. code-block:: python
-
-        def f(a, (b, c)):  # wrong
-            ...
-
-        def f(a, others):
-            b, c = others  # right
-
-* Always use ``print(...)`` as a function, and add
-
-    .. code-block:: python
-
-        from __future__ import print_function
-
-  At the top of files that use ``print``, above all other imports.
-
-* If you mean to iterate over a range, use ``six.moves.xrange``. If you need
-  a range as a list, use ``list(range(...))``.
-
-* If you mean to iterate over a zipped sequence, use ``theano.compat.izip``.
-  If you actually need it to be a list, use ``list(zip(...))``.
-
-* If you mean to iterate over the output of a ``map``, use
-  ``theano.compat.imap``.  If you actually need it to be a list, use
-  ``list(map(...))``.
-
-* If using ``reduce``, ``StringIO``, ``pickle``/``cPickle``, or ``copy_reg``,
-  use the one in ``six.moves`` (note that ``copy_reg`` is named ``copyreg``).
-
-* For iterating over the items, keys, values of a list, use
-  ``six.iteritems(...)``, ``six.iterkeys(...)``, ``six.itervalues(...)``. If
-  you wish to grab these as lists, wrap them in a ``list(...)`` call.
-
-* Avoid ``itertools.{izip,ifilter,imap}``. Use the equivalently named functions
-  from ``theano.compat``.
-
-* Don't use `__builtin__`. Rather, ``six.moves.builtins``.
-
-* Rather than the three-argument version of ``raise``, use
-  ``six.reraise(...)``.
-
-* Don't call the ``.next()`` method of iterators. Use the builtin
-  ``next(...)``.
-
-* When type testing for ``str`` or ``int``/``long``, use ``six.string_types``
-  and ``six.integer_types``, respectively.
-
-* Use the ``six.add_metaclass`` class decorator for adding metaclasses.
-
-
-Compatibility package (theano.compat)
-.....................................
-
-Compatibility between 2.x and 3.x is implemented using six_, a Python
-2 and 3 compatibility library by Benjamin Peterson.  Additionally,
-certain additional or modified functions are available in the ``theano.compat``
-module.
-
-
-Strings, bytes and unicode
-..........................
-``str`` and ``bytes`` are different types in Python 3. Mostly this doesn't
-come up for Theano's purposes. ``str`` represents encoded text, i.e. character
-encoding-aware; all Python 3 strings are internally stored as Unicode. When
-converting from one or the other, you must `.encode(...)` bytes and
-`.decode(...)` strings with a given character encoding (e.g. ``'ascii'``,
-``'utf8'``, ``'latin1'``).
-
-C code compatibility
---------------------
-
-Module initialization
-.....................
-
-PyCapsule and PyCObject
-.......................
-
-
-
-References
-----------
-
-.. _six: http://pythonhosted.org/six

doc/developer/index.txt

-:orphan:
-
-.. _developer:
-
-==============================================
-Theano Design and Implementation Documentation
-==============================================
-
-
-.. toctree::
-   :maxdepth: 2
-
-   tensor
-   scan
-   compat

doc/developer/tensor.txt

-.. _tensor:
-
-=======
-Tensor
-=======
-
-This file describes the design of theano.tensor.
-
-Elemwise grad and R_op 
-======================
-
-Here's another straightforward example, though a bit more elaborate
-than adding two numbers together. Let's say that you want to compute
-the logistic curve, which is given by:
-
-.. math::
-
-   s(x) = \frac{1}{1 + e^{-x}}
-

doc/extending/extending_theano_gpu.txt

@@ -156,25 +156,92 @@ in most cases in your :meth:`c_code` method.  This is done by using
 the provided wrapper function. An example calling the above kernel
 would be::
 
-    size_t ls, gs;
     size_t dims[2];
+    size_t n = 256;
 
     // ...
 
-    ls = 1;
-    gs = 256;
-    err = k_call(1, &gs, &ls, 0, input->ga.data, dims[0], dims[1]);
+    err = k_scall(1, &n, 0, input->ga.data, dims[0], dims[1]);
 
     // ...
 
+If you want explicit control over the scheduling, you can use the
+`_call` wrapper instead which works like this::
+
+    size_t ls, gs;
+
+    // ...
+
+    gs = 1;
+    ls = 256;
+    err = k_call(1, &gs, &ls, 0, input->ga.data, dims[0], dims[1]);
+
+
 The name of the wrapper function depends on the name you passed to
 ``Kernel()`` when you declared it (or the name in your `#kernel`
-statement).  It defaults to `name + '_call'`.
+statement).  It defaults to `name + '_call' or '_scall'`.
 
 For other operations in the C code you should refer to the
 `libgpuarray documentation
 <http://deeplearning.net/software/libgpuarray/>`_.
 
+Dealing with float16
+====================
+
+To support limited-precision storage in a kernel you have to be
+careful to load values properly, declare working memory in float32 and
+write results properly.  To help with that some functions have been
+declared in :mod:`theano.gpuarray.fp16_help`.
+
+To load the inputs you should wrap the read with the function returned
+by :func:`load_w() <theano.gpuarray.fp16_help.load_w>`. Similarly writes
+should be wrapped in the function returned by :func:`write_w()
+<theano.gpuarray.fp16_help.write_w>`.  Finally working data should
+have the type returned by :func:`work_dtype()
+<theano.gpuarray.fp16_help.work_dtype>`.
+
+Here is a +1 kernel that is not ready to deal with float16 input::
+
+    type_x = dtype_to_ctype(x.dtype)
+    type_y = dtype_to_ctype(y.dtype)
+    """
+    KERNEL void k(const ga_size n, %(type_x)s *x, %(type_y)s *y) {
+        ga_size i = GID_0 * LDIM_0 + LID_0;
+        %(type_x) z = x[i];
+        z += 1;
+        y[i] = z;
+    }
+    """ % dict(dtype_x=dtype_x, dtype_y=dtype_y)
+
+Here is the same kernel, but now ready to handle float16::
+
+    type_x = dtype_to_ctype(x.dtype)
+    type_y = dtype_to_ctype(y.dtype)
+    work_x = dtype_to_ctype(work_dtype(x.dtype))
+    load_x = load_w(x.dtype)
+    write_y = write_w(y.dtype)
+    """
+    KERNEL void k(const ga_size n, %(type_x)s *x, %(type_y)s *y) {
+        ga_size i = GID_0 * LDIM_0 + LID_0;
+        %(work_x) z = %(load_x)(x[i]);
+        z += 1;
+        y[i] = %(write_y)(z);
+    }
+    """ % dict(dtype_x=dtype_x, dtype_y=dtype_y, work_x=work_x, load_x=load_x,
+               write_y=write_y)
+
+Once you have converted your kernels for float16 support you need to
+tag your op with ``_f16_ok = True`` so that the linker will accept to
+generate C code on float16 types.  This is done by inserting it as a
+class property like this::
+
+    class SomeOp(Op):
+        _f16_ok = True
+
+If this attribute is not present or is False, the linker will print a
+message saying that it's refusing to use C code for float16 for the
+op.
+
 A Complete Example
 ==================
 

doc/extending/index.txt

@@ -49,4 +49,5 @@ with Theano itself.
     optimization
     tips
     unittest
+    scan
     extending_faq

doc/developer/scan.txt → doc/extending/scan.txt

@@ -91,27 +91,30 @@ designated **inner inputs** and **inner outputs**, respectively.
 Scan variables
 ==============
 
-The following are the different types of variables that Scan has the capacity to
-handle, along with their various caracteristics.
-
-**Sequence** : A sequence is a Theano variable which Scan will iterate over and
-give sub-elements to its inner function as input. A sequence has no associated
-output. For a sequence variable ``X``, at timestep ``t``, the inner function will
-receive as input the sequence element ``X[t]``. These variables are used through
-the argument ``sequences`` of the ``theano.scan()`` function.
-
-**Non-sequences** : A non-sequence is a Theano variable which Scan will provide
-*as-is* to its inner function. Like a sequence, a non-sequence has no
-associated output. For a non-sequence variable ``X``, at timestep ``t``, the inner
-function will receive as input the variable ``X``. These variables are used through
-the argument ``non_sequences`` of the ``theano.scan()`` function.
-
-**Nitsot (no input tap, single output tap)** : A nitsot is an output variable of
-the inner function that is not fed back as an input to the next iteration of the
-inner function. Nitsots are typically encountered in situations where Scan is
-used to perform a 'map' operation (every element in a tensor is independently
-altered using a given operation to produce a new tensor) such as squaring every
-number in a vector.
+The following are the different types of variables that Scan has the
+capacity to handle, along with their various caracteristics.
+
+**Sequence** : A sequence is a Theano variable which Scan will iterate
+over and give sub-elements to its inner function as input. A sequence
+has no associated output. For a sequence variable ``X``, at timestep
+``t``, the inner function will receive as input the sequence element
+``X[t]``. These variables are used through the argument ``sequences``
+of the ``theano.scan()`` function.
+
+**Non-sequences** : A non-sequence is a Theano variable which Scan
+*will provide as-is* to its inner function. Like a sequence, a
+non-sequence has no associated output. For a non-sequence variable
+``X``, at timestep ``t``, the inner function will receive as input
+the variable ``X``. These variables are used through the argument
+``non_sequences`` of the ``theano.scan()`` function.
+
+**Nitsot (no input tap, single output tap)** : A nitsot is an output
+variable of the inner function that is not fed back as an input to the
+next iteration of the inner function. Nitsots are typically
+encountered in situations where Scan is used to perform a 'map'
+operation (every element in a tensor is independently altered using a
+given operation to produce a new tensor) such as squaring every number
+in a vector.
 
 **Sitsot (single input tap, single output tap)** : A sitsot is an output
 variable of the inner function that is fed back as an input to the next
@@ -120,13 +123,14 @@ encountered is the case where Scan is used to compute the cumulative sum over
 the elements of a vector and a sitsot output is employed to act as an
 accumulator.
 
-**Mitsot (multiple input taps, single output tap)** : A mitsot is an output
-variable of the inner function that is fed back as an input to future iterations
-of the inner function (either multiple future iterations or a single one that
-isn't the immediate next one). For example, a mitsot might be used in the case
-where Scan is used to compute the Fibonacci sequence, one term of the sequence
-at every timestep, since every computed term needs to be reused to compute
-the two next terms of the sequence.
+**Mitsot (multiple input taps, single output tap)** : A mitsot is an
+output variable of the inner function that is fed back as an input to
+future iterations of the inner function (either multiple future
+iterations or a single one that isn't the immediate next one). For
+example, a mitsot might be used in the case where Scan is used to
+compute the Fibonacci sequence, one term of the sequence at every
+timestep, since every computed term needs to be reused to compute the
+two next terms of the sequence.
 
 **Mitmot (multiple input taps, multiple output taps)** : These outputs exist
 but they cannot be directly created by the user. They can appear in a theano
@@ -272,12 +276,13 @@ by usage.
 Accessing/manipulating Scan's inputs and outputs by type
 --------------------------------------------------------
 
-Declared in ``scan_utils.py``, the class ``scan_args`` handles the parsing
-of the inputs and outputs (both inner and outer) to a format that is easier to
-analyse and manipulate. Without this class, analysing Scan's inputs and
-outputs often required convoluted logic which make for code that is hard to
-read and to maintain. Because of this, you should favor using ``scan_args`` when
-it is practical and appropriate to do so.
+Declared in ``scan_utils.py``, the class ``scan_args`` handles the
+parsing of the inputs and outputs (both inner and outer) to a format
+that is easier to analyse and manipulate. Without this class,
+analysing Scan's inputs and outputs often required convoluted logic
+which make for code that is hard to read and to maintain. Because of
+this, you should favor using ``scan_args`` when it is practical and
+appropriate to do so.
 
 The scan op also defines a few helper functions for this purpose, such as
 ``inner_nitsot_outs()`` or ``mitmot_out_taps()``, but they are often poorly
@@ -320,10 +325,11 @@ individual indices or -1 if there is not corresponding outer variable.
 For mappings to inner variables, the values are list of indices because
 multiple inner variables may be associated with the same state.
 
-If the goal is to navigate between variables that are *connected* (meaning
-that one of them is used to compute the other), the methods
-``connection_pattern()`` and ``inner_connection_pattern()`` can be used.
-The method ``connection_pattern()`` returns a list of lists detailing, for every
-pair of outer input and outer output whether they are connected or not. The
-method ``inner_connection_pattern()`` accomplishes the same goal but for every
-possible pair of inner output and inner input.
+If the goal is to navigate between variables that are *connected*
+(meaning that one of them is used to compute the other), the methods
+``connection_pattern()`` and ``inner_connection_pattern()`` can be
+used.  The method ``connection_pattern()`` returns a list of lists
+detailing, for every pair of outer input and outer output whether they
+are connected or not. The method ``inner_connection_pattern()``
+accomplishes the same goal but for every possible pair of inner output
+and inner input.

doc/index.txt

@@ -141,7 +141,6 @@ Roughly in order of what you'll want to check out:
 * :ref:`optimizations` -- Guide to Theano's graph optimizations.
 * :ref:`extending` -- Learn to add a Type, Op, or graph optimization.
 * :ref:`dev_start_guide` -- How to contribute code to Theano.
-* :ref:`developer` -- Primarily of interest to developers of Theano
 * :ref:`internal` -- How to maintain Theano and more...
 * :ref:`release` -- How our release should work.
 * :ref:`acknowledgement` -- What we took from other projects.

doc/library/gpuarray/extra.txt

@@ -15,3 +15,9 @@ Kernel generation
 
 .. automodule:: theano.gpuarray.kernel_codegen
    :members:
+
+float16
+-------
+
+.. automodule:: theano.gpuarray.fp16_help
+   :members:

doc/library/gpuarray/fft.txt

 .. _libdoc_gpuarray_fft:
 
-==============================================
-:mod:`gpuarray.fft` -- Fast Fourier Transforms
-==============================================
+=====================================================
+:mod:`theano.gpuarray.fft` -- Fast Fourier Transforms
+=====================================================
 
 Performs Fast Fourier Transforms (FFT) on the GPU.
 

doc/library/gpuarray/type.txt

 .. _libdoc_gpuarray_type:
 
 ===================================================
-:mod:`gpuarray.type` -- Type classes
+:mod:`theano.gpuarray.type` -- Type classes
 ===================================================
 
 .. automodule:: theano.gpuarray.type

theano/gpuarray/fp16_help.py

@@ -2,6 +2,10 @@ from __future__ import absolute_import, print_function, division
 
 
 def work_dtype(dtype):
+    """
+    Return the data type for working memory.
+
+    """
     if dtype == 'float16':
         return 'float32'
     else:
@@ -9,6 +13,14 @@ def work_dtype(dtype):
 
 
 def load_w(dtype):
+    """
+    Return the function name to load data.
+
+    This should be used like this::
+
+        code = '%s(ival)' % (load_w(input_type),)
+
+    """
     if dtype == 'float16':
         return '__half2float'
     else:
@@ -16,6 +28,14 @@ def load_w(dtype):
 
 
 def write_w(dtype):
+    """
+    Return the function name to write data.
+
+    This should be used like this::
+
+        code = 'res = %s(oval)' % (write_w(output_type),)
+
+    """
     if dtype == 'float16':
         return '__float2half_rn'
     else:

theano/gpuarray/tests/test_cgpukernelbase.py

@@ -23,7 +23,6 @@ class GpuEye(CGpuKernelBase, Op):
 
     """
     __props__ = ('dtype', 'context_name')
-    _f16_ok = True
 
     def __init__(self, dtype=None, context_name=None):
         if dtype is None: