Skip to content

P
pytensor
提交 372685c8 authored 作者: Pascal Lamblin's avatar Pascal Lamblin
浏览文件
操作

Merge pull request #1663 from nouiz/doc_op

Document how to add scalar op and random number distribution.
上级 b5af3406 f1b4c59a
隐藏空白字符变更
内嵌 并排
正在显示 包含 266 行增加 和 120 行删除
.travis.yml
浏览文件 @ 372685c8
......@@ -43,3 +43,5 @@ script:
after_failure:
- cat /home/travis/.pip/pip.log
#after_success:
cache: apt
\ No newline at end of file
doc/extending/index.txt
浏览文件 @ 372685c8
......@@ -30,11 +30,10 @@ a C implementation.
type
op
inplace
other_ops
ctype
cop
optimization
tips
unittest
extending_faq
doc/extending/other_ops.txt 0 → 100644
浏览文件 @ 372685c8
.. _other_ops:
=============================
Implementing some specific Op
=============================
This page is a guide on the implementation of some specific types of Ops,
and point to some examples of such implementations.
For the random number generating Ops, it explains different possible
implementation strategies.
.. _scalar_ops:
Scalar/Elemwise/Reduction Ops
=============================
Implementing a Theano scalar Op allows that scalar operation to be reused
by our elemwise operations on tensors. If the scalar operation has C code, the
elemwise implementation it will automaticaly have C code too. This
will enable the fusion of elemwise operations using your new scalar
operation. It can also reuse the GPU elemwise code. It is similar for
reduction operation.
For examples of how to add new scalar operations, you can have a look at
those 2 pull requests, that add `GammaLn and Psi
<https://github.com/Theano/Theano/pull/686/>`_ and `Gamma
<https://github.com/Theano/Theano/pull/826/>`_ scalar Ops.
Be careful about some possible problems in the definition of the
``grad`` method, and about dependencies that may not be available. In
particular, see the following fixes:
`Fix to grad() methods
<https://github.com/Theano/Theano/commit/002872ad97919b97eaf58e095044e3c3067668e4>`_
and `impl() methods related to SciPy
<https://github.com/Theano/Theano/commit/08d16c0aa6681fc53d8d0f40342551eb47ff536e>`_.
.. _scipy_ops:
SciPy Ops
=========
We can wrap SciPy functions in Theano. But SciPy is an optional dependency.
Here is some code that allows the Op to be optional:
.. code-block:: python
try:
import scipy.linalg
imported_scipy = True
except ImportError:
# some ops (e.g. Cholesky, Solve, A_Xinv_b) won't work
imported_scipy = False
class SomeOp(Op):
...
def make_node(self, x):
assert imported_scipy, (
"SciPy not available. SciPy is needed for the SomeOp op.")
...
from nose.plugins.skip import SkipTest
class test_SomeOp(utt.InferShapeTester):
...
def test_infer_shape(self):
if not imported_scipy:
raise SkipTest("SciPy needed for the SomeOp op.")
...
.. _sparse_ops:
Sparse Ops
==========
There are a few differences to keep in mind if you want to make an op
that uses :ref:`sparse <tutsparse>` inputs or outputs, rather than the
usual dense tensors. In particular, in the
``make_node()`` function, you have to call
``theano.sparse.as_sparse_variable(x)`` on sparse input variables,
instead of ``as_tensor_variable(x)``.
Another difference is that you need to use ``SparseVariable`` and
``SparseType`` instead of ``TensorVariable`` and ``TensorType``.
Don't forget that we support only sparse matrices (so only 2 dimensions)
and they don't support broadcasting operation by default, as SciPy sparse
matrix class does (but a few Ops do it when called manually). Also, we support 2
formats for sparse type: ``csr`` and ``csc``. So in ``make_mode()``,
you can create outputs variables like this:
.. code-block:: python
out_format = inputs[0].format # or 'csr' or 'csc' if the output format is fixed
SparseType(dtype=inputs[0].dtype, format=out_format).make_variable()
See the sparse :class:`theano.sparse.basic.Cast` op `code
<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L753>`_
for a good example for a sparse op with Python code.
.. note::
From the definition of CSR and CSC format, CSR column indices are
not necessarily sorted. Likewise for CSC row indices. Use
:class:`EnsureSortedIndices
<theano.sparse.basic.EnsureSortedIndices>` if your code does not
support it.
Also, there can be explicit zeros in your inputs. Use
:class:`Remove0 <theano.sparse.basic.Remove0>` or ``remove0`` to
make sure they aren't present in your input if you don't support
that.
To remove explicit zeros and make sure indices are sorted, use
:func:`clean <theano.sparse.basic.clean>`.
Sparse Gradient
---------------
There are 2 types of :ref:`gradients <tutsparse_gradient>` for sparse
operations: ``normal``
gradient and ``structured`` gradient. Please document what your op
implements in its docstring. It is important that the user knows it, and
it is not always easy to infer from the code. Also make clear which
inputs/outputs are sparse and which ones are dense.
Sparse C code
-------------
Theano does not have a native C code interface for sparse matrices. The
reason is simple, we use the SciPy sparse matrix object and they don't
have a C object. So we use a simple trick: a sparse matrix is made of
4 fields that are NumPy vector arrays: ``data``, ``indices``, ``indptr``
and ``shape``. So to make
an op with C code that has sparse variables as inputs, we actually make an op
that takes as input the needed fields of those sparse variables.
You can extract the 4 fields with
:func:`theano.sparse.basic.csm_properties`. You can use
:func:`theano.sparse.basic.csm_data`,
:func:`theano.sparse.basic.csm_indices`,
:func:`theano.sparse.basic.csm_indptr` and
:func:`theano.sparse.basic.csm_shape` to extract the individual
fields.
You can look at the `AddSD
<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L1704>`_
sparse op for an example with C code. It implements the addition of a
sparse matrix with a dense matrix.
Sparse Tests
------------
You can reuse the test system for tensor variables. To generate the
needed sparse variable and data, you can use
:func:`theano.sparse.tests.test_basic.sparse_random_inputs`. It takes
many parameters, including parameters for the format (csr or csc), the shape, the
dtype, whether to have explicit 0 and whether to have unsorted indices.
.. _random_ops:
Random distribution
===================
We have 3 base random number generators. One that wraps NumPy's random
generator, one that implements MRG31k3p and one that wraps CURAND.
The fastest, but less developed, is CURAND. It works only on CUDA-enabled
GPUs. It does not work on the CPU and it has fewer random distributions
implemented.
The recommended and 2nd faster is MRG. It works on the GPU and CPU and
has more implemented distributions.
The slowest is our wrapper on NumPy's random generator.
We explain and provide advice on 3 possibles implementations of new
distributions here::
1) Extend our wrapper around NumPy random functions.
See this `PR <https://github.com/Theano/Theano/pull/1607>`_ as an example.
2) Extend MRG implementation by reusing existing Theano Op. Look into
the ``theano/sandbox/rng_mrg.py`` file and grep for all code about
binomal(). This distribution uses the output of the uniform
distribution and converts it to a binomial distribution with
existing Theano operations. The tests go in
``theano/sandbox/test_rng_mrg.py``
3) Extend MRG implementation with a new Op that takes an uniform as
input. Look in the ``theano/sandbox/{rng_mrg,multinomial}.py`` file
and its test in ``theano/sandbox/test_multinomal.py``. This is
recommended when current Theano ops aren't well suited to modify
the uniform to the target distribution. This can happen in
particular is there is a loop or complicated condition.
.. note::
In all cases, you must reuse the same interface as NumPy for compatibility.
.. _openmp_ops:
OpenMP Ops
==========
To allow consistent interface of Ops that support OpenMP, we have some
helper code. Doing this also allows to enable/disable OpenMP globally
or per op for fine-grained control.
Your Op needs to inherit from ``theano.gof.OpenMPOp``. If it overrides
the ``__init__()`` method, it must have an ``openmp=None`` parameter
and must call ``super(MyOpClass, self).__init__(openmp=openmp)``.
The ``OpenMPOp`` class also implements ``c_compile_args`` and
``make_thunk``. This makes it add the correct g++ flag to compile with
OpenMP. It also disables OpenMP and prints a warning if the version of
g++ don't support it.
The Theano flag ``openmp`` is currently False by default as we don't
have code that gets speed up with it. The only current implementation
is ConvOp. It speeds up some cases, but slows down others. That is why
we disable it by default. But we have all the code to have it enabled
by default if there is more then 1 core and that the environment
variable OMP_NUM_THREADS isn't 1. This allows Theano to respect the
current convention.
.. note:
The OpenMP parameter of an Op should not be used in its __eq__ and
__hash__ methods. Those methods are used to merge equivalent
computation in a Theano graph. If we have 2 Apply nodes with the
same inputs and they execute 2 ConvOp that only differ on the
OpenMP parameter, we want them to be merged.
.. _numba_ops:
Numba Ops
=========
Want C speed without writing C code for your new Op? You can use Numba
to generate the C code for you! Here is an `example
Op <https://gist.github.com/nouiz/5492778#file-theano_op-py>`_ doing that.
doc/tutorial/extending_theano.txt
浏览文件 @ 372685c8
......@@ -408,35 +408,6 @@ only applicable to computations involving a single output. Hence, to gain
efficiency over the basic solution that is asked here, the two operations would
have to be jointly optimized explicitly in the code.)
SciPy
=====
We can wrap SciPy functions in Theano. But SciPy is an optional dependency.
Here is some code that allows the Op to be optional:
.. code-block:: python
try:
import scipy.linalg
imported_scipy = True
except ImportError:
# some ops (e.g. Cholesky, Solve, A_Xinv_b) won't work
imported_scipy = False
class SomeOp(Op):
...
def make_node(self, x):
assert imported_scipy, (
"SciPy not available. SciPy is needed for the SomeOp op.")
...
from nose.plugins.skip import SkipTest
class test_SomeOp(utt.InferShapeTester):
...
def test_infer_shape(self):
if not imported_scipy:
raise SkipTest("SciPy needed for the SomeOp op.")
...
Random numbers in tests
=======================
......@@ -455,95 +426,8 @@ For more details see :ref:`random_value_in_tests`.
:download:`Solution<extending_theano_solution_1.py>`
Sparse
======
There is few differences if you want to make an op that use
:ref:`sparse <tutsparse>` inputs or outputs. In particular, in the
``make_node()`` function, you call
``theano.sparse.as_sparse_variable(x)`` on sparse input variable
instead of ``as_tensor_variable(x)``.
Another difference is that you need to use SparseVariable and
SparseType instead of TensorVariable and TensorType.
Don't forget that we support only sparse matrix (so only 2 dimensions)
and they don't support broadcast operation by default as scipy sparse
matrix (but a few op do it when called manually). Also, we support 2
formats for sparse type: ``csr`` and ``csr``. So in ``make_mode()``,
you create outputs variables like this:
.. code-block:: python
out_format = inputs[0].format # or 'csr' or 'csc' if the output format is fixed
SparseType(dtype=inputs[0].dtype, format=out_format).make_variable()
See the sparse :class:`theano.sparse.basic.Cast` op `code
<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L753>`_
for a good example for a sparse op with python code.
.. note::
From the definition of CSR and CSC format, CSR column indices are
not necessarily sorted. Likewise for CSC row indices. Use
:class:`EnsureSortedIndices
<theano.sparse.basic.EnsureSortedIndices>` if your code don't
support it.
Also, there can be explicit zeros in your inputs. Use
:class:`Remove0 <theano.sparse.basic.Remove0>` or ``remove0`` to
make sure they aren't present in your input if you don't support
that.
To remove explicit zeros and make sure indices are sorted, use
:func:`clean <theano.sparse.basic.clean>`.
Sparse Gradient
---------------
There is 2 types of :ref:`gradients <tutsparse_gradient>` : ``normal``
gradient and ``structured`` gradient. Please document what your op
implement in its docstring. It is important that the user know it and
it is not always easy to infer from the code. Also make clear witch
inputs/outputs are sparse and witch ones are dense.
Sparse c code
-------------
Theano don't have a native c code interface for sparse matrix. The
reason is simple, we use the scipy sparse matrix object and they don't
have a c object. So we use a simple trick: a sparse matrix is made of
4 fields that are vector: data, indices, indptr and shape. So to make
an op with c code that have sparse variables as inputs, we make an op
that take as input the needed fields of those sparse variables.
You can extract the 4 fields with
:func:`theano.sparse.basic.csm_properties`. You can use
:func:`theano.sparse.basic.csm_data`,
:func:`theano.sparse.basic.csm_indices`,
:func:`theano.sparse.basic.csm_indptr` and
:func:`theano.sparse.basic.csm_shape` to extract the individual
fields.
You can look at the `AddSD
<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L1704>`_
sparse op for an example with c code. It implement the addition of a
sparse matrix with a dense matrix.
Sparse Tests
------------
You can reuse the test system for tensor variable. To generate the
needed sparse variable and data, you can use
:func:`theano.sparse.tests.test_basic.sparse_random_inputs`. It take
take many paramters including parameters for the format (csr or csc), the shape, the
dtype, to have explicit 0 and to have unsorted indices.
Final Note
==========
A more extensive discussion of this section's content may be found in the advanced
tutorial :ref:`Extending Theano<extending>`
Documentation
=============
See :ref:`metadocumentation`, for some information on how to generate
the documentation.
......@@ -578,3 +462,19 @@ documentation:
.. automodule:: theano.misc.doubleop
:members:
Final Note
==========
A more extensive discussion of this section's content may be found in
the advanced tutorial :ref:`Extending Theano<extending>`
The section :ref:`Other ops <other_ops>` include more instruction for
specific case:
- :ref:`scalar_ops`
- :ref:`scipy_ops`
- :ref:`sparse_ops`
- :ref:`Random ops <random_ops>`
- :ref:`openmp_ops`
- :ref:`numba_ops`
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论