Skip to content

P
pytensor
提交 5a0d273c authored 作者: abergeron's avatar abergeron
浏览文件
操作

Merge pull request #3924 from fvisin/fix_tutorial

Move the extending_theano documentation out of tutorial
上级 9ae1ab8e 5621b267
显示空白字符变更
内嵌 并排
正在显示 包含 424 行增加 和 530 行删除
doc/cifarSC2011/index.txt
浏览文件 @ 5a0d273c
......@@ -66,7 +66,7 @@ from gurus on hand if you get stuck.
introduction
theano
advanced_theano
/tutorial/extending_theano
/extending/extending_theano
pyCUDA
gpundarray
doc/crei2013/index.txt
浏览文件 @ 5a0d273c
......@@ -69,4 +69,4 @@ from gurus on hand if you get stuck.
theano
advanced_theano
gpundarray
/tutorial/extending_theano
/extending/extending_theano
doc/tutorial/extending_theano.txt → doc/extending/extending_theano.txt
浏览文件 @ 5a0d273c
.. _extending_theano:
================
Extending Theano
================
Creating a new Op: Python implementation
========================================
This tutorial covers how to extend Theano with novel ops. It mainly focuses on ops that offer a Python implementation, refers to :ref:`extending_theano_c` for C-based op.
The first section of this tutorial introduces the Theano Graphs,
as providing a novel Theano op requires a basic understanting of the Theano Graphs. It then proposes an overview of the most important methods that define an op.
So suppose you have looked through the library documentation and you don't see
a function that does what you want.
As an illustration, this tutorial shows how to write a simple Python-based op which performs operations on Double. It also shows how to implement tests that ensure the proper working of an op.
If you can implement something in terms of existing Ops, you should do that.
Odds are your function that uses existing Theano expressions is short,
has no bugs, and potentially profits from optimizations that have already been
implemented.
However, if you cannot implement an Op in terms of existing Ops, you have to
write a new one. Don't worry, Theano was designed to make it easy to add new
Ops, Types, and Optimizations.
.. These first few pages will walk you through the definition of a new :ref:`type`,
.. ``double``, and a basic arithmetic :ref:`operations <op>` on that Type.
As an illustration, this tutorial shows how to write a simple Python-based
:ref:`operations <op>` which performs operations on
:ref:`type`, ``double<Double>``.
.. It also shows how to implement tests that
.. ensure the proper working of an op.
.. note::
This tutorial does not cover how to make an op that returns a view or
modifies the values in its inputs. Thus, all ops created with the
instructions described here MUST return newly allocated
memory or reuse the memory provided in the parameter
``output_storage`` of the :func:`perform` function. See :ref:`views_and_inplace`
for an explanation on how to do this.
This is an introductury tutorial and as such it does not cover how to make
an op that returns a view or modifies the values in its inputs. Thus, all
ops created with the instructions described here MUST return newly
allocated memory or reuse the memory provided in the parameter
``output_storage`` of the :func:`perform` function. See
:ref:`views_and_inplace` for an explanation on how to do this.
If your op returns a view or changes the value of its inputs
without doing as prescribed in that page, Theano will run, but will
......@@ -28,35 +42,36 @@ As an illustration, this tutorial shows how to write a simple Python-based op wh
``mode=DebugMode``) since it verifies if your op behaves correctly in this
regard.
.. note::
See the :ref:`dev_start_guide` for information regarding the versioning
framework, namely about *git* and *GitHub*, regarding the development workflow and
how to make a quality contribution.
Theano Graphs
=============
Theano Graphs refresher
-----------------------
.. image:: ../hpcs2011_tutorial/pics/apply_node.png
:width: 500 px
Theano represents symbolic mathematical computations as graphs. Those graphs are bi-partite graphs (graphs with 2 types of nodes), they are composed of interconnected :ref:`apply` and :ref:`variable` nodes.
:ref:`variable` nodes represent data in the graph, either inputs, outputs or intermediary values. As such, Inputs and Outputs of a graph are lists of Theano :ref:`variable` nodes. :ref:`apply` nodes perform computation on these variables to produce new variables. Each :ref:`apply` node has a link to an instance of :ref:`Op` which describes the computation to perform. This tutorial details how to write such an Op instance. Please refers to :ref:`graphstructures` for a more detailed explanation about the graph structure.
Theano represents symbolic mathematical computations as graphs. Those graphs
are bi-partite graphs (graphs with 2 types of nodes), they are composed of
interconnected :ref:`apply` and :ref:`variable` nodes.
:ref:`variable` nodes represent data in the graph, either inputs, outputs or
intermediary values. As such, Inputs and Outputs of a graph are lists of Theano
:ref:`variable` nodes. :ref:`apply` nodes perform computation on these
variables to produce new variables. Each :ref:`apply` node has a link to an
instance of :ref:`Op` which describes the computation to perform. This tutorial
details how to write such an Op instance. Please refers to
:ref:`graphstructures` for a more detailed explanation about the graph
structure.
Op Structure
============
Op's basic methods
------------------
An op is any Python object which inherits from :class:`gof.Op`.
This section provides an overview of the methods you typically have to implement to make a new op. It does not provide extensive coverage of all the
This section provides an overview of the basic methods you typically have to
implement to make a new op. It does not provide extensive coverage of all the
possibilities you may encounter or need. For that refer to
:ref:`op_contract`.
.. testcode::
.. testcode:: python
import theano
......@@ -102,10 +117,13 @@ possibilities you may encounter or need. For that refer to
def infer_shape(node, input_shapes):
pass
.. ../extending/op.txt
An op has to implement some methods defined in the the interface of
:class:`gof.Op`. More specifically, it is mandatory for an op to define either the method :func:`make_node` or :attr:`itypes`, :attr:`otypes` and one of the implementation methods, either :func:`perform`, :meth:`Op.c_code` or :func:`make_thunk`.
:class:`gof.Op`. More specifically, it is mandatory for an op to define either
the method :func:`make_node` or :attr:`itypes`, :attr:`otypes` and one of the
implementation methods, either :func:`perform`, :meth:`Op.c_code`
or :func:`make_thunk`.
method :func:`make_node` and one of the implementation methods, either
:func:`perform`, :meth:`Op.c_code` or :func:`make_thunk`.
:func:`make_node` method creates an Apply node representing the application
of the op on the inputs provided. This method is reponsible for three things:
......@@ -118,7 +136,8 @@ An op has to implement some methods defined in the the interface of
the symbolic output Variables. It creates output Variables of a suitable
symbolic Type to serve as the outputs of this op's
application.
- it creates an Apply instance with the input and output Variable, and return the Apply instance.
- it creates an Apply instance with the input and output Variable, and
return the Apply instance.
......@@ -151,7 +170,8 @@ An op has to implement some methods defined in the the interface of
For instance, it is possible to define :meth:`Op.c_code` to provide a
C-implementation to the op. Please refers to tutorial
:ref:`extending_theano_c` for a description of :meth:`Op.c_code` and other
related c_methods. Note that an op can provide both Python and C implementation.
related c_methods. Note that an op can provide both Python and C
implementation.
:func:`make_thunk` method is another alternative to :func:`perform`.
It returns a thunk. A thunk is defined as a zero-arguments
......@@ -186,7 +206,10 @@ An op has to implement some methods defined in the the interface of
are used by the Op's :func:`make_node` method to implement the functionality
of :func:`make_node` method mentioned above.
Other methods can be optionally defined by the op.
Op's auxiliary methods
----------------------
There are other methods that can be optionally defined by the op:
The :func:`__str__` method provides a meaningful string representation of
your op.
......@@ -216,18 +239,19 @@ Other methods can be optionally defined by the op.
:attr:`__props__` will also generate a suitable :func:`__str__` for your op.
This requires development version after September 1st, 2014 or version 0.7.
The :func:`infer_shape` method allows to infer the shape of the op
output variables, without actually computing the outputs.
It takes as input ``node``, a reference to the op Apply node,
and a list of Theano symbolic Varables (``i0_shape``, ``i1_shape``, ...)
which are the shape of the op input Variables.
:func:`infer_shape` returns a list where each element is a tuple representing the shape of one output.
:func:`infer_shape` returns a list where each element is a tuple representing
the shape of one output.
This could be helpful if one only
needs the shape of the output instead of the actual outputs, which
can be useful, for instance, for optimization procedures.
The :func:`grad` method is required if you want to differentiate some cost whose expression includes your op. The gradient may be
The :func:`grad` method is required if you want to differentiate some cost
whose expression includes your op. The gradient may be
specified symbolically in this method. It takes two arguments ``inputs`` and
``output_gradients`` which are both lists of symbolic Theano Variables and
those must be operated on using Theano's symbolic language. The grad
......@@ -242,7 +266,6 @@ Other methods can be optionally defined by the op.
NullType for that input. Please refer to :func:`grad` for a more detailed
view.
The :func:`R_op` method is needed if you want ``theano.tensor.Rop`` to
work with your op.
This function implements the application of the R-operator on the
......@@ -257,9 +280,8 @@ Other methods can be optionally defined by the op.
(particularly for scalars) and reduce the number of generated C files.
Op Example
==========
Example: Op definition
----------------------
.. testcode:: example
......@@ -272,12 +294,9 @@ Op Example
__props__ = ()
def make_node(self, x):
# check that the theano version has support for __props__.
# This next line looks like it has a typo,
# but it's actually a way to detect the theano version
# is sufficiently recent to support the use of __props__.
assert hasattr(self, '_props'), "Your version of theano is too old to support __props__."
x = theano.tensor.as_tensor_variable(x)
# Note: using x_.type() is dangerous, as it copies x's broadcasting
# behaviour
return theano.Apply(self, [x], [x.type()])
def perform(self, node, inputs, output_storage):
......@@ -300,6 +319,8 @@ Op Example
return eval_points
return self.grad(inputs, eval_points)
doubleOp1 = DoubleOp1()
#Using itypes and otypes
......@@ -329,7 +350,67 @@ Op Example
return eval_points
return self.grad(inputs, eval_points)
You can try it as follows:
doubleOp2 = DoubleOp2()
At a high level, the code fragment declares a class (e.g., ``DoubleOp1``) and then
creates one instance of it (e.g., ``doubleOp1``).
We often gloss over this distinction, but will be precise here:
``doubleOp1`` (the instance) is an Op, not ``DoubleOp1`` (the class which is a
subclass of ``theano.Op``). You can call ``doubleOp1(tensor.vector())`` on a
Variable to build an expression, and in the expression there will be
a ``.op`` attribute that refers to ``doubleOp1``.
.. The first two methods in the Op are relatively boilerplate: ``__eq__``
.. and ``__hash__``.
.. When two Ops are equal, Theano will merge their outputs if they are applied to the same inputs.
.. The base class (Op) says two objects are equal if (and only if)
.. they are the same object.
.. Writing these boilerplate definitions ensures that the logic of the equality comparison is always explicit.
.. It is an essential part of the :ref:`op_contract` that if two Ops compare
.. equal, then they must compute the same result when presented with the same
.. inputs. Here, if we allocated another instance of ``Fibby`` by typing ``fibby2
.. = Fibby()`` then we would have two Ops that behave identically.
..
.. When should the implementation of ``__eq__`` be more complicated?
.. If ``Fibby.__init__`` had parameters, then we could
.. have configured ``fibby2`` differently from ``fibby`` by passing different
.. arguments to the constructor. If we had done that, and if that different
.. configuration made ``fibby2`` compute different results from ``fibby`` (for the
.. same inputs) then we would have to add logic to the ``__eq__`` and ``__hash__``
.. function so that he two ``Fibby`` Ops would *not be equal*. The reason why: Theano's merge
.. optimization looks for Ops comparing equal and merges them. If two Ops compare
.. equal but don't always produce equal results from equal inputs, then you might
.. see wrong calculation.
The ``make_node`` method creates a node to be included in the expression graph.
It runs when we apply our Op (``doubleOp1``) to the Variable (``x``), as
in ``doubleOp1(tensor.vector())``.
When an Op has multiple inputs, their order in the inputs argument to ``Apply``
is important: Theano will call ``make_node(*inputs)`` to copy the graph,
so it is important not to change the semantics of the expression by changing
the argument order.
All the ``inputs`` and ``outputs`` arguments to ``Apply`` must be Variables.
A common and easy way to ensure inputs are variables is to run them through
``as_tensor_variable``. This function leaves TensorType variables alone, raises
an error for non-TensorType variables, and copies any ``numpy.ndarray`` into
the storage for a TensorType Constant. The ``make_node`` method dictates the
appropriate Type for all output variables.
The ``perform`` method implements the Op's mathematical logic in Python.
The inputs (here ``x``) are passed by value, but a single output is returned
indirectly as the first element of single-element lists. If ``doubleOp1`` had
a second output, it would be stored in ``output_storage[1][0]``.
.. jpt: DOn't understand the following
In some execution modes, the output storage might contain the return value of
a previous call. That old value can be reused to avoid memory re-allocation,
but it must not influence the semantics of the Op output.
You can try the new Op as follows:
.. testcode:: example(Using make_node)
......@@ -395,8 +476,8 @@ You can try it as follows:
[ 0.48165539 0.98642904 0.4913309 0.30702264]]
Example for properties of a Op
==============================
Example: __props__ definition
-----------------------------
We can modify the previous piece of code in order to demonstrate
the usage of the :attr:`__props__` attribute.
......@@ -422,7 +503,8 @@ and ``b`` are equal.
def make_node(self, x):
# check that the theano version has support for __props__.
assert hasattr(self, '_props'), "Your version of theano is too old to support __props__."
assert hasattr(self, '_props'), "Your version of theano is too old
to support __props__."
x = theano.tensor.as_tensor_variable(x)
return theano.Apply(self, [x], [x.type()])
......@@ -439,8 +521,9 @@ and ``b`` are equal.
The use of :attr:`__props__` saves
the user the trouble of implementing :func:`__eq__` and :func:`__hash__` manually.
It also generates a default :func:`__str__` method that prints the attribute names and their values.
the user the trouble of implementing :func:`__eq__` and :func:`__hash__`
manually. It also generates a default :func:`__str__` method that prints the
attribute names and their values.
We can test this by running the following segment:
......@@ -464,14 +547,14 @@ We can test this by running the following segment:
How To Test it
==============
--------------
Theano has some functionalities to simplify testing. These help test the
``infer_shape``, ``grad`` and ``R_op`` methods. Put the following code
in a file and execute it with the ``theano-nose`` program.
Basic Tests
-----------
^^^^^^^^^^^
Basic tests are done by you just by using the op and checking that it
returns the right answer. If you detect an error, you must raise an
......@@ -507,7 +590,7 @@ defaul value do the most strict comparison, 1 and 2 make less strict
comparison.
Testing the infer_shape
-----------------------
^^^^^^^^^^^^^^^^^^^^^^^
When a class inherits from the ``InferShapeTester`` class, it gets the
``self._compile_and_check`` method that tests the op's ``infer_shape``
......@@ -555,7 +638,7 @@ your op works only with such matrices, you can disable the warning with the
self.op_class)
Testing the gradient
--------------------
^^^^^^^^^^^^^^^^^^^^
The function :ref:`verify_grad <validating_grad>`
verifies the gradient of an op or Theano graph. It compares the
......@@ -573,7 +656,7 @@ the multiplication by 2).
[numpy.random.rand(5, 7, 2)])
Testing the Rop
---------------
^^^^^^^^^^^^^^^
.. TODO: repair defective links in the following paragraph
......@@ -595,24 +678,22 @@ For instance, to verify the Rop method of the DoubleOp, you can use this:
def test_double_rop(self):
self.check_rop_lop(DoubleRop()(self.x), self.in_shape)
Testing GPU Ops
---------------
^^^^^^^^^^^^^^^
Ops to be executed on the GPU should inherit from the
``theano.sandbox.cuda.GpuOp`` and not ``theano.Op``. This allows
Theano to distinguish them. Currently, we use this to test if the
NVIDIA driver works correctly with our sum reduction code on the GPU.
Running Your Tests
==================
^^^^^^^^^^^^^^^^^^
To perform your tests, you may select either one of the three
following methods:
theano-nose
-----------
"""""""""""
The method of choice to conduct tests is to run the file
``theano-nose``. In a regular Theano installation, the latter will be
......@@ -630,17 +711,18 @@ purposes:
The following are particularly useful for development purposes since
they call for particular classes or even for particular tests:
* ``theano-nose test_file.py:test_DoubleRop``: Run every test found inside the class *test_DoubleRop*.
* ``theano-nose test_file.py:test_DoubleRop``: Run every test found inside the
class *test_DoubleRop*.
* ``theano-nose test_file.py:test_DoubleRop.test_double_op``: Run only the test *test_double_op*
in the class *test_DoubleRop*.
* ``theano-nose test_file.py:test_DoubleRop.test_double_op``: Run only the test
*test_double_op* in the class *test_DoubleRop*.
Help with the use and functionalities of ``theano-nose`` may be
obtained by running it with the command line parameter ``--help
(-h)``.
nosetests
---------
"""""""""
The command ``nosetests`` can also be used. Although it lacks the
useful functionalities that ``theano-nose`` provides, ``nosetests``
......@@ -653,7 +735,7 @@ More documentation on ``nosetests`` is available here:
`nosetests <http://readthedocs.org/docs/nose/en/latest/>`_.
In-file
-------
"""""""
One may also add a block of code similar to the following at the end
of the file containing a specific test of interest and run the
......@@ -675,9 +757,8 @@ file. This can be done by adding this at the end of your test files:
if __name__ == '__main__':
unittest.main()
Exercise
========
""""""""
Run the code of the *DoubleOp* example above.
......@@ -693,9 +774,26 @@ only applicable to computations involving a single output. Hence, to gain
efficiency over the basic solution that is asked here, the two operations would
have to be jointly optimized explicitly in the code.)
Random numbers in tests
"""""""""""""""""""""""
Making tests errors more reproducible is a good practice. To make your
tests more reproducible, you need a way to get the same random
numbers. You can do this by seeding NumPy's random number
generator.
For convenience, the classes InferShapeTester and RopLop_checker
already do this for you. If you implement your own ``setUp`` function,
don't forget to call the parent ``setUp`` function.
For more details see :ref:`random_value_in_tests`.
:download:`Solution<extending_theano_solution_1.py>`
as_op
=====
-----
as_op is a python decorator that converts a python function into a
basic Theano op that will call the supplied function during execution.
......@@ -706,7 +804,7 @@ implementation.
It takes an optional :func:`infer_shape` parameter that must have this
signature:
.. code-block:: python
.. code-block:: python
def infer_shape(node, input_shapes):
# ...
......@@ -733,10 +831,11 @@ signature:
inputs Theano variables that were declared.
.. note::
The python function wrapped by the `as_op` decorator needs to return a new data allocation, no views or in place modification of the input.
The python function wrapped by the `as_op` decorator needs to return a new
data allocation, no views or in place modification of the input.
as_op Example
-------------
^^^^^^^^^^^^^
.. testcode:: asop
......@@ -767,7 +866,7 @@ You can try it as follows:
Exercise
--------
^^^^^^^^
Run the code of the *numpy_dot* example above.
......@@ -777,32 +876,15 @@ Modify and execute the example to return two outputs: x + y
and x - y.
Random numbers in tests
=======================
Making tests errors more reproducible is a good practice. To make your
tests more reproducible, you need a way to get the same random
numbers. You can do this by seeding NumPy's random number
generator.
For convenience, the classes InferShapeTester and RopLop_checker
already do this for you. If you implement your own ``setUp`` function,
don't forget to call the parent ``setUp`` function.
For more details see :ref:`random_value_in_tests`.
:download:`Solution<extending_theano_solution_1.py>`
Documentation
=============
-------------
See :ref:`metadocumentation`, for some information on how to generate
the documentation.
Here is an example how to add docstring to a class.
.. testcode::
.. testcode:: python
import theano
......@@ -832,7 +914,7 @@ documentation:
:members:
Final Note
==========
----------
A more extensive discussion of this section's content may be found in
the advanced tutorial :ref:`Extending Theano<extending>`.
......
doc/tutorial/extending_theano_c.txt → doc/extending/extending_theano_c.txt
浏览文件 @ 5a0d273c
......@@ -410,6 +410,36 @@ commonly used.
this function should return a tuple of integers as previously
described.
Important restrictions when implementing an Op
==============================================
There are some important restrictions to remember when implementing an Op.
Unless your Op correctly defines a ``view_map`` attribute, the ``perform`` and ``c_code`` must not
produce outputs whose memory is aliased to any input (technically, if changing the
output could change the input object in some sense, they are aliased).
Unless your Op correctly defines a ``destroy_map`` attribute, ``perform`` and ``c_code`` must
not modify any of the inputs.
TODO: EXPLAIN DESTROYMAP and VIEWMAP BETTER AND GIVE EXAMPLE.
When developing an Op, you should run computations in DebugMode, by using
argument ``mode='DebugMode'`` to ``theano.function``. DebugMode is
slow, but it can catch many common violations of the Op contract.
TODO: Like what? How? Talk about Python vs. C too.
DebugMode is no silver bullet though.
For example, if you modify an Op ``self.*`` during any of
``make_node``, ``perform``, or ``c_code``, you are probably doing something
wrong but DebugMode will not detect this.
TODO: jpt: I don't understand the following sentence.
Ops and Types should usually be considered immutable -- you should
definitely not make a change that would have an impact on ``__eq__``,
``__hash__``, or the mathematical value that would be computed by ``perform``
or ``c_code``.
Simple C Op example
===================
......@@ -526,6 +556,11 @@ storage with the right shape and number of dimensions.
return c_code % locals()
The ``c_code`` method accepts variable names as arguments (``name``, ``inp``,
``out``, ``sub``) and returns a C code fragment that computes the expression
output. In case of error, the ``%(fail)s`` statement cleans up and returns
properly.
More complex C Op example
=========================
......
doc/extending/fibby.txt
浏览文件 @ 5a0d273c
......@@ -3,19 +3,6 @@
Writing an Op to work on an ``ndarray`` in C
=============================================
So suppose you have looked through the library documentation and you don't see a
function that does what you want.
If you can implement something in terms of existing Ops, you should do that.
Odds are your function that uses existing Theano expressions is short,
has no bugs, and potentially profits from optimizations that have already been
implemented.
However, if you cannot implement an Op in terms of existing Ops, you have to
write a new one.
Don't worry,
Theano was designed to make it easy to add new Ops, Types, and Optimizations.
This section walks through a non-trivial example Op that does something pretty
weird and unrealistic, that is hard to express with existing Ops.
(Technically, we could use ``Scan`` to implement the Op we're about to describe,
......@@ -73,73 +60,12 @@ you should check the strides and alignment.
fibby = Fibby()
At a high level, the code fragment declares a class (``Fibby``) and then
creates one instance of it (``fibby``).
We often gloss over this distinction, but will be precise here:
``fibby`` (the instance) is an Op, not ``Fibby`` (the class which is a subclass of ``theano.Op``).
You can call ``fibby(tensor.vector())`` on a Variable to build an
expression, and in the expression there will be a ``.op`` attribute that refers
to ``fibby``.
The first two methods in the Op are relatively boilerplate: ``__eq__`` and ``__hash__``.
When two Ops are equal, Theano will merge their outputs if they are applied to the same inputs.
The base class (Op) says two objects are equal if (and only if)
they are the same object.
Writing these boilerplate definitions ensures that the logic of the equality comparison is always explicit.
It is an essential part of the :ref:`op_contract` that if two Ops compare
equal, then they must compute the same result when presented with the same
inputs. Here, if we allocated another instance of ``Fibby`` by typing ``fibby2
= Fibby()`` then we would have two Ops that behave identically.
When should the implementation of ``__eq__`` be more complicated?
If ``Fibby.__init__`` had parameters, then we could
have configured ``fibby2`` differently from ``fibby`` by passing different
arguments to the constructor. If we had done that, and if that different
configuration made ``fibby2`` compute different results from ``fibby`` (for the
same inputs) then we would have to add logic to the ``__eq__`` and ``__hash__``
function so that he two ``Fibby`` Ops would *not be equal*. The reason why: Theano's merge
optimization looks for Ops comparing equal and merges them. If two Ops compare
equal but don't always produce equal results from equal inputs, then you might
see wrong calculation.
The ``make_node`` method creates a node to be included in the expression graph.
It runs when we apply our Op (``fibby``) to Variable (``x``), as in ``fibby(tensor.vector())``.
When an Op has multiple inputs, their order in the inputs argument to ``Apply``
is important: Theano will call ``make_node(*inputs)`` to copy the graph,
so it is important not to change the semantics of the expression by changing the argument order.
All the ``inputs`` and ``outputs`` arguments to ``Apply`` must be Variables.
A common and easy way to ensure inputs are variables is to run them through
``as_tensor_variable``.
This function leaves TensorType variables alone, raises an
error for non-TensorType variables, and copies any ``numpy.ndarray`` into the
storage for a TensorType Constant.
The ``make_node`` method dictates the appropriate Type for all output
variables.
The ``perform`` method implements the Op's mathematical logic in Python.
The inputs (here ``x``) are passed by value,
but a single output is returned indirectly as the first element of
single-element lists. If ``fibby`` had a second output, it would be stored
in ``output_storage[1][0]``.
.. jpt: DOn't understand the following
In some execution modes, the output storage might
contain the return value of a previous call. That old value can be reused to avoid
memory re-allocation, but it must not influence the semantics of the Op output.
The ``c_code`` method accepts variable names as arguments (``name``, ``inames``,
``onames``) and returns a C code fragment that computes the expression output.
In case of error, the ``%(fail)s`` statement cleans up and returns properly.
The variables ``%(x)s`` and ``%(y)s`` are set up by the TensorType to be ``PyArrayObject`` pointers.
TensorType also set up ``dtype_%(x)s`` to be a typdef to the C type for ``x``.
In the first two lines of the C function, we make y point to a new array with
the correct size for the output. This is essentially simulating the line
``y = x.copy()``.
The variables ``%(x)s`` and ``%(y)s`` are set up by the TensorType to be ``PyArrayObject`` pointers.
TensorType also set up ``dtype_%(x)s`` to be a typdef to the C type for ``x``.
.. code-block:: c
......@@ -157,34 +83,6 @@ http://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures.html
TODO: NEEDS MORE EXPLANATION.
There are some important restrictions to remember when implementing an Op.
Unless your Op correctly defines a ``view_map`` attribute, the ``perform`` and ``c_code`` must not
produce outputs whose memory is aliased to any input (technically, if changing the
output could change the input object in some sense, they are aliased).
Unless your Op correctly defines a ``destroy_map`` attribute, ``perform`` and ``c_code`` must
not modify any of the inputs.
TODO: EXPLAIN DESTROYMAP and VIEWMAP BETTER AND GIVE EXAMPLE.
When developing an Op, you should run computations in DebugMode, by using
argument ``mode='DebugMode'`` to ``theano.function``. DebugMode is
slow, but it can catch many common violations of the Op contract.
TODO: Like what? How? Talk about Python vs. C too.
DebugMode is no silver bullet though.
For example, if you modify an Op ``self.*`` during any of
``make_node``, ``perform``, or ``c_code``, you are probably doing something
wrong but DebugMode will not detect this.
TODO: jpt: I don't understand the following sentence.
Ops and Types should usually be considered immutable -- you should
definitely not make a change that would have an impact on ``__eq__``,
``__hash__``, or the mathematical value that would be computed by ``perform``
or ``c_code``.
.. _op_contract_fibby:
Writing an Optimization
......
doc/extending/graphstructures.txt
浏览文件 @ 5a0d273c
......@@ -5,16 +5,28 @@
Graph Structures
================
Theano represents symbolic mathematical computations as graphs. These
graphs are composed of interconnected :ref:`apply` and :ref:`variable`
nodes. They are associated to *function application* and *data*,
respectively. Operations are represented by :ref:`op` instances and data
types are represented by :ref:`type` instances. Here is a piece of code
and a diagram showing the structure built by that piece of code. This
should help you understand how these pieces fit together:
Debugging or profiling code written in Theano is not that simple if you
do not know what goes on under the hood. This chapter is meant to
introduce you to a required minimum of the inner workings of Theano.
The first step in writing Theano code is to write down all mathematical
relations using symbolic placeholders (**variables**). When writing down
these expressions you use operations like ``+``, ``-``, ``**``,
``sum()``, ``tanh()``. All these are represented internally as **ops**.
An *op* represents a certain computation on some type of inputs
producing some type of output. You can see it as a *function definition*
in most programming languages.
Theano represents symbolic mathematical computations as graphs. These
graphs are composed of interconnected :ref:`apply`, :ref:`variable` and
:ref:`op` nodes. *apply* node represents the application of an *op* to some
*variables*. It is important to draw the difference between the
definition of a computation represented by an *op* and its application
to some actual data which is represented by the *apply* node.
Furthermore, data types are represented by :ref:`type` instances. Here is a
piece of code and a diagram showing the structure built by that piece of code.
This should help you understand how these pieces fit together:
-----------------------
**Code**
......@@ -28,12 +40,14 @@ should help you understand how these pieces fit together:
**Diagram**
.. _tutorial-graphfigure:
.. image:: apply.png
:align: center
-----------------------
Arrows represent references to the Python objects pointed at. The blue
box is an :ref:`apply` node. Red boxes are :ref:`variable` nodes. Green
box is an :ref:`Apply` node. Red boxes are :ref:`Variable` nodes. Green
circles are :ref:`Ops <op>`. Purple boxes are :ref:`Types <type>`.
.. TODO
......@@ -58,110 +72,52 @@ Note that the ``Apply`` instance's outputs points to
``z``, and ``z.owner`` points back to the ``Apply`` instance.
An explicit example
===================
In this example we will compare two ways of defining the same graph.
First, a short bit of code will build an expression (graph) the *normal* way, with most of the
graph construction being done automatically.
Second, we will walk through a longer re-coding of the same thing
without any shortcuts, that will make the graph construction very explicit.
**Short example**
This is what you would normally type:
.. testcode::
# create 3 Variables with owner = None
x = T.matrix('x')
y = T.matrix('y')
z = T.matrix('z')
# create 2 Variables (one for 'e', one intermediate for y*z)
# create 2 Apply instances (one for '+', one for '*')
e = x + y * z
**Long example**
This is what you would type to build the graph explicitly:
.. testcode::
from theano.tensor import add, mul, Apply, Variable, Constant, TensorType
# Instantiate a type that represents a matrix of doubles
float64_matrix = TensorType(dtype='float64', # double
broadcastable=(False, False)) # matrix
# We make the Variable instances we need.
x = Variable(type=float64_matrix, name='x')
y = Variable(type=float64_matrix, name='y')
z = Variable(type=float64_matrix, name='z')
# This is the Variable that we want to symbolically represents y*z
mul_variable = Variable(type=float64_matrix)
assert mul_variable.owner is None
# Instantiate a symbolic multiplication
node_mul = Apply(op=mul,
inputs=[y, z],
outputs=[mul_variable])
# Fields 'owner' and 'index' are set by Apply
assert mul_variable.owner is node_mul
# 'index' is the position of mul_variable in mode_mul's outputs
assert mul_variable.index == 0
# This is the Variable that we want to symbolically represents x+(y*z)
add_variable = Variable(type=float64_matrix)
assert add_variable.owner is None
# Instantiate a symbolic addition
node_add = Apply(op=add,
inputs=[x, mul_variable],
outputs=[add_variable])
# Fields 'owner' and 'index' are set by Apply
assert add_variable.owner is node_add
assert add_variable.index == 0
e = add_variable
# We have access to x, y and z through pointers
assert e.owner.inputs[0] is x
assert e.owner.inputs[1] is mul_variable
assert e.owner.inputs[1].owner.inputs[0] is y
assert e.owner.inputs[1].owner.inputs[1] is z
Note how the call to ``Apply`` modifies the ``owner`` and ``index``
fields of the :ref:`Variables <variable>` passed as outputs to point to
itself and the rank they occupy in the output list. This whole
machinery builds a DAG (Directed Acyclic Graph) representing the
computation, a graph that Theano can compile and optimize.
Automatic wrapping
------------------
Traversing the graph
====================
All nodes in the graph must be instances of ``Apply`` or ``Result``, but
``<Op subclass>.make_node()`` typically wraps constants to satisfy those
constraints. For example, the :func:`tensor.add`
Op instance is written so that:
The graph can be traversed starting from outputs (the result of some
computation) down to its inputs using the owner field.
Take for example the following code:
.. testcode::
e = T.dscalar('x') + 1
builds the following graph:
>>> import theano
>>> x = theano.tensor.dmatrix('x')
>>> y = x * 2.
If you enter ``type(y.owner)`` you get ``<class 'theano.gof.graph.Apply'>``,
which is the apply node that connects the op and the inputs to get this
output. You can now print the name of the op that is applied to get
*y*:
>>> y.owner.op.name
'Elemwise{mul,no_inplace}'
Hence, an elementwise multiplication is used to compute *y*. This
multiplication is done between the inputs:
>>> len(y.owner.inputs)
2
>>> y.owner.inputs[0]
x
>>> y.owner.inputs[1]
DimShuffle{x,x}.0
Note that the second input is not 2 as we would have expected. This is
because 2 was first :term:`broadcasted <broadcasting>` to a matrix of
same shape as *x*. This is done by using the op ``DimShuffle`` :
>>> type(y.owner.inputs[1])
<class 'theano.tensor.var.TensorVariable'>
>>> type(y.owner.inputs[1].owner)
<class 'theano.gof.graph.Apply'>
>>> y.owner.inputs[1].owner.op # doctest: +SKIP
<theano.tensor.elemwise.DimShuffle object at 0x106fcaf10>
>>> y.owner.inputs[1].owner.inputs
[TensorConstant{2.0}]
.. testcode::
node = Apply(op=add,
inputs=[Variable(type=T.dscalar, name='x'),
Constant(type=T.lscalar, data=1)],
outputs=[Variable(type=T.dscalar)])
e = node.outputs[0]
Starting from this graph structure it is easier to understand how
*automatic differentiation* proceeds and how the symbolic relations
can be *optimized* for performance or stability.
Graph Structures
......@@ -224,7 +180,7 @@ An Apply instance can be created by calling ``gof.Apply(op, inputs, outputs)``.
.. _op:
--
Op
--
......@@ -242,16 +198,13 @@ structures, code going like ``def f(x): ...`` would produce an Op for
Apply node involving the ``f`` Op.
.. index::
single: Type
single: graph construct; Type
.. _type:
----
Type
----
......@@ -297,7 +250,6 @@ Theano Type.
--------
Variable
--------
......@@ -426,3 +378,77 @@ Sum{acc_dtype=float64} [id A] '' 1
>>> client
('output', 0)
>>> assert f.maker.fgraph.outputs[client[1]] is var
Automatic Differentiation
=========================
Having the graph structure, computing automatic differentiation is
simple. The only thing :func:`tensor.grad` has to do is to traverse the
graph from the outputs back towards the inputs through all *apply*
nodes (*apply* nodes are those that define which computations the
graph does). For each such *apply* node, its *op* defines
how to compute the *gradient* of the node's outputs with respect to its
inputs. Note that if an *op* does not provide this information,
it is assumed that the *gradient* is not defined.
Using the
`chain rule <http://en.wikipedia.org/wiki/Chain_rule>`_
these gradients can be composed in order to obtain the expression of the
*gradient* of the graph's output with respect to the graph's inputs .
A following section of this tutorial will examine the topic of :ref:`differentiation<tutcomputinggrads>`
in greater detail.
Optimizations
=============
When compiling a Theano function, what you give to the
:func:`theano.function <function.function>` is actually a graph
(starting from the output variables you can traverse the graph up to
the input variables). While this graph structure shows how to compute
the output from the input, it also offers the possibility to improve the
way this computation is carried out. The way optimizations work in
Theano is by identifying and replacing certain patterns in the graph
with other specialized patterns that produce the same results but are either
faster or more stable. Optimizations can also detect
identical subgraphs and ensure that the same values are not computed
twice or reformulate parts of the graph to a GPU specific version.
For example, one (simple) optimization that Theano uses is to replace
the pattern :math:`\frac{xy}{y}` by *x.*
Further information regarding the optimization
:ref:`process<optimization>` and the specific :ref:`optimizations<optimizations>` that are applicable
is respectively available in the library and on the entrance page of the documentation.
**Example**
Symbolic programming involves a change of paradigm: it will become clearer
as we apply it. Consider the following example of optimization:
>>> import theano
>>> a = theano.tensor.vector("a") # declare symbolic variable
>>> b = a + a ** 10 # build symbolic expression
>>> f = theano.function([a], b) # compile function
>>> print(f([0, 1, 2])) # prints `array([0,2,1026])`
[ 0. 2. 1026.]
>>> theano.printing.pydotprint(b, outfile="./pics/symbolic_graph_unopt.png", var_with_name_simple=True) # doctest: +SKIP
The output file is available at ./pics/symbolic_graph_unopt.png
>>> theano.printing.pydotprint(f, outfile="./pics/symbolic_graph_opt.png", var_with_name_simple=True) # doctest: +SKIP
The output file is available at ./pics/symbolic_graph_opt.png
We used :func:`theano.printing.pydotprint` to visualize the optimized graph
(right), which is much more compact than the unoptimized graph (left).
.. |g1| image:: ./pics/symbolic_graph_unopt.png
:width: 500 px
.. |g2| image:: ./pics/symbolic_graph_opt.png
:width: 500 px
================================ ====================== ================================
Unoptimized graph Optimized graph
================================ ====================== ================================
|g1| |g2|
================================ ====================== ================================
doc/extending/index.txt
浏览文件 @ 5a0d273c
......@@ -5,24 +5,35 @@
Extending Theano
================
This advanced tutorial is for users who want to extend Theano with new Types, new
Operations (Ops), and new graph optimizations.
Along the way, it also introduces many aspects of how Theano works, so it is
also good for you if you are interested in getting more under the hood with
Theano itself.
Before tackling this more advanced presentation, it is highly recommended to read the
introductory :ref:`Tutorial<tutorial>`.
The first few pages will walk you through the definition of a new :ref:`type`,
``double``, and a basic arithmetic :ref:`operations <op>` on that Type. We
will start by defining them using a Python implementation and then we will add
a C implementation.
This advanced tutorial is for users who want to extend Theano with new Types,
new Operations (Ops), and new graph optimizations. This first page of the
tutorial mainly focuses on the Python implementation of an Op and then
proposes an overview of the most important methods that define an op.
The second page of the tutorial (:ref:`extending_theano_c`) provides then
information on the C implementation of an Op. The rest of the tutorial
goes more in depth on advanced topics related to Ops, such as how to write
efficient code for an Op and how to write an optimization to speed up the
execution of an Op.
Along the way, this tutorial also introduces many aspects of how Theano works,
so it is also good for you if you are interested in getting more under the hood
with Theano itself.
.. note::
Before tackling this more advanced presentation, it is highly recommended
to read the introductory :ref:`Tutorial<tutorial>`, especially the sections
that introduce the Theano Graphs, as providing a novel Theano op requires a
basic understanting of the Theano Graphs.
See also the :ref:`dev_start_guide` for information regarding the
versioning framework, namely about *git* and *GitHub*, regarding the
development workflow and how to make a quality contribution.
.. toctree::
extending_theano
extending_theano_c
fibby
pipeline
theano_vs_c
......
doc/glossary.txt
浏览文件 @ 5a0d273c
......@@ -63,7 +63,7 @@ Glossary
then compiling them with :term:`theano.function`.
See also :term:`Variable`, :term:`Op`, :term:`Apply`, and
:term:`Type`, or read more about :ref:`tutorial_graphstructures`.
:term:`Type`, or read more about :ref:`graphstructures`.
Destructive
An :term:`Op` is destructive (of particular input[s]) if its
......@@ -108,7 +108,7 @@ Glossary
are provided with Theano, but you can add more.
See also :term:`Variable`, :term:`Type`, and :term:`Apply`,
or read more about :ref:`tutorial_graphstructures`.
or read more about :ref:`graphstructures`.
Optimizer
An instance of :class:`Optimizer`, which has the capacity to provide
......@@ -141,7 +141,7 @@ Glossary
``.type`` attribute of a :term:`Variable`.
See also :term:`Variable`, :term:`Op`, and :term:`Apply`,
or read more about :ref:`tutorial_graphstructures`.
or read more about :ref:`graphstructures`.
Variable
The the main data structure you work with when using Theano.
......@@ -153,7 +153,7 @@ Glossary
``x`` and ``y`` are both `Variables`, i.e. instances of the :class:`Variable` class.
See also :term:`Type`, :term:`Op`, and :term:`Apply`,
or read more about :ref:`tutorial_graphstructures`.
or read more about :ref:`graphstructures`.
View
Some Tensor Ops (such as Subtensor and Transpose) can be computed in
......
doc/tutorial/index.txt
浏览文件 @ 5a0d273c
......@@ -22,30 +22,54 @@ Throughout the tutorial, bear in mind that there is a :ref:`glossary` as well
as *index* and *modules* links in the upper-right corner of each page to help
you out.
Prerequisites
-------------
.. toctree::
python
numpy
Basics
------
.. toctree::
adding
examples
symbolic_graphs
printing_drawing
gradients
modes
loading_and_saving
conditions
loop
shape_info
Advanced
--------
.. toctree::
sparse
using_gpu
using_multi_gpu
gpu_data_convert
aliasing
shape_info
Advanced configuration and debugging
------------------------------------
.. toctree::
modes
printing_drawing
debug_faq
nan_tutorial
profiling
extending_theano
extending_theano_c
Further readings
----------------
.. toctree::
../extending/graphstructures
loading_and_saving
gpu_data_convert
aliasing
python-memory-management
multi_cores
faq_tutorial
doc/tutorial/symbolic_graphs.txt deleted 100644 → 0
浏览文件 @ 9ae1ab8e
.. _tutorial_graphstructures:
================
Graph Structures
================
Theano Graphs
=============
Debugging or profiling code written in Theano is not that simple if you
do not know what goes on under the hood. This chapter is meant to
introduce you to a required minimum of the inner workings of Theano.
For more detail see :ref:`extending`.
The first step in writing Theano code is to write down all mathematical
relations using symbolic placeholders (**variables**). When writing down
these expressions you use operations like ``+``, ``-``, ``**``,
``sum()``, ``tanh()``. All these are represented internally as **ops**.
An *op* represents a certain computation on some type of inputs
producing some type of output. You can see it as a *function definition*
in most programming languages.
Theano builds internally a graph structure composed of interconnected
**variable** nodes, **op** nodes and **apply** nodes. An
*apply* node represents the application of an *op* to some
*variables*. It is important to draw the difference between the
definition of a computation represented by an *op* and its application
to some actual data which is represented by the *apply* node. For more
detail about these building blocks refer to :ref:`variable`, :ref:`op`,
:ref:`apply`. Here is an example of a graph:
**Code**
.. testcode::
import theano.tensor as T
x = T.dmatrix('x')
y = T.dmatrix('y')
z = x + y
**Diagram**
.. _tutorial-graphfigure:
.. figure:: apply.png
:align: center
Interaction between instances of Apply (blue), Variable (red), Op (green),
and Type (purple).
.. # COMMENT
WARNING: hyper-links and ref's seem to break the PDF build when placed
into this figure caption.
Arrows in this figure represent references to the
Python objects pointed at. The blue
box is an :ref:`Apply` node. Red boxes are :ref:`Variable` nodes. Green
circles are :ref:`Ops <op>`. Purple boxes are :ref:`Types <type>`.
The graph can be traversed starting from outputs (the result of some
computation) down to its inputs using the owner field.
Take for example the following code:
>>> import theano
>>> x = theano.tensor.dmatrix('x')
>>> y = x * 2.
If you enter ``type(y.owner)`` you get ``<class 'theano.gof.graph.Apply'>``,
which is the apply node that connects the op and the inputs to get this
output. You can now print the name of the op that is applied to get
*y*:
>>> y.owner.op.name
'Elemwise{mul,no_inplace}'
Hence, an elementwise multiplication is used to compute *y*. This
multiplication is done between the inputs:
>>> len(y.owner.inputs)
2
>>> y.owner.inputs[0]
x
>>> y.owner.inputs[1]
DimShuffle{x,x}.0
Note that the second input is not 2 as we would have expected. This is
because 2 was first :term:`broadcasted <broadcasting>` to a matrix of
same shape as *x*. This is done by using the op ``DimShuffle`` :
>>> type(y.owner.inputs[1])
<class 'theano.tensor.var.TensorVariable'>
>>> type(y.owner.inputs[1].owner)
<class 'theano.gof.graph.Apply'>
>>> y.owner.inputs[1].owner.op # doctest: +SKIP
<theano.tensor.elemwise.DimShuffle object at 0x106fcaf10>
>>> y.owner.inputs[1].owner.inputs
[TensorConstant{2.0}]
Starting from this graph structure it is easier to understand how
*automatic differentiation* proceeds and how the symbolic relations
can be *optimized* for performance or stability.
Automatic Differentiation
=========================
Having the graph structure, computing automatic differentiation is
simple. The only thing :func:`tensor.grad` has to do is to traverse the
graph from the outputs back towards the inputs through all *apply*
nodes (*apply* nodes are those that define which computations the
graph does). For each such *apply* node, its *op* defines
how to compute the *gradient* of the node's outputs with respect to its
inputs. Note that if an *op* does not provide this information,
it is assumed that the *gradient* is not defined.
Using the
`chain rule <http://en.wikipedia.org/wiki/Chain_rule>`_
these gradients can be composed in order to obtain the expression of the
*gradient* of the graph's output with respect to the graph's inputs .
A following section of this tutorial will examine the topic of :ref:`differentiation<tutcomputinggrads>`
in greater detail.
Optimizations
=============
When compiling a Theano function, what you give to the
:func:`theano.function <function.function>` is actually a graph
(starting from the output variables you can traverse the graph up to
the input variables). While this graph structure shows how to compute
the output from the input, it also offers the possibility to improve the
way this computation is carried out. The way optimizations work in
Theano is by identifying and replacing certain patterns in the graph
with other specialized patterns that produce the same results but are either
faster or more stable. Optimizations can also detect
identical subgraphs and ensure that the same values are not computed
twice or reformulate parts of the graph to a GPU specific version.
For example, one (simple) optimization that Theano uses is to replace
the pattern :math:`\frac{xy}{y}` by *x.*
Further information regarding the optimization
:ref:`process<optimization>` and the specific :ref:`optimizations<optimizations>` that are applicable
is respectively available in the library and on the entrance page of the documentation.
**Example**
Symbolic programming involves a change of paradigm: it will become clearer
as we apply it. Consider the following example of optimization:
>>> import theano
>>> a = theano.tensor.vector("a") # declare symbolic variable
>>> b = a + a ** 10 # build symbolic expression
>>> f = theano.function([a], b) # compile function
>>> print(f([0, 1, 2])) # prints `array([0,2,1026])`
[ 0. 2. 1026.]
>>> theano.printing.pydotprint(b, outfile="./pics/symbolic_graph_unopt.png", var_with_name_simple=True) # doctest: +SKIP
The output file is available at ./pics/symbolic_graph_unopt.png
>>> theano.printing.pydotprint(f, outfile="./pics/symbolic_graph_opt.png", var_with_name_simple=True) # doctest: +SKIP
The output file is available at ./pics/symbolic_graph_opt.png
.. |g1| image:: ./pics/symbolic_graph_unopt.png
:width: 500 px
.. |g2| image:: ./pics/symbolic_graph_opt.png
:width: 500 px
We used :func:`theano.printing.pydotprint` to visualize the optimized graph
(right), which is much more compact than the unoptimized graph (left).
====================================================== =====================================================
Unoptimized graph Optimized graph
====================================================== =====================================================
|g1| |g2|
====================================================== =====================================================
theano/tensor/signal/pool.py
浏览文件 @ 5a0d273c
......@@ -47,7 +47,8 @@ def max_pool_2d_same_size(input, patch_size):
def pool_2d(input, ds, ignore_border=None, st=None, padding=(0, 0),
mode='max'):
"""
"""Downscale the input by a specified factor
Takes as input a N-D tensor, where N >= 2. It downscales the input image by
the specified factor, by keeping only the maximum value of non-overlapping
patches of size (ds[0],ds[1])
......
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论