So suppose you have looked through the library documentation and you don't see
Extending Theano
a function that does what you want.
================
This tutorial covers how to extend Theano with novel ops. It mainly focuses on ops that offer a Python implementation, refers to :ref:`extending_theano_c` for C-based op.
If you can implement something in terms of existing Ops, you should do that.
The first section of this tutorial introduces the Theano Graphs,
Odds are your function that uses existing Theano expressions is short,
as providing a novel Theano op requires a basic understanting of the Theano Graphs. It then proposes an overview of the most important methods that define an op.
has no bugs, and potentially profits from optimizations that have already been
implemented.
As an illustration, this tutorial shows how to write a simple Python-based op which performs operations on Double. It also shows how to implement tests that ensure the proper working of an op.
However, if you cannot implement an Op in terms of existing Ops, you have to
write a new one. Don't worry, Theano was designed to make it easy to add new
Ops, Types, and Optimizations.
.. These first few pages will walk you through the definition of a new :ref:`type`,
.. ``double``, and a basic arithmetic :ref:`operations <op>` on that Type.
As an illustration, this tutorial shows how to write a simple Python-based
:ref:`operations <op>` which performs operations on
:ref:`type`, ``double<Double>``.
.. It also shows how to implement tests that
.. ensure the proper working of an op.
.. note::
.. note::
This tutorial does not cover how to make an op that returns a view or
This is an introductury tutorial and as such it does not cover how to make
modifies the values in its inputs. Thus, all ops created with the
an op that returns a view or modifies the values in its inputs. Thus, all
instructions described here MUST return newly allocated
ops created with the instructions described here MUST return newly
memory or reuse the memory provided in the parameter
allocated memory or reuse the memory provided in the parameter
``output_storage`` of the :func:`perform` function. See :ref:`views_and_inplace`
``output_storage`` of the :func:`perform` function. See
for an explanation on how to do this.
:ref:`views_and_inplace` for an explanation on how to do this.
If your op returns a view or changes the value of its inputs
If your op returns a view or changes the value of its inputs
without doing as prescribed in that page, Theano will run, but will
without doing as prescribed in that page, Theano will run, but will
...
@@ -28,31 +39,32 @@ As an illustration, this tutorial shows how to write a simple Python-based op wh
...
@@ -28,31 +39,32 @@ As an illustration, this tutorial shows how to write a simple Python-based op wh
``mode=DebugMode``) since it verifies if your op behaves correctly in this
``mode=DebugMode``) since it verifies if your op behaves correctly in this
regard.
regard.
.. note::
See the :ref:`dev_start_guide` for information regarding the versioning
framework, namely about *git* and *GitHub*, regarding the development workflow and
Theano represents symbolic mathematical computations as graphs. Those graphs are bi-partite graphs (graphs with 2 types of nodes), they are composed of interconnected :ref:`apply` and :ref:`variable` nodes.
Theano represents symbolic mathematical computations as graphs. Those graphs
:ref:`variable` nodes represent data in the graph, either inputs, outputs or intermediary values. As such, Inputs and Outputs of a graph are lists of Theano :ref:`variable` nodes. :ref:`apply` nodes perform computation on these variables to produce new variables. Each :ref:`apply` node has a link to an instance of :ref:`Op` which describes the computation to perform. This tutorial details how to write such an Op instance. Please refers to :ref:`graphstructures` for a more detailed explanation about the graph structure.
are bi-partite graphs (graphs with 2 types of nodes), they are composed of
interconnected :ref:`apply` and :ref:`variable` nodes.
:ref:`variable` nodes represent data in the graph, either inputs, outputs or
intermediary values. As such, Inputs and Outputs of a graph are lists of Theano
:ref:`variable` nodes. :ref:`apply` nodes perform computation on these
variables to produce new variables. Each :ref:`apply` node has a link to an
instance of :ref:`Op` which describes the computation to perform. This tutorial
details how to write such an Op instance. Please refers to
:ref:`graphstructures` for a more detailed explanation about the graph
structure.
Op's basic methods
Op Structure
------------------
============
An op is any Python object which inherits from :class:`gof.Op`.
An op is any Python object which inherits from :class:`gof.Op`.
This section provides an overview of the methods you typically have to implement to make a new op. It does not provide extensive coverage of all the
This section provides an overview of the basic methods you typically have to
implement to make a new op. It does not provide extensive coverage of all the
possibilities you may encounter or need. For that refer to
possibilities you may encounter or need. For that refer to
:ref:`op_contract`.
:ref:`op_contract`.
...
@@ -102,10 +114,13 @@ possibilities you may encounter or need. For that refer to
...
@@ -102,10 +114,13 @@ possibilities you may encounter or need. For that refer to
def infer_shape(node, input_shapes):
def infer_shape(node, input_shapes):
pass
pass
.. ../extending/op.txt
An op has to implement some methods defined in the the interface of
An op has to implement some methods defined in the the interface of
:class:`gof.Op`. More specifically, it is mandatory for an op to define either the method :func:`make_node` or :attr:`itypes`, :attr:`otypes` and one of the implementation methods, either :func:`perform`, :meth:`Op.c_code` or :func:`make_thunk`.
:class:`gof.Op`. More specifically, it is mandatory for an op to define either
the method :func:`make_node` or :attr:`itypes`, :attr:`otypes` and one of the
implementation methods, either :func:`perform`, :meth:`Op.c_code`
or :func:`make_thunk`.
method :func:`make_node` and one of the implementation methods, either
:func:`perform`, :meth:`Op.c_code` or :func:`make_thunk`.
:func:`make_node` method creates an Apply node representing the application
:func:`make_node` method creates an Apply node representing the application
of the op on the inputs provided. This method is reponsible for three things:
of the op on the inputs provided. This method is reponsible for three things:
...
@@ -118,7 +133,8 @@ An op has to implement some methods defined in the the interface of
...
@@ -118,7 +133,8 @@ An op has to implement some methods defined in the the interface of
the symbolic output Variables. It creates output Variables of a suitable
the symbolic output Variables. It creates output Variables of a suitable
symbolic Type to serve as the outputs of this op's
symbolic Type to serve as the outputs of this op's
application.
application.
- it creates an Apply instance with the input and output Variable, and return the Apply instance.
- it creates an Apply instance with the input and output Variable, and
return the Apply instance.
...
@@ -151,7 +167,8 @@ An op has to implement some methods defined in the the interface of
...
@@ -151,7 +167,8 @@ An op has to implement some methods defined in the the interface of
For instance, it is possible to define :meth:`Op.c_code` to provide a
For instance, it is possible to define :meth:`Op.c_code` to provide a
C-implementation to the op. Please refers to tutorial
C-implementation to the op. Please refers to tutorial
:ref:`extending_theano_c` for a description of :meth:`Op.c_code` and other
:ref:`extending_theano_c` for a description of :meth:`Op.c_code` and other
related c_methods. Note that an op can provide both Python and C implementation.
related c_methods. Note that an op can provide both Python and C
implementation.
:func:`make_thunk` method is another alternative to :func:`perform`.
:func:`make_thunk` method is another alternative to :func:`perform`.
It returns a thunk. A thunk is defined as a zero-arguments
It returns a thunk. A thunk is defined as a zero-arguments
...
@@ -186,7 +203,10 @@ An op has to implement some methods defined in the the interface of
...
@@ -186,7 +203,10 @@ An op has to implement some methods defined in the the interface of
are used by the Op's :func:`make_node` method to implement the functionality
are used by the Op's :func:`make_node` method to implement the functionality
of :func:`make_node` method mentioned above.
of :func:`make_node` method mentioned above.
Other methods can be optionally defined by the op.
Op's auxiliary methods
----------------------
There are other methods that can be optionally defined by the op:
The :func:`__str__` method provides a meaningful string representation of
The :func:`__str__` method provides a meaningful string representation of
your op.
your op.
...
@@ -216,18 +236,19 @@ Other methods can be optionally defined by the op.
...
@@ -216,18 +236,19 @@ Other methods can be optionally defined by the op.
:attr:`__props__` will also generate a suitable :func:`__str__` for your op.
:attr:`__props__` will also generate a suitable :func:`__str__` for your op.
This requires development version after September 1st, 2014 or version 0.7.
This requires development version after September 1st, 2014 or version 0.7.
The :func:`infer_shape` method allows to infer the shape of the op
The :func:`infer_shape` method allows to infer the shape of the op
output variables, without actually computing the outputs.
output variables, without actually computing the outputs.
It takes as input ``node``, a reference to the op Apply node,
It takes as input ``node``, a reference to the op Apply node,
and a list of Theano symbolic Varables (``i0_shape``, ``i1_shape``, ...)
and a list of Theano symbolic Varables (``i0_shape``, ``i1_shape``, ...)
which are the shape of the op input Variables.
which are the shape of the op input Variables.
:func:`infer_shape` returns a list where each element is a tuple representing the shape of one output.
:func:`infer_shape` returns a list where each element is a tuple representing
the shape of one output.
This could be helpful if one only
This could be helpful if one only
needs the shape of the output instead of the actual outputs, which
needs the shape of the output instead of the actual outputs, which
can be useful, for instance, for optimization procedures.
can be useful, for instance, for optimization procedures.
The :func:`grad` method is required if you want to differentiate some cost whose expression includes your op. The gradient may be
The :func:`grad` method is required if you want to differentiate some cost
whose expression includes your op. The gradient may be
specified symbolically in this method. It takes two arguments ``inputs`` and
specified symbolically in this method. It takes two arguments ``inputs`` and
``output_gradients`` which are both lists of symbolic Theano Variables and
``output_gradients`` which are both lists of symbolic Theano Variables and
those must be operated on using Theano's symbolic language. The grad
those must be operated on using Theano's symbolic language. The grad
...
@@ -242,7 +263,6 @@ Other methods can be optionally defined by the op.
...
@@ -242,7 +263,6 @@ Other methods can be optionally defined by the op.
NullType for that input. Please refer to :func:`grad` for a more detailed
NullType for that input. Please refer to :func:`grad` for a more detailed
view.
view.
The :func:`R_op` method is needed if you want ``theano.tensor.Rop`` to
The :func:`R_op` method is needed if you want ``theano.tensor.Rop`` to
work with your op.
work with your op.
This function implements the application of the R-operator on the
This function implements the application of the R-operator on the
...
@@ -257,9 +277,8 @@ Other methods can be optionally defined by the op.
...
@@ -257,9 +277,8 @@ Other methods can be optionally defined by the op.
(particularly for scalars) and reduce the number of generated C files.
(particularly for scalars) and reduce the number of generated C files.
Example: Op definition
Op Example
----------------------
==========
.. testcode:: example
.. testcode:: example
...
@@ -276,8 +295,11 @@ Op Example
...
@@ -276,8 +295,11 @@ Op Example
# This next line looks like it has a typo,
# This next line looks like it has a typo,
# but it's actually a way to detect the theano version
# but it's actually a way to detect the theano version
# is sufficiently recent to support the use of __props__.
# is sufficiently recent to support the use of __props__.
assert hasattr(self, '_props'), "Your version of theano is too old to support __props__."
assert hasattr(self, '_props'), "Your version of theano is too old
to support __props__."
x = theano.tensor.as_tensor_variable(x)
x = theano.tensor.as_tensor_variable(x)
# Note: using x_.type() is dangerous, as it copies x's broadcasting
# behaviour
return theano.Apply(self, [x], [x.type()])
return theano.Apply(self, [x], [x.type()])
def perform(self, node, inputs, output_storage):
def perform(self, node, inputs, output_storage):
...
@@ -300,6 +322,8 @@ Op Example
...
@@ -300,6 +322,8 @@ Op Example
return eval_points
return eval_points
return self.grad(inputs, eval_points)
return self.grad(inputs, eval_points)
doubleOp1 = DoubleOp1()
#Using itypes and otypes
#Using itypes and otypes
...
@@ -329,7 +353,66 @@ Op Example
...
@@ -329,7 +353,66 @@ Op Example
return eval_points
return eval_points
return self.grad(inputs, eval_points)
return self.grad(inputs, eval_points)
You can try it as follows:
doubleOp2 = DoubleOp2()
At a high level, the code fragment declares a class (e.g., ``DoubleOp1``) and then
creates one instance of it (e.g., ``doubleOp1``).
We often gloss over this distinction, but will be precise here:
``doubleOp1`` (the instance) is an Op, not ``DoubleOp1`` (the class which is a
subclass of ``theano.Op``). You can call ``doubleOp1(tensor.vector())`` on a
Variable to build an expression, and in the expression there will be
a ``.op`` attribute that refers to ``doubleOp1``.
.. The first two methods in the Op are relatively boilerplate: ``__eq__``
.. and ``__hash__``.
.. When two Ops are equal, Theano will merge their outputs if they are applied to the same inputs.
.. The base class (Op) says two objects are equal if (and only if)
.. they are the same object.
.. Writing these boilerplate definitions ensures that the logic of the equality comparison is always explicit.
.. It is an essential part of the :ref:`op_contract` that if two Ops compare
.. equal, then they must compute the same result when presented with the same
.. inputs. Here, if we allocated another instance of ``Fibby`` by typing ``fibby2
.. = Fibby()`` then we would have two Ops that behave identically.
..
.. When should the implementation of ``__eq__`` be more complicated?
.. If ``Fibby.__init__`` had parameters, then we could
.. have configured ``fibby2`` differently from ``fibby`` by passing different
.. arguments to the constructor. If we had done that, and if that different
.. configuration made ``fibby2`` compute different results from ``fibby`` (for the
.. same inputs) then we would have to add logic to the ``__eq__`` and ``__hash__``
.. function so that he two ``Fibby`` Ops would *not be equal*. The reason why: Theano's merge
.. optimization looks for Ops comparing equal and merges them. If two Ops compare
.. equal but don't always produce equal results from equal inputs, then you might
.. see wrong calculation.
The ``make_node`` method creates a node to be included in the expression graph.
It runs when we apply our Op (``doubleOp1``) to the Variable (``x``), as
in ``doubleOp1(tensor.vector())``.
When an Op has multiple inputs, their order in the inputs argument to ``Apply``
is important: Theano will call ``make_node(*inputs)`` to copy the graph,
so it is important not to change the semantics of the expression by changing
the argument order.
All the ``inputs`` and ``outputs`` arguments to ``Apply`` must be Variables.
A common and easy way to ensure inputs are variables is to run them through
``as_tensor_variable``. This function leaves TensorType variables alone, raises
an error for non-TensorType variables, and copies any ``numpy.ndarray`` into
the storage for a TensorType Constant. The ``make_node`` method dictates the
appropriate Type for all output variables.
The ``perform`` method implements the Op's mathematical logic in Python.
The inputs (here ``x``) are passed by value, but a single output is returned
indirectly as the first element of single-element lists. If ``doubleOp1`` had
a second output, it would be stored in ``output_storage[1][0]``.
.. jpt: DOn't understand the following
In some execution modes, the output storage might contain the return value of
a previous call. That old value can be reused to avoid memory re-allocation,
but it must not influence the semantics of the Op output.
You can try the new Op as follows:
.. testcode:: example(Using make_node)
.. testcode:: example(Using make_node)
...
@@ -395,8 +478,8 @@ You can try it as follows:
...
@@ -395,8 +478,8 @@ You can try it as follows:
[ 0.48165539 0.98642904 0.4913309 0.30702264]]
[ 0.48165539 0.98642904 0.4913309 0.30702264]]
Example for properties of a Op
Example: __props__ definition
==============================
-----------------------------
We can modify the previous piece of code in order to demonstrate
We can modify the previous piece of code in order to demonstrate
the usage of the :attr:`__props__` attribute.
the usage of the :attr:`__props__` attribute.
...
@@ -422,7 +505,8 @@ and ``b`` are equal.
...
@@ -422,7 +505,8 @@ and ``b`` are equal.
def make_node(self, x):
def make_node(self, x):
# check that the theano version has support for __props__.
# check that the theano version has support for __props__.
assert hasattr(self, '_props'), "Your version of theano is too old to support __props__."
assert hasattr(self, '_props'), "Your version of theano is too old
to support __props__."
x = theano.tensor.as_tensor_variable(x)
x = theano.tensor.as_tensor_variable(x)
return theano.Apply(self, [x], [x.type()])
return theano.Apply(self, [x], [x.type()])
...
@@ -439,8 +523,9 @@ and ``b`` are equal.
...
@@ -439,8 +523,9 @@ and ``b`` are equal.
The use of :attr:`__props__` saves
The use of :attr:`__props__` saves
the user the trouble of implementing :func:`__eq__` and :func:`__hash__` manually.
the user the trouble of implementing :func:`__eq__` and :func:`__hash__`
It also generates a default :func:`__str__` method that prints the attribute names and their values.
manually. It also generates a default :func:`__str__` method that prints the
attribute names and their values.
We can test this by running the following segment:
We can test this by running the following segment:
...
@@ -464,14 +549,14 @@ We can test this by running the following segment:
...
@@ -464,14 +549,14 @@ We can test this by running the following segment:
How To Test it
How To Test it
==============
--------------
Theano has some functionalities to simplify testing. These help test the
Theano has some functionalities to simplify testing. These help test the
``infer_shape``, ``grad`` and ``R_op`` methods. Put the following code
``infer_shape``, ``grad`` and ``R_op`` methods. Put the following code
in a file and execute it with the ``theano-nose`` program.
in a file and execute it with the ``theano-nose`` program.
Basic Tests
Basic Tests
-----------
^^^^^^^^^^^
Basic tests are done by you just by using the op and checking that it
Basic tests are done by you just by using the op and checking that it
returns the right answer. If you detect an error, you must raise an
returns the right answer. If you detect an error, you must raise an
...
@@ -507,7 +592,7 @@ defaul value do the most strict comparison, 1 and 2 make less strict
...
@@ -507,7 +592,7 @@ defaul value do the most strict comparison, 1 and 2 make less strict
comparison.
comparison.
Testing the infer_shape
Testing the infer_shape
-----------------------
^^^^^^^^^^^^^^^^^^^^^^^
When a class inherits from the ``InferShapeTester`` class, it gets the
When a class inherits from the ``InferShapeTester`` class, it gets the
``self._compile_and_check`` method that tests the op's ``infer_shape``
``self._compile_and_check`` method that tests the op's ``infer_shape``
...
@@ -555,7 +640,7 @@ your op works only with such matrices, you can disable the warning with the
...
@@ -555,7 +640,7 @@ your op works only with such matrices, you can disable the warning with the
self.op_class)
self.op_class)
Testing the gradient
Testing the gradient
--------------------
^^^^^^^^^^^^^^^^^^^^
The function :ref:`verify_grad <validating_grad>`
The function :ref:`verify_grad <validating_grad>`
verifies the gradient of an op or Theano graph. It compares the
verifies the gradient of an op or Theano graph. It compares the
...
@@ -573,7 +658,7 @@ the multiplication by 2).
...
@@ -573,7 +658,7 @@ the multiplication by 2).
[numpy.random.rand(5, 7, 2)])
[numpy.random.rand(5, 7, 2)])
Testing the Rop
Testing the Rop
---------------
^^^^^^^^^^^^^^^
.. TODO: repair defective links in the following paragraph
.. TODO: repair defective links in the following paragraph
...
@@ -595,24 +680,22 @@ For instance, to verify the Rop method of the DoubleOp, you can use this:
...
@@ -595,24 +680,22 @@ For instance, to verify the Rop method of the DoubleOp, you can use this:
