a lot of documentation stuff

doc/conf.py

@@ -39,11 +39,11 @@ templates_path = ['.templates']
 source_suffix = '.txt'
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = 'contents'
 
 # General substitutions.
 project = 'theano'
-copyright = '2008, LISA lab'
+copyright = '2008-2009, LISA lab'
 
 # The default replacements for |version| and |release|, also used in various
 # other places throughout the built documents.

doc/contents.txt

+
+.. _contents:
+
+========
+Contents
+========
+
+.. toctree::
+   :maxdepth: 2
+
+   index
+   install
+   tutorials/index
+   advanced/index
+   indexes/index
+   examples/index
+   glossary
+   links
+   LICENSE
+
+

doc/glossary.txt

@@ -56,7 +56,7 @@ Glossary of terminology
 
         Examples of elementwise operations in Theano: ``add, sub, mul,
         div, neg, inv, log, exp, sin, cos, tan`` and many
-        others. These operations are all subclasses of :api:`Elemwise
+        others. These operations are all instances of :api:`Elemwise
         <theano.tensor.elemwise.Elemwise>`.
 
     graph
@@ -65,6 +65,9 @@ Glossary of terminology
     inplace
         WRITEME
 
+    merge
+        WRITEME
+
     op
         WRITEME
 

doc/index.txt

@@ -37,45 +37,100 @@ been Pythagoras' wife.
 
 Theano is released under a BSD license (:ref:`link <license>`)
 
-You can keep reading from :ref:`here <usingtheano>`.
+
+Sneak peek
+==========
+
+Here's a very simple example of how to use Theano.  It doesn't show
+off many of Theano's features, but it illustrates concretely what
+Theano is.
+
+.. code-block:: python
+
+    import theano
+    from theano import tensor
+
+    # declare two symbolic floating-point scalars
+    a = tensor.dscalar()
+    b = tensor.dscalar()
+
+    # create a simple expression
+    c = a + b
+
+    # convert the expression into a callable object that takes (a,b)
+    # values as input and computes a value for c
+    f = theano.function([a,b], c)
+
+    # bind 1.5 to 'a', 2.5 to 'b', and evaluate 'c'
+    assert 4.0 == f(1.5, 2.5)
+
+
+Theano is not a programming language in the normal sense because you
+write a program in Python that builds expressions for Theano.  Still
+it is like a programming language in the sense that to use theano, you
+have to
+
+- declare variables (``a,b``) and give their types
+
+- build expressions for how to put those variables together
+
+- compile expression graphs to functions in order to use them for computation.
+
+It is good to think of ``theano.function`` as the interface to a
+compiler which builds a callable object from a purely symbolic graph;
+one of theano's most important features is that ``theano.function``
+can optimize a graph and even compile some or all of it into native
+machine instructions.
+
+
+What does it do that they don't?
+================================
+
+Theano is a python library and optimizing compiler for manipulating
+and evaluating expressions, especially matrix-valued
+ones. Manipulation of matrices is typically done using the numpy
+package, so what does Theano do that Python and numpy do not?
+
+- *execution speed optimizations*: Theano can use `g++` to compile
+  parts your expression graph into native machine code, which runs
+  much faster than python.
+
+- *symbolic differentiation*: Theano can convert a symbolic graph
+  build symbolic graphs for computing gradients.
+
+- *stability optimizations*: Theano can recognize numerically unstable
+  expressions and compute them with more stable algorithms.
+
+There also exists symbolic packages in Python, namely sympy_. Theano
+is different from them in the sense that while it allows symbolic
+manipulation it puts more emphasis on the evaluation of these
+expressions and being able to repeatedly evaluate them on many
+different sets of inputs. It is also better suited to handling very
+large tensors which have no assumed structures.
+
+If numpy_ is to be compared to MATLAB_ and sympy_ to Mathematica_,
+Theano is a sort of hybrid of the two which tries to make the best of
+both worlds.
 
 
 
 Getting started
 ===============
 
-
 :ref:`install`
   Instructions to download and install Theano on your system.
 
-
 :ref:`basictutorial`
   Getting started with Theano's basic features. Go there if you are
   new!
 
-
 :ref:`advtutorial`
   This tutorial is for more advanced users who want to define their
   own operations and optimizations. It is recommended to go through
   the :ref:`basictutorial` first.
 
-
-
-Contents
-========
-
-.. toctree::
-   :maxdepth: 2
-
-   theano
-   install
-   tutorials/index
-   advanced/index
-   indexes/index
-   examples/index
-   glossary
-   links
-   LICENSE
+For a complete map of the documentation you may check the
+:ref:`contents`.
 
 
 Contact us
@@ -101,6 +156,10 @@ theano-dev_ mailing list.
 .. _numpy: http://numpy.scipy.org/
 .. _BLAS: http://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms
 
+.. _sympy: http://code.google.com/p/sympy/
+.. _MATLAB: http://www.mathworks.com/products/matlab/
+.. _Mathematica: http://www.wolfram.com/products/mathematica/index.html
+
 .. _theano-users: http://groups.google.com/group/theano-users?pli=1
 .. _theano-dev: http://groups.google.com/group/theano-dev?pli=1
 .. _task list: http://lgcm.iro.umontreal.ca/theano/query?status=accepted&status=assigned&status=new&status=reopened&group=milestone&max=200&col=id&col=summary&col=status&col=owner&col=type&col=priority&col=component&col=time&report=9&order=priority

doc/theano.txt

-
-.. _whatistheano:
-
-===============
-What is Theano?
-===============
-
-
-Introduction
-============
-
-Theano is a Python library aiming to allow definition, optimization
-and efficient evaluation of mathematical expressions involving
-multi-dimensional arrays (though it may be extended to support many
-other types). Theano melds some aspects of a computer algebra system
-(CAS) with aspects of an optimizing compiler. This is particularly
-useful in fields such as machine learning where complicated algorithms
-must be run over large amounts of data.
-
-Theano supports a wide range of numerical types in multiple
-dimensions, a rapidly growing number of well-tested operations as well
-as utilities to compute the gradient of an expression with respect to
-another. Symbolic expressions may be compiled into functions, which
-work merrily on the same data structures as numpy_, allowing for easy
-interoperability.
-
-Theano's compiler applies many optimizations of varying
-complexity. These optimizations include, but are not limited to
-constant folding, merging of similar subgraphs (to avoid calculating
-the same values more than once), simple arithmetic simplification
-(``x*y/x -> y``), inserting efficient BLAS_ operations and using
-inplace operations wherever it is safe to do so. Theano also defines
-several optimizations which improve the numerical stability of
-computations and it provides a framework to add and test new
-optimizers.
-
-Theano was written at the LISA_ to support the development of
-efficient machine learning algorithms while minimizing human
-time. Theano was named after the `Greek mathematician`_ who may have
-been Pythagoras' wife.
-
-Theano is released under a BSD license (:ref:`link <license>`)
-
-
-.. _usingtheano:
-
-Using Theano
-============
-
-Here's a very simple example of how to use Theano.  It doesn't show
-off many of Theano's features, but it illustrates concretely what
-Theano is.
-
-.. code-block:: python
-
-    import theano
-    from theano import tensor
-
-    # declare two symbolic floating-point scalars
-    a = tensor.dscalar()
-    b = tensor.dscalar()
-
-    # create a simple expression
-    c = a + b
-
-    # convert the expression into a callable object that takes (a,b)
-    # values as input and computes a value for c
-    f = theano.function([a,b], c)
-
-    # bind 1.5 to 'a', 2.5 to 'b', and evaluate 'c'
-    assert 4.0 == f(1.5, 2.5)
-
-
-Theano is not a programming language in the normal sense because you
-write a program in Python that builds expressions for Theano.  Still
-it is like a programming language in the sense that to use theano, you
-have to
-
-- declare variables (``a,b``) and give their types
-
-- build expressions for how to put those variables together
-
-- compile expression graphs to functions in order to use them for computation.
-
-It is good to think of ``theano.function`` as the interface to a
-compiler which builds a callable object from a purely symbolic graph;
-one of theano's most important features is that ``theano.function``
-can optimize a graph and even compile some or all of it into native
-machine instructions.
-
-
-What does it do that they don't?
-================================
-
-Theano is a python library and optimizing compiler for manipulating
-and evaluating expressions, especially matrix-valued
-ones. Manipulation of matrices is typically done using the numpy
-package, so what does Theano do that Python and numpy do not?
-
-- *execution speed optimizations*: Theano can use `g++` to compile
-  parts your expression graph into native machine code, which runs
-  much faster than python.
-
-- *symbolic differentiation*: Theano can convert a symbolic graph
-  build symbolic graphs for computing gradients.
-
-- *stability optimizations*: Theano can recognize numerically unstable
-  expressions and compute them with more stable algorithms.
-
-There also exists symbolic packages in Python, namely sympy_. Theano
-is different from them in the sense that while it allows symbolic
-manipulation it puts more emphasis on the evaluation of these
-expressions and being able to repeatedly evaluate them on many
-different sets of inputs. It is also better suited to handling very
-large tensors which have no assumed structures.
-
-If numpy_ is to be compared to MATLAB_ and sympy_ to Mathematica_,
-Theano is a sort of hybrid of the two which tries to make the best of
-both worlds.
-
-
-Getting Started
-===============
-
-:ref:`install`
-  Instructions to download and install Theano on your system.
-
-
-:ref:`basictutorial`
-  Getting started with Theano's basic features. Go there if you are new!
-
-
-:ref:`advtutorial`
-  This tutorial is for more advanced users who want to define their own
-  operations and optimizations. It is recommended to go through the
-  :ref:`basictutorial` first.
-
-
-
-
-.. _LISA:  http://www.iro.umontreal.ca/rubrique.php3?id_rubrique=27
-.. _Greek mathematician: http://en.wikipedia.org/wiki/Theano_(mathematician)
-.. _numpy: http://numpy.scipy.org/
-.. _BLAS: http://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms
-
-.. _sympy: http://code.google.com/p/sympy/
-.. _MATLAB: http://www.mathworks.com/products/matlab/
-.. _Mathematica: http://www.wolfram.com/products/mathematica/index.html
-

doc/tutorials/advanced/ex1/cop.txt

+
+====================================
+Implementing the arithmetic Ops in C
+====================================
+
+
+**Next:** `Example 2 - cons_cell`_
+
+.. _Example 2 - cons_cell: ../ex2/index.html

doc/tutorials/advanced/ex1/ctype.txt

+
+========================
+Implementing double in C
+========================
+
+
+**Next:** `Implementing the arithmetic Ops in C`_
+
+.. _Implementing the arithmetic Ops in C: cop.html

doc/tutorials/advanced/ex1/index.txt

 
-
-=========
-Example 1
-=========
-
-.. rubric:: Contents
+==================
+Example 1 - double
+==================
 
 .. toctree::
-   :maxdepth: 2
 
    type
    op
    ctype
    cop
 
-
+WRITEME

doc/tutorials/advanced/ex1/op.txt

+
+===============================
+Making arithmetic Ops on double
+===============================
+
+Now that we have a ``double`` type, we have yet to use it to perform
+computations. We'll start with defining multiplication.
+
+
+What is an Op?
+==============
+
+An :ref:`op` in Theano defines a certain computation on some types of
+inputs, producing some types of outputs. It is equivalent to a
+function definition in most programming languages. From a list of
+input :ref:`Results <result>` and an Op, you can build an :ref:`apply`
+node representing the application of the Op to the inputs.
+
+It is important to understand the distinction between the definition
+of a function (an Op) and the application of a function (an Apply
+node). If you were to interpret the Python language using Theano's
+structures, code going like ``def f(x): ...`` would produce an Op for
+``f`` whereas code like ``a = f(x)`` or ``g(f(4), 5)`` would produce
+an Apply node involving the ``f`` Op.
+
+
+
+Op's contract
+=============
+
+An Op is any object which defines the following methods:
+
+- **make_node(*inputs)**
+  
+  - This important function creates an Apply node representing the
+    application of the Op on the inputs provided. If the Op cannot be
+    applied on these inputs, it must raise an appropriate
+    exception. This method is also responsible for creating Results of
+    the suitable Type to serve as the outputs of the Op's application.
+
+- **__call__(*inputs)**
+
+  - Syntactic shortcut to make_node which returns the output Results
+    of the Op.
+
+  - *Default*: this is done for you by Op.
+
+- **perform(node, inputs, output_storage)**
+
+  - This function computes the function associated to this Op. The
+    ``node`` is an Apply node created by the Op's ``make_node``
+    method, the inputs are a list of references to data to operate on
+    and output_storage is a list of storage cells where the results of
+    the computation must be put.
+
+- **grad(inputs, output_gradients)** *Optional*
+
+  - If the Op you are defining is differentiable, you can define its
+    gradient symbolically in this method. Both the inputs and
+    output_gradients are Results and you must return one Result for
+    each input representing the gradient wrt these inputs provided
+    that the gradients wrt the outputs are computed by
+    output_gradients.
+
+
+For each method, the *default* is what the Op class defines for you.
+
+For more details you can go see the documentation for :ref:`op`.
+
+
+
+Defining mul
+============
+
+We are going to redefine the two functions that are absolutely
+necessary to redefine: ``make_node`` and ``perform``. First, we'll
+instantiate a ``mul`` Op:
+
+.. code-block:: python
+
+   from theano import gof
+   mul = gof.Op()
+
+
+**make_node**
+
+This function must take as many arguments as the operation we are
+defining is supposed to take as inputs - in this example that would be
+two (we'll define multiplication as a binary operation here, even
+though a multiplication Op could technically take an arbitrary number
+of arguments). It should ensure that both inputs have the ``double``
+type and it should make an Apply node with an output Result of type
+``double`` (since multiplying two doubles yields a double).
+
+.. code-block:: python
+
+   def make_node(x, y):
+       if x.type != double or y.type != double:
+           raise TypeError('mul only works on doubles')
+       return gof.Apply(mul, [x, y], [double()])
+   mul.make_node = make_node
+
+
+This is a pretty simple definition: the first two lines make sure that
+both inputs are Results of the ``double`` type that we created in the
+previous section. We would not want to multiply two arbitrary types,
+it would not make much sense (and we'd be screwed when we implement
+this in C!)
+
+The last line is the meat of the definition. There we create an Apply
+node representing the application of ``mul`` to ``x`` and ``y``. Apply
+takes three arguments: the first one is the Op we are applying. In
+this case, we are applying ``mul``. The second argument is a list of
+input Results - here, ``x`` and ``y``. The third is a list of output
+Results. Since the multiplication of two doubles ought to give us a
+double again, we create a Result of type ``double`` and we place it in
+a list. Since the list only has one element, ``mul`` only has one
+output.
+
+.. note::
+   Theano relies on the fact that if you call the ``make_node`` method
+   of Apply's first argument on the inputs passed as the Apply's
+   second argument, the call will not fail and the returned Apply
+   instance will be equivalent. We can see that this is trivially true
+   here.
+
+
+**perform**
+
+This code should actually compute the function. It is important to
+understand the role of all three arguments of ``perform``:
+
+- *node*: This is a reference to an Apply node which was previously
+  obtained via ``mul``'s ``make_node`` method. It is not typically
+  useful, but it contains symbolic information that could be required
+  for complex Ops.
+
+- *inputs*: This is a list of data. In this example, the data in
+  ``inputs`` will be instances of Python's built-in type ``float``
+  because this is the type that ``double.filter()`` will always
+  return, per our own definition.
+
+- *output_storage*: This is a list of storage cells. There is one
+  storage cell for each output of the Op. A storage cell is quite
+  simply a one-element list (note: it is forbidden to change the
+  length of the list(s) contained in output_storage). In this example,
+  output_storage will contain a single storage cell for the
+  multiplication's result.
+
+.. code-block:: python
+
+   def perform(node, inputs, output_storage):
+       x, y = inputs[0], inputs[1]
+       z = output_storage[0]
+       z[0] = x * y
+   mul.perform = perform
+
+Here, ``z`` is a list of one element. By default, ``z == [None]``.
+
+.. note::
+   It is possible that ``z`` does not contain ``None``. If it contains
+   anything else, Theano guarantees that whatever it contains is what
+   ``perform`` put there the last time it was called with this
+   particular storage. Furthermore, Theano gives you permission to do
+   whatever you want with ``z``'s contents, chiefly reusing it or the
+   memory allocated for it. More information can be found in the
+   :ref:`op` documentation.
+
+.. warning::
+   The data you put in the output_storage must match the type of the
+   symbolic output (this is a situation where the ``node`` argument
+   can come in handy). In the previous example, if you put, say, an
+   ``int`` in ``z[0]`` (even though we gave ``z`` the Theano type
+   ``double`` in ``make_node``, which means that a Python ``float``
+   must be put there) you might have nasty problems further down the
+   line since Theano often assumes Ops handle typing properly.
+
+
+Trying out our new Op
+=====================
+
+>>> x, y = double('x'), double('y')
+>>> z = mul(x, y)
+>>> f = theano.function([x, y], z)
+>>> f(5, 6)
+30.0
+>>> f(5.6, 6.7)
+37.519999999999996
+
+Seems to work. Note that there is an implicit call to
+``double.filter()`` on each argument, so if we give integers as inputs
+they are magically casted to the right type. Now, what if we try this?
+
+>>> x = double('x')
+>>> z = mul(x, 2)
+Traceback (most recent call last):
+  File "<stdin>", line 1, in <module>
+  File "/u/breuleuo/hg/theano/theano/gof/op.py", line 207, in __call__
+  File "<stdin>", line 2, in make_node
+AttributeError: 'int' object has no attribute 'type'
+
+Well, ok. We'd like our Op to be a bit more flexible. This can be done
+by fixing ``make_node`` a little bit:
+
+.. code-block:: python
+
+   def make_node(x, y):
+       if isinstance(x, (int, float)):
+           x = gof.Constant(double, x)
+       if isinstance(y, (int, float)):
+           y = gof.Constant(double, y)
+       if x.type != double or y.type != double:
+           raise TypeError('mul only works on doubles')
+       return gof.Apply(mul, [x, y], [double()])
+   mul.make_node = make_node
+
+Whenever we pass a Python int or float instead of a Result as ``x`` or
+``y``, make_node will convert it to :ref:`constant` for us. Constant
+is basically a :ref:`result` we statically know the value of.
+
+>>> x = double('x')
+>>> z = mul(x, 2)
+>>> f = theano.function([x], z)
+>>> f(10)
+20.0
+>>> f(3.4)
+6.7999999999999998
+
+And now it works the way we want it to.
+
+
+
+
+
+**Next:** `Implementing double in C`_
+
+.. _Implementing double in C: ctype.html
+
+
+

doc/tutorials/advanced/ex1/type.txt

 
 
-==========================
-Making the ``double`` type
-==========================
+======================
+Making the double type
+======================
 
-We will piggyback this type on Python's "float" type. Note that a
-Theano :ref:`type` is not equivalent to a Python type or
-class. Indeed, in Theano, :ref:`dmatrix` an :ref:`ivector` both use
-``numpy.ndarray`` as the working data type, yet they are different
-Types. The Python and C implementations may also use different types
-as long as a way to convert back and forth between them is provided.
 
+What is a Type?
+===============
 
-A :ref:`type` in Theano is any object which defines the following
-methods:
+A :ref:`type` in Theano, generally speaking, represents a set of
+constraints on potential data objects. These constraints allow Theano
+to tailor C code to handle them and to statically optimize the
+computation graph. For instance, the :ref:`irow <predefinedtypes>`
+type in the ``theano.tensor`` package gives the following constraints
+on the data the Results of type ``irow`` may contain:
 
-* **filter(value, strict [= False])**: this casts or wraps a value to
-  match the Type and returns the casted/wrapped value. If the value is
-  incompatible with the type, it must raise an exception. If strict is
-  True, filter must return a reference to ``value`` (i.e. casting
-  prohibited)
+#. Must be an instance of ``numpy.ndarray`` (``isinstance(x, numpy.ndarray)``)
+#. Must be an array of 32-bit integers (``str(x.dtype) == 'int32'``)
+#. Must have a shape of 1xN (``len(x.shape) == 2 and x.shape[0] == 1``)
 
-* **is_valid_value(value)**: returns True iff the value is exactly
-  compatible with the Type.
-    *Default*: defined in terms of ``filter(value, strict = True)``
+Knowing these restrictions, Theano may generate C code for addition,
+etc. which contains the right number of loops over the dimensions and
+declares the right data types.
 
-* **values_eq(a, b)**: returns True iff ``a`` and ``b`` are valid
-  values of this Type and are equal.
-    *Default*: a == b
+Note that a Theano :ref:`type` is not equivalent to a Python type or
+class. Indeed, in Theano, :ref:`irow <predefinedtypes>` and
+:ref:`dmatrix <predefinedtypes>` both use ``numpy.ndarray`` as the
+working data type, yet they are different Types. Indeed, the
+constraints set by ``dmatrix`` are:
 
-* **values_eq_approx(a, b)**: returns True iff ``a`` and ``b`` are
-  valid values of this Type and are approximately equal, for a
-  definition of approximately which varies from Type to Type.
-    *Default*: same as values_eq
+#. Must be an instance of ``numpy.ndarray`` (``isinstance(x, numpy.ndarray)``)
+#. Must be an array of 64-bit floating point numbers (``str(x.dtype) == 'float64'``)
+#. Must have a shape of MxN, no restriction on M or N (``len(x.shape) == 2``)
 
-* **make_result(name = None)**: makes a :term:`Result` of this Type
-  with the specified name. The Result will have its ``type`` field set
-  to the Type object.
-    *Default*: there is a generic definition of this in Type.
+These are different from ``irow``'s which I listed above. There are
+cases where a Type can fully correspond to a Python type (such as the
+``double`` type we will define here which corresponds to Python's
+``float``) but it's good to know that this is not always the case.
 
-* **__call__()**:
-  Syntactic shortcut to make_result.
-    *Default*: this is done for you by Type.
+
+Type's contract
+===============
+
+Concretely speaking, in Theano's framework, a Type is any object which
+defines the following methods:
+
+- **filter(value, strict [= False])**
+
+  - This casts or wraps a value to match the Type and returns the
+    casted/wrapped value. If the value is incompatible with the type,
+    it must raise an exception. If strict is True, filter must return a
+    reference to ``value`` (i.e. casting prohibited)
+
+- **is_valid_value(value)**
+
+  - Returns True iff the value is exactly compatible with the Type.
+
+  - *Default*: defined in terms of ``filter(value, strict = True)``
+
+- **values_eq(a, b)**
+
+  - Returns True iff ``a`` and ``b`` are valid values of this Type and
+    are equal.
+
+  - *Default*: a == b
+
+- **values_eq_approx(a, b)**
+
+  - Returns True iff ``a`` and ``b`` are valid values of this Type and
+    are approximately equal, for a definition of approximately which
+    varies from Type to Type.
+
+  - *Default*: same as values_eq
+
+- **make_result(name [= None])**
+
+  - Makes a :term:`Result` of this Type with the specified name. The
+    Result will have its ``type`` field set to the Type object.
+
+  - *Default*: there is a generic definition of this in Type.
+
+- **__call__()**:
+
+  - Syntactic shortcut to make_result.
+
+  - *Default*: this is done for you by Type.
 
 
 For each method, the *default* is what Type defines for you. This
 means you will rarely need to define all of these methods.
 
+For more details you can go see the documentation for :ref:`type`.
+
+
+Defining double
+===============
+
+We are going to piggyback ``double`` on Python's ``float``. We are
+going to redefine two functions: filter and values_eq_approx.
 
-In the past, some confusion has arised on what an instance of Type is
-versus a subclass of Type or an instance of Result. Some of this
-confusion is syntactic. A Type is any object which has fields
-corresponding to the functions defined above. The Type class provides
-sensible defaults for most of them but the most important one
-(filter), which means that in most cases all you have to do is:
+
+**filter**
 
 .. code-block:: python
 
-   def filter(x, strict = False):
-       ...
+   def filter(x, strict=False):
+       if strict and not isinstance(x, float):
+           raise TypeError('Expected a float!')
+       return float(x)
+
+We need to define ``filter`` with two arguments. The second argument
+must be called ``strict`` (Theano often calls it by keyword) and must
+have a default value of False.
 
-   mytype = theano.gof.Type()
-   mytype.filter = filter
+If ``strict == True`` we need to return ``x``. If ``x`` is not a
+``float`` (for example, ``x`` could easily be an ``int``) then it is
+incompatible with our Type and we are forced to raise an exception. If
+``strict == False`` then we are allowed to cast ``x`` to a ``float``,
+so if ``x`` is an ``int`` it we will return an equivalent ``float``.
 
-Another way to make a new Type is to subclass Type:
+
+**values_eq_approx**
 
 .. code-block:: python
 
-   class MyType(theano.gof.Type):
-       def filter(self, x, strict = False):
-           ...
-
-   mytype = MyType()
-
-The issue with defining a new Type like this is that all instances of
-MyType are technically the same Type since they filter in the same
-way. This is relevant because Theano often compares Types using ``==``
-to see if they are the same - for example, if the inputs of two
-different :ref:`applications <apply>` have the same Types and that the
-operation applied on them is the same, they can be :term:`merged
-<merge>`.  The workarounds are to define ``MyType.__eq__`` so that all
-instances of MyType are equal *or* to override ``MyType.__new__`` to
-always return the same instance *or* to hide MyType and only publish a
-single instance of it.
+   def values_eq_approx(x, y, tolerance=1e-4):
+       return abs(x - y) / (x + y) < tolerance
+
+The second function we are defining is ``values_eq_approx``. This
+method is meant to allow approximate comparison between two values
+respecting our Type's constraints. It might happen that an
+optimization changes the computation graph in such a way that it
+produces slightly different results due to numerical instability such
+as rounding errors at the end of the mantissa. For instance, ``a + a +
+a + a + a + a`` might not actually produce the exact same output as
+``6 * a`` (try with a=0.1), but we don't necessarily mind.
+
+We added an extra ``tolerance`` argument here. Since this argument is
+not part of the API, it must have a default value which we reasonably
+chose to be 1e-4.
+
+
+**Putting them together**
+
+What we want in the end is an object which respects the aforementioned
+contract. Any way to achieve this is fine, but one must not forget
+that Type defines standard, default implementations for most required
+methods of the interface. One way to make the Type is to just
+instantiate a plain Type and set the needed fields:
+
+.. code-block:: python
+
+   from theano import gof
+
+   double = gof.Type()
+   double.filter = filter
+   double.values_eq_approx = values_eq_approx
+
+
+Another is to make a subclass of Type and define filter and
+values_eq_approx there:
+
+.. code-block:: python
+
+   from theano import gof
+
+   class Double(gof.Type):
+   
+       def filter(self, x, strict=False):
+           if strict and not isinstance(x, float):
+               raise TypeError('Expected a float!')
+           return float(x)
+   
+       def values_eq_approx(self, x, y, tolerance=1e-4):
+           return abs(x - y) / (x + y) < tolerance
+   
+   double = Double()
+
+
+There is a small issue with defining double that way in that all
+instances of Double are technically the same Type. Indeed, they all
+filter in the same way. This is relevant because Theano often compares
+Types using ``==`` to see if they are the same - for example, if the
+inputs of two different :ref:`applications <apply>` have the same
+Types and that the operation applied on them is the same, they can be
+:term:`merged <merge>`.  The workarounds are to define
+``Double.__eq__`` so that all instances of Double are equal *or* to
+override ``Double.__new__`` to always return the same instance *or* to
+hide Double and only publish a single instance of it.
+
+
+Untangling some concepts
+========================
+
+From feedback I have gotten in the past, confusion is common on what
+an instance of Type is versus a subclass of Type or an instance of
+Result. Some of this confusion is syntactic. A Type is any object
+which has fields corresponding to the functions defined above. The
+Type class provides sensible defaults for most of them but the most
+important one (filter) so when defining new types it is natural to
+subclass Type. Therefore, we often end up with Type subclasses and it
+is not completely clear what these represent semantically. Here is an
+attempt to clear up the confusion:
+
+
+* An **instance of Type** is a set of constraints on real data. It is
+  akin to a primitive type or class in C and it is a *static*
+  annotation.
+
+* An **instance of Result** symbolizes data nodes in a data flow
+  graph. If you were to parse the C expression ``int x;``, ``int``
+  would be a Type instance and ``x`` would be a Result instance of
+  that Type instance. If you were to parse the C expression ``c = a +
+  b;``, ``a``, ``b`` and ``c`` would all be Result instances.
+
+* A **subclass of Type** represents a set of Type instances that share
+  structural similarities. In the ``double`` example that we are
+  doing, there is actually only one Type in that set, therefore the
+  subclass doesn't represent anything that one of its instances
+  doesn't. In this case it is a singleton. However, the Tensor class
+  which is a subclass of Type represents a set of types of tensors
+  parametrized by their data type or number of dimensions. We could
+  say that subclassing Type builds a hierarchy of Types which is based
+  upon structural similarity rather than compatibility.
+
+
+Final version
+=============
+
+.. code-block:: python
+
+   from theano import gof
+
+   class Double(gof.Type):
+   
+       def filter(self, x, strict=False):
+           if strict and not isinstance(x, float):
+               raise TypeError('Expected a float!')
+           return float(x)
+   
+       def values_eq_approx(self, x, y, tolerance=1e-4):
+           return abs(x - y) / (x + y) < tolerance
+
+       def __str__(self):
+           return "double"
+   
+   double = Double()
+
+
+I added one utility function, ``__str__``. That way, when we print
+``double`` it will print out something sensible!
+
+
+
+**Next:** `Making arithmetic Ops on double`_
+
+.. _Making arithmetic Ops on double: op.html
+
+
+
+
+
 
 

doc/tutorials/advanced/ex2/index.txt

 
-=========
-Example 2
-=========
 
-.. rubric:: Contents
+=====================
+Example 2 - cons_cell
+=====================
 
 .. toctree::
-   :maxdepth: 2
 
    type
+

doc/tutorials/advanced/ex2/type.txt

 
 
-==========================
-Making the ``cons`` type
-==========================
+====================
+Making the cons type
+====================
 
 WRITEME

doc/tutorials/advanced/index.txt

@@ -18,42 +18,13 @@ such as optimization.
 
 This tutorial should be of most use to users who want to extend Theano
 with custom types and operations related to these types. Users who
-want to extend Theano with new operations on tensors should check the
+want to extend Theano with new operations on tensors should check
 :ref:`tensoroptutorial`, but it is a good idea to read this tutorial
 as well since it probably provides better grounding for the many
 concepts at work here.
 
 
----------------------------------------
-
-`Example 1`_
-  Making a basic arithmetic system on doubles
-
-`Example 2`_
-  Making a higher-level type: ``cons`` (pair)
-
-`Views and inplace operations`_
-  A guide to making Ops that return a :term:`view` on their inputs or
-  operate :term:`inplace` on them.
-
-`Graph optimization`_
-  A guide to the different ways of defining new custom optimizations
-  to simplify the computation graph and/or improve its numerical
-  stability or other desirable properties.
-
-`Tips`_
-  Tips and tricks about writing types, ops and optimizations. This
-  page is good reference - check it and come back to it!
-
-`Wrapping up`_
-  A guide to what to look at next
-
----------------------------------------
-
-.. rubric:: Contents
-
 .. toctree::
-   :maxdepth: 2
 
    ex1/index
    ex2/index
@@ -63,14 +34,39 @@ concepts at work here.
    wrapup
 
 
-.. _Example 1: ex1/index.html
-.. _Example 2: ex2/index.html
-.. _Views and inplace operations: inplace.html
-.. _Graph optimization: optimization.html
-.. _Tips: tips.html
-.. _Wrapping up: wrapup.html
-
 
+..
+    `Example 1`_
+      Making a basic arithmetic system on doubles
+    
+    `Example 2`_
+      Making a higher-level type: ``cons_cell`` (pair)
+    
+    `Views and inplace operations`_
+      A guide to making Ops that return a :term:`view` on their inputs or
+      operate :term:`inplace` on them.
+    
+    `Graph optimization`_
+      A guide to the different ways of defining new custom optimizations
+      to simplify the computation graph and/or improve its numerical
+      stability or other desirable properties.
+    
+    `Tips`_
+      Tips and tricks about writing types, ops and optimizations. This
+      page is good reference - check it and come back to it!
+    
+    `Wrapping up`_
+      A guide to what to look at next
+    
+    
+    .. _Example 1: ex1/index.html
+    .. _Example 2: ex2/index.html
+    .. _Views and inplace operations: inplace.html
+    .. _Graph optimization: optimization.html
+    .. _Tips: tips.html
+    .. _Wrapping up: wrapup.html
+
+..
 
 
 

doc/tutorials/basic/adding.txt

 
-===========================
-Adding two numbers together
-===========================
+========================================
+Baby steps - Adding two numbers together
+========================================
 
 
 Adding two scalars
@@ -141,13 +141,9 @@ The following types are readily available:
    prefix vs the l prefix) and between 32 and 64 bit floats (f prefix
    vs the d prefix).
 
-
-Section
--------
-
-Try to mix and match them and see what happens. A complete list of
-types compatible with numpy arrays may be found :ref:`here
-<typelist>`.
+Try to mix and match them and see what happens. The previous list is
+not exhaustive. A guide to all types compatible with numpy arrays may
+be found :ref:`here <predefinedtypes>`.
 
 
 **Next:** `More examples`_

doc/tutorials/basic/examples.txt

@@ -9,8 +9,9 @@ More examples
 Logistic function
 =================
 
-Let's say that you want to compute the logistic curve, which is given
-by:
+Here's another straightforward example, though a bit more elaborate
+than adding two numbers together. Let's say that you want to compute
+the logistic curve, which is given by:
 
 .. math::
 
@@ -19,6 +20,8 @@ by:
 You want to compute the function :term:`elementwise` on matrices of
 doubles.
 
+Well, what you do is this:
+
 >>> x = T.dmatrix('x')
 >>> s = 1 / (1 + T.exp(-x))
 >>> logistic = function([x], s)
@@ -33,16 +36,17 @@ Computing more than one thing at the same time
 ==============================================
 
 Theano supports functions with multiple outputs. For example, we can
-compute the absolute :term:`elementwise` difference between two
+compute the :term:`elementwise` absolute difference between two
 matrices ``x`` and ``y`` and the squared difference at the same time:
 
 >>> x, y = T.dmatrices('xy')
->>> d = x - y
->>> f = function([x, y], [abs(d), d**2])
+>>> diff = x - y
+>>> abs_diff = abs(x - y)
+>>> diff_squared = diff**2
+>>> f = function([x, y], [abs_diff, diff_squared])
 
-Theano will make ``f`` in such a way that it will only compute the
-difference once. When we use the function, it will return the two
-results (reformatted for readability):
+When we use the function, it will return the two results (the printing
+was reformatted for readability):
 
 >>> f([[1, 1], [1, 1]], [[0, 1], [2, 3]])
 [array([[ 1.,  0.],

doc/tutorials/basic/index.txt

@@ -25,42 +25,11 @@ of theano. Let's import that subpackage under a handy name. I like
 
 Now we're ready for the tour:
 
----------------------------------------
-
-`Adding two numbers together`_
-  Starting small
-
-`More examples`_
-  Getting comfortable
-
-`Using Module`_
-  Getting serious
-
-`Tools`_
-  An overview of what you have access to
-
-`Wrapping up`_
-  A guide to what to look at next
-
----------------------------------------
-
-
-.. rubric:: Contents
 
 .. toctree::
-   :maxdepth: 2
 
    adding
    examples
    module
    tools
    wrapup
-
-
-
-.. _Adding two numbers together: adding.html
-.. _More examples: examples.html
-.. _Using Module: module.html
-.. _Tools: tools.html
-.. _Wrapping up: wrapup.html
-

doc/tutorials/basic/tools.txt

@@ -16,6 +16,8 @@ Types
 NOTE: I'm not sure this actually goes in the tutorial - it ended up
 much longer than intended - maybe we should just link to it! --OB
 
+.. _predefinedtypes:
+
 Predefined types
 ----------------
 
@@ -102,6 +104,14 @@ complex64   complex          64 (two float32)
 complex128  complex          128 (two float64)
 =========== ================ =================
 
+.. note::
+
+   There are no premade complex types, so you need to make them
+   explicitly with Tensor. Furthermore, few operations are fully
+   supported for complex types: as of version 0.1, only elementary
+   operations (``+-*/``) have C implementations.
+
+
 The broadcastable pattern, on the other hand, indicates both the
 number of dimensions and whether a particular dimension has length
 1. Here is a handy table mapping the :term:`broadcastable

doc/tutorials/howtotest.txt

+
+.. _howtotest:
+
+===========
+How to test
+===========
+
+
+How to test an Op
+=================
+
+blah blah WRITEME
+
+
+How to test an Optimizer
+========================
+
+yadda WRITEME yadda
+

doc/tutorials/index.txt

@@ -4,10 +4,10 @@ Tutorials
 =========
 
 .. toctree::
-   :maxdepth: 2
 
    basic/index
    advanced/index
    tensorop
    tensoroptools
+   howtotest