Merge pull request #3924 from fvisin/fix_tutorial

doc/cifarSC2011/index.txt

@@ -66,7 +66,7 @@ from gurus on hand if you get stuck.
     introduction
     theano
     advanced_theano
-    /tutorial/extending_theano
+    /extending/extending_theano
     pyCUDA
     gpundarray
 

doc/crei2013/index.txt

@@ -69,4 +69,4 @@ from gurus on hand if you get stuck.
     theano
     advanced_theano
     gpundarray
-    /tutorial/extending_theano
+    /extending/extending_theano

doc/tutorial/extending_theano_c.txt → doc/extending/extending_theano_c.txt

@@ -410,6 +410,36 @@ commonly used.
         this function should return a tuple of integers as previously
         described.
 
+Important restrictions when implementing an Op
+==============================================
+
+There are some important restrictions to remember when implementing an Op.
+Unless your Op correctly defines a ``view_map`` attribute, the ``perform`` and ``c_code`` must not
+produce outputs whose memory is aliased to any input (technically, if changing the
+output could change the input object in some sense, they are aliased).
+Unless your Op correctly defines a ``destroy_map`` attribute, ``perform`` and ``c_code`` must
+not modify any of the inputs.
+
+TODO: EXPLAIN DESTROYMAP and VIEWMAP BETTER AND GIVE EXAMPLE.
+
+When developing an Op, you should run computations in DebugMode, by using
+argument ``mode='DebugMode'`` to ``theano.function``. DebugMode is
+slow, but it can catch many common violations of the Op contract.
+
+TODO: Like what? How? Talk about Python vs. C too.
+
+DebugMode is no silver bullet though.
+For example, if you modify an Op ``self.*`` during any of
+``make_node``, ``perform``, or ``c_code``, you are probably doing something
+wrong but DebugMode will not detect this.
+
+TODO: jpt: I don't understand the following sentence.
+
+Ops and Types should usually be considered immutable -- you should
+definitely not make a change that would have an impact on ``__eq__``,
+``__hash__``, or the mathematical value that would be computed by  ``perform``
+or ``c_code``.
+
 
 Simple C Op example
 ===================
@@ -526,6 +556,11 @@ storage with the right shape and number of dimensions.
             return c_code % locals()
 
 
+The ``c_code`` method accepts variable names as arguments (``name``, ``inp``,
+``out``, ``sub``) and returns a C code fragment that computes the expression 
+output. In case of error, the ``%(fail)s`` statement cleans up and returns 
+properly.
+
 More complex C Op example
 =========================
 

doc/extending/fibby.txt

@@ -3,19 +3,6 @@
 Writing an Op to work on an ``ndarray`` in C
 =============================================
 
-So suppose you have looked through the library documentation and you don't see a
-function that does what you want.
-
-If you can implement something in terms of existing Ops, you should do that.
-Odds are your function that uses existing Theano expressions is short,
-has no bugs, and potentially profits from optimizations that have already been
-implemented.
-
-However, if you cannot implement an Op in terms of existing Ops, you have to
-write a new one.
-Don't worry,
-Theano was designed to make it easy to add new Ops, Types, and Optimizations.
-
 This section walks through a non-trivial example Op that does something pretty
 weird and unrealistic, that is hard to express with existing Ops.
 (Technically, we could use ``Scan`` to implement the Op we're about to describe,
@@ -73,73 +60,12 @@ you should check the strides and alignment.
 
    fibby = Fibby()
 
-At a high level, the code fragment declares a class (``Fibby``) and then
-creates one instance of it (``fibby``).
-We often gloss over this distinction, but will be precise here:
-``fibby`` (the instance) is an Op, not ``Fibby`` (the class which is a subclass of ``theano.Op``).
-You can call ``fibby(tensor.vector())`` on a Variable to build an
-expression, and in the expression there will be a ``.op`` attribute that refers
-to ``fibby``.
-
-The first two methods in the Op are relatively boilerplate: ``__eq__`` and ``__hash__``.
-When two Ops are equal, Theano will merge their outputs if they are applied to the same inputs.
-The base class (Op) says two objects are equal if (and only if)
-they are the same object.
-Writing these boilerplate definitions ensures that the logic of the equality comparison is always explicit.
-
-It is an essential part of the :ref:`op_contract` that if two Ops compare
-equal, then they must compute the same result when presented with the same
-inputs.  Here, if we allocated another instance of ``Fibby`` by typing ``fibby2
-= Fibby()`` then we would have two Ops that behave identically.
-
-When should the implementation of ``__eq__`` be more complicated?
-If ``Fibby.__init__`` had parameters, then we could
-have configured ``fibby2`` differently from ``fibby`` by passing different
-arguments to the constructor. If we had done that, and if that different
-configuration made ``fibby2`` compute different results from ``fibby`` (for the
-same inputs) then we would have to add logic to the ``__eq__`` and ``__hash__``
-function so that he two ``Fibby`` Ops would *not be equal*.  The reason why: Theano's merge
-optimization looks for Ops comparing equal and merges them. If two Ops compare
-equal but don't always produce equal results from equal inputs, then you might
-see wrong calculation.
-
-
-The ``make_node`` method creates a node to be included in the expression graph.
-It runs when we apply our Op (``fibby``) to Variable (``x``), as in ``fibby(tensor.vector())``.
-When an Op has multiple inputs, their order in the inputs argument to ``Apply``
-is important:  Theano will call ``make_node(*inputs)`` to copy the graph,
-so it is important not to change the semantics of the expression by changing the argument order.
-
-
-
-All the ``inputs`` and ``outputs`` arguments to ``Apply`` must be Variables.
-A common and easy way to ensure inputs are variables is to run them through
-``as_tensor_variable``.
-This function leaves TensorType variables alone, raises an
-error for non-TensorType variables, and copies any ``numpy.ndarray`` into the
-storage for a TensorType Constant.
-The ``make_node`` method dictates the appropriate Type for all output
-variables.
-
-The ``perform`` method implements the Op's mathematical logic in Python.
-The inputs (here ``x``) are passed by value,
-but a single output is returned indirectly as the first element of
-single-element lists.  If ``fibby`` had a second output, it would be stored
-in ``output_storage[1][0]``.
-.. jpt: DOn't understand the following
-In some execution modes, the output storage might
-contain the return value of a previous call.  That old value can be reused to avoid
-memory re-allocation, but it must not influence the semantics of the Op output.
-
-The ``c_code`` method accepts variable names as arguments (``name``, ``inames``,
-``onames``) and returns a C code fragment that computes the expression output.
-In case of error, the ``%(fail)s`` statement cleans up and returns properly.
-The variables ``%(x)s`` and ``%(y)s`` are set up by the TensorType to be ``PyArrayObject`` pointers.
-TensorType also set up ``dtype_%(x)s`` to be a typdef to the C type for ``x``.
 
 In the first two lines of the C function, we make y point to a new array with
 the correct size for the output. This is essentially simulating the line
 ``y = x.copy()``.
+The variables ``%(x)s`` and ``%(y)s`` are set up by the TensorType to be ``PyArrayObject`` pointers.
+TensorType also set up ``dtype_%(x)s`` to be a typdef to the C type for ``x``.
 
 
 .. code-block:: c
@@ -157,34 +83,6 @@ http://docs.scipy.org/doc/numpy/reference/c-api.types-and-structures.html
 
 TODO: NEEDS MORE EXPLANATION.
 
-There are some important restrictions to remember when implementing an Op.
-Unless your Op correctly defines a ``view_map`` attribute, the ``perform`` and ``c_code`` must not
-produce outputs whose memory is aliased to any input (technically, if changing the
-output could change the input object in some sense, they are aliased).
-Unless your Op correctly defines a ``destroy_map`` attribute, ``perform`` and ``c_code`` must
-not modify any of the inputs.
-
-TODO: EXPLAIN DESTROYMAP and VIEWMAP BETTER AND GIVE EXAMPLE.
-
-When developing an Op, you should run computations in DebugMode, by using
-argument ``mode='DebugMode'`` to ``theano.function``. DebugMode is
-slow, but it can catch many common violations of the Op contract.
-
-TODO: Like what? How? Talk about Python vs. C too.
-
-DebugMode is no silver bullet though.
-For example, if you modify an Op ``self.*`` during any of
-``make_node``, ``perform``, or ``c_code``, you are probably doing something
-wrong but DebugMode will not detect this.
-
-TODO: jpt: I don't understand the following sentence.
-
-Ops and Types should usually be considered immutable -- you should
-definitely not make a change that would have an impact on ``__eq__``,
-``__hash__``, or the mathematical value that would be computed by  ``perform``
-or ``c_code``.
-
-
 .. _op_contract_fibby:
 
 Writing an Optimization

doc/extending/index.txt

@@ -5,24 +5,35 @@
 Extending Theano
 ================
 
-
-This advanced tutorial is for users who want to extend Theano with new Types, new
-Operations (Ops), and new graph optimizations.
-
-Along the way, it also introduces many aspects of how Theano works, so it is
-also good for you if you are interested in getting more under the hood with
-Theano itself.
-
-Before tackling this more advanced presentation, it is highly recommended to read the
-introductory :ref:`Tutorial<tutorial>`.
-
-The first few pages will walk you through the definition of a new :ref:`type`,
-``double``, and a basic arithmetic :ref:`operations <op>` on that Type. We
-will start by defining them using a Python implementation and then we will add
-a C implementation.
+This advanced tutorial is for users who want to extend Theano with new Types, 
+new Operations (Ops), and new graph optimizations. This first page of the 
+tutorial mainly focuses on the Python implementation of an Op and then 
+proposes an overview of the most important methods that define an op. 
+The second page of the tutorial (:ref:`extending_theano_c`) provides then 
+information on the C implementation of an Op. The rest of the tutorial 
+goes more in depth on advanced topics related to Ops, such as how to write 
+efficient code for an Op and how to write an optimization to speed up the 
+execution of an Op.
+
+Along the way, this tutorial also introduces many aspects of how Theano works, 
+so it is also good for you if you are interested in getting more under the hood 
+with Theano itself.
+
+.. note::
+
+    Before tackling this more advanced presentation, it is highly recommended 
+    to read the introductory :ref:`Tutorial<tutorial>`, especially the sections 
+    that introduce the Theano Graphs, as providing a novel Theano op requires a 
+    basic understanting of the Theano Graphs.
+
+    See also the :ref:`dev_start_guide` for information regarding the 
+    versioning framework, namely about *git* and *GitHub*, regarding the 
+    development workflow and how to make a quality contribution.
 
 .. toctree::
 
+    extending_theano
+    extending_theano_c
     fibby
     pipeline
     theano_vs_c

doc/glossary.txt

@@ -63,7 +63,7 @@ Glossary
         then compiling them with :term:`theano.function`.
 
         See also :term:`Variable`, :term:`Op`, :term:`Apply`, and
-        :term:`Type`, or read more about :ref:`tutorial_graphstructures`.
+        :term:`Type`, or read more about :ref:`graphstructures`.
 
     Destructive
         An :term:`Op` is destructive (of particular input[s]) if its
@@ -108,7 +108,7 @@ Glossary
         are provided with Theano, but you can add more.
 
         See also :term:`Variable`, :term:`Type`, and :term:`Apply`, 
-        or read more about :ref:`tutorial_graphstructures`.
+        or read more about :ref:`graphstructures`.
 
     Optimizer
         An instance of :class:`Optimizer`, which has the capacity to provide
@@ -141,7 +141,7 @@ Glossary
         ``.type`` attribute of a :term:`Variable`.  
 
         See also :term:`Variable`, :term:`Op`, and :term:`Apply`, 
-        or read more about :ref:`tutorial_graphstructures`.
+        or read more about :ref:`graphstructures`.
 
     Variable
         The the main data structure you work with when using Theano.
@@ -153,7 +153,7 @@ Glossary
         ``x`` and ``y`` are both `Variables`, i.e. instances of the :class:`Variable` class.
 
         See also :term:`Type`, :term:`Op`, and :term:`Apply`, 
-        or read more about :ref:`tutorial_graphstructures`.
+        or read more about :ref:`graphstructures`.
 
     View
         Some Tensor Ops (such as Subtensor and Transpose) can be computed in

doc/tutorial/index.txt

@@ -22,30 +22,54 @@ Throughout the tutorial, bear in mind that there is a :ref:`glossary` as well
 as *index* and *modules* links in the upper-right corner of each page to help
 you out.
 
+Prerequisites
+-------------
 .. toctree::
 
     python
     numpy
+
+Basics
+------
+
+.. toctree::
+
     adding
     examples
-    symbolic_graphs
-    printing_drawing
     gradients
-    modes
-    loading_and_saving
     conditions
     loop
+    shape_info
+
+Advanced
+--------
+
+.. toctree::
+
     sparse
     using_gpu
     using_multi_gpu
-    gpu_data_convert
-    aliasing
-    shape_info
+
+Advanced configuration and debugging
+------------------------------------
+
+.. toctree::
+
+    modes
+    printing_drawing
     debug_faq
     nan_tutorial
     profiling
-    extending_theano
-    extending_theano_c
+
+Further readings
+----------------
+
+.. toctree::
+
+    ../extending/graphstructures 
+    loading_and_saving
+    gpu_data_convert
+    aliasing
     python-memory-management
     multi_cores
     faq_tutorial

doc/tutorial/symbolic_graphs.txt

-
-.. _tutorial_graphstructures:
-
-================
-Graph Structures
-================
-
-
-Theano Graphs
-=============
-
-Debugging or profiling code written in Theano is not that simple if you
-do not know what goes on under the hood. This chapter is meant to
-introduce you to a required minimum of the inner workings of Theano.
-For more detail see :ref:`extending`.
-
-The first step in writing Theano code is to write down all mathematical
-relations using symbolic placeholders (**variables**). When writing down
-these expressions you use operations like ``+``, ``-``, ``**``,
-``sum()``, ``tanh()``. All these are represented internally as **ops**.
-An *op* represents a certain computation on some type of inputs
-producing some type of output. You can see it as a *function definition*
-in most programming languages.
-
-Theano builds internally a graph structure composed of interconnected
-**variable** nodes, **op** nodes and **apply** nodes. An
-*apply* node represents the application of an *op* to some
-*variables*. It is important to draw the difference between the
-definition of a computation represented by an *op* and its application
-to some actual data which is represented by the *apply* node. For more
-detail about these building blocks refer to :ref:`variable`, :ref:`op`,
-:ref:`apply`. Here is an example of a graph:
-
-
-**Code**
-
-.. testcode::
-
-    import theano.tensor as T
-    x = T.dmatrix('x')
-    y = T.dmatrix('y')
-    z = x + y
-
-**Diagram**
-
-.. _tutorial-graphfigure:
-
-.. figure:: apply.png
-    :align: center
-
-    Interaction between instances of Apply (blue), Variable (red), Op (green),
-    and Type (purple).
-
-.. # COMMENT
-    WARNING: hyper-links and ref's seem to break the PDF build when placed
-    into this figure caption.
-
-Arrows in this figure represent references to the
-Python objects pointed at. The blue
-box is an :ref:`Apply` node. Red boxes are :ref:`Variable` nodes. Green
-circles are :ref:`Ops <op>`. Purple boxes are :ref:`Types <type>`.
-
-
-The graph can be traversed starting from outputs (the result of some
-computation) down to its inputs using the owner field.
-Take for example the following code:
-
->>> import theano
->>> x = theano.tensor.dmatrix('x')
->>> y = x * 2.
-
-If you enter ``type(y.owner)`` you get ``<class 'theano.gof.graph.Apply'>``,
-which is the apply node that connects the op and the inputs to get this
-output. You can now print the name of the op that is applied to get
-*y*:
-
->>> y.owner.op.name
-'Elemwise{mul,no_inplace}'
-
-Hence, an elementwise multiplication is used to compute *y*. This
-multiplication is done between the inputs:
-
->>> len(y.owner.inputs)
-2
->>> y.owner.inputs[0]
-x
->>> y.owner.inputs[1]
-DimShuffle{x,x}.0
-
-Note that the second input is not 2 as we would have expected. This is
-because 2 was first :term:`broadcasted <broadcasting>` to a matrix of
-same shape as *x*. This is done by using the op ``DimShuffle`` :
-
->>> type(y.owner.inputs[1])
-<class 'theano.tensor.var.TensorVariable'>
->>> type(y.owner.inputs[1].owner)
-<class 'theano.gof.graph.Apply'>
->>> y.owner.inputs[1].owner.op # doctest: +SKIP
-<theano.tensor.elemwise.DimShuffle object at 0x106fcaf10>
->>> y.owner.inputs[1].owner.inputs
-[TensorConstant{2.0}]
-
-
-Starting from this graph structure it is easier to understand how
-*automatic differentiation* proceeds and how the symbolic relations
-can be *optimized* for performance or stability.
-
-
-Automatic Differentiation
-=========================
-
-Having the graph structure, computing automatic differentiation is
-simple. The only thing :func:`tensor.grad` has to do is to traverse the
-graph from the outputs back towards the inputs through all *apply*
-nodes (*apply* nodes are those that define which computations the
-graph does). For each such *apply* node, its *op* defines
-how to compute the *gradient* of the node's outputs with respect to its
-inputs. Note that if an *op* does not provide this information,
-it is assumed that the *gradient* is not defined.
-Using the
-`chain rule <http://en.wikipedia.org/wiki/Chain_rule>`_
-these gradients can be composed in order to obtain the expression of the
-*gradient* of the graph's output with respect to the graph's inputs .
-
-A following section of this tutorial will examine the topic of :ref:`differentiation<tutcomputinggrads>`
-in greater detail.
-
-
-Optimizations
-=============
-
-When compiling a Theano function, what you give to the
-:func:`theano.function <function.function>` is actually a graph
-(starting from the output variables you can traverse the graph up to
-the input variables). While this graph structure shows how to compute
-the output from the input, it also offers the possibility to improve the
-way this computation is carried out. The way optimizations work in
-Theano is by identifying and replacing certain patterns in the graph
-with other specialized patterns that produce the same results but are either
-faster or more stable. Optimizations can also detect
-identical subgraphs and ensure that the same values are not computed
-twice or reformulate parts of the graph to a GPU specific version.
-
-For example, one (simple) optimization that Theano uses is to replace
-the pattern :math:`\frac{xy}{y}` by *x.*
-
-Further information regarding the optimization
-:ref:`process<optimization>` and the specific :ref:`optimizations<optimizations>` that are applicable
-is respectively available in the library and on the entrance page of the documentation.
-
-
-**Example**
-
-Symbolic programming involves a change of paradigm: it will become clearer
-as we apply it. Consider the following example of optimization:
-
->>> import theano
->>> a = theano.tensor.vector("a")      # declare symbolic variable
->>> b = a + a ** 10                    # build symbolic expression
->>> f = theano.function([a], b)        # compile function
->>> print(f([0, 1, 2]))                # prints `array([0,2,1026])`
-[    0.     2.  1026.]
->>> theano.printing.pydotprint(b, outfile="./pics/symbolic_graph_unopt.png", var_with_name_simple=True)  # doctest: +SKIP
-The output file is available at ./pics/symbolic_graph_unopt.png
->>> theano.printing.pydotprint(f, outfile="./pics/symbolic_graph_opt.png", var_with_name_simple=True)  # doctest: +SKIP
-The output file is available at ./pics/symbolic_graph_opt.png
-
-
-.. |g1| image:: ./pics/symbolic_graph_unopt.png
-    :width: 500 px
-
-.. |g2| image:: ./pics/symbolic_graph_opt.png
-    :width: 500 px
-
-We used :func:`theano.printing.pydotprint` to visualize the optimized graph
-(right), which is much more compact than the unoptimized graph (left).
-
-======================================================  =====================================================
-        Unoptimized graph                                    Optimized graph
-======================================================  =====================================================
-|g1|                                                    |g2|
-======================================================  =====================================================
-

theano/tensor/signal/pool.py

@@ -47,7 +47,8 @@ def max_pool_2d_same_size(input, patch_size):
 
 def pool_2d(input, ds, ignore_border=None, st=None, padding=(0, 0),
             mode='max'):
-    """
+    """Downscale the input by a specified factor
+
     Takes as input a N-D tensor, where N >= 2. It downscales the input image by
     the specified factor, by keeping only the maximum value of non-overlapping
     patches of size (ds[0],ds[1])