more documentation in the advanced tutorial

doc/advanced/env.txt

@@ -4,3 +4,13 @@
 ===
 Env
 ===
+
+WRITEME
+
+.. _envfeature:
+
+Feature
+=======
+
+WRITEME
+

doc/advanced/index.txt

@@ -13,6 +13,7 @@ Structure
    
    graphstructures
    env
+   features
    optimization
    compilation
    ccodegen

doc/advanced/optimization.txt

@@ -4,3 +4,12 @@
 ============
 Optimization
 ============
+
+WRITEME
+
+.. _navigator:
+
+Navigator
+=========
+
+WRITEME

doc/glossary.txt

@@ -71,6 +71,9 @@ Glossary of terminology
     op
         WRITEME
 
+    pure
+        WRITEME
+
     Result 
         A :ref:`result` is the main data structure you work with when
         using Theano. The symbolic inputs that you operate on are

doc/tutorials/advanced/ex1/cop.txt

@@ -161,6 +161,6 @@ version that it produces in the code I gave above.
 
 
 
-**Next:** `Example 2 - cons_cell`_
+**Next:** `Views and inplace operations`_
 
-.. _Example 2 - cons_cell: ../ex2/index.html
+.. _Views and inplace operations: ../inplace.html

doc/tutorials/advanced/ex1/index.txt

@@ -3,11 +3,11 @@
 Example 1 - double
 ==================
 
+WRITEME
+
 .. toctree::
 
    type
    op
    ctype
    cop
-
-WRITEME

doc/tutorials/advanced/index.txt

@@ -10,11 +10,11 @@ Before tackling this tutorial, it is highly recommended to read the
 
 
 The advanced tutorial is meant to give the reader a greater
-understanding of the building blocks of Theano. It contains two
-examples which cover most of the conceptual space associated with
-:ref:`type` and :ref:`op` and then expands on other important matters
-such as optimization.
-
+understanding of the building blocks of Theano. Through this tutorial
+we are going to define one :ref:`type`, ``double`` and basic
+arithmetic :ref:`operations <op>` on that Type. We will first define
+them using a Python implementation and then we will add a C
+implementation.
 
 This tutorial should be of most use to users who want to extend Theano
 with custom types and operations related to these types. Users who
@@ -26,8 +26,10 @@ concepts at work here.
 
 .. toctree::
 
-   ex1/index
-   ex2/index
+   ex1/type
+   ex1/op
+   ex1/ctype
+   ex1/cop
    inplace
    optimization
    tips
@@ -35,40 +37,5 @@ concepts at work here.
 
 
 
-..
-    `Example 1`_
-      Making a basic arithmetic system on doubles
-    
-    `Example 2`_
-      Making a higher-level type: ``cons_cell`` (pair)
-    
-    `Views and inplace operations`_
-      A guide to making Ops that return a :term:`view` on their inputs or
-      operate :term:`inplace` on them.
-    
-    `Graph optimization`_
-      A guide to the different ways of defining new custom optimizations
-      to simplify the computation graph and/or improve its numerical
-      stability or other desirable properties.
-    
-    `Tips`_
-      Tips and tricks about writing types, ops and optimizations. This
-      page is good reference - check it and come back to it!
-    
-    `Wrapping up`_
-      A guide to what to look at next
-    
-    
-    .. _Example 1: ex1/index.html
-    .. _Example 2: ex2/index.html
-    .. _Views and inplace operations: inplace.html
-    .. _Graph optimization: optimization.html
-    .. _Tips: tips.html
-    .. _Wrapping up: wrapup.html
-
-..
-
-
-
 
 

doc/tutorials/advanced/inplace.txt

@@ -4,5 +4,196 @@
 Views and inplace operations
 ============================
 
-WRITEME
+Theano allows the definition of Ops which return a :term:`view` on one
+of their inputs or operates :term:`inplace` on one or several
+inputs. However, in order to work correctly, these Ops need to
+implement an additional interface.
+
+Theano recognizes views and inplace operations specially. It ensures
+that they are used in a consistent manner and it ensures that
+operations will be carried in a compatible order.
+
+An unfortunate fact is that it is impossible to return a view on an
+input with the ``double`` type or to operate inplace on it (Python
+floats are immutable). Therefore, we can't make examples of these
+concepts out of what we've just built. Nonetheless, we will present
+the concepts:
+
+
+Views
+=====
+
+A "view" on an object ``x`` is an object ``y`` which shares memory
+with ``x`` in some way. In other words, changing ``x`` might also
+change ``y`` and vice versa. For example, imagine a "vector" structure
+which contains two fields: an integer length and a pointer to a memory
+buffer. Suppose we have:
+
+::
+
+   x = vector {length: 256,
+               address: 0xDEADBEEF}
+
+   y = vector {length: 224,
+               address: 0xDEADBEEF + 0x10}
+
+   z = vector {length: 256,
+               address: 0xCAFEBABE}
+
+
+So ``x`` uses the memory range ``0xDEADBEEF - 0xDEADBFEF``, ``y`` the
+range ``0xDEADBEFF - 0xDEADBFDF`` and z the range ``0xCAFEBABE -
+0xCAFEBBBE``. Since the ranges for ``x`` and ``y`` overlap, ``y`` is
+considered to be a view of ``x`` and vice versa.
+
+Suppose you had an Op which took ``x`` as input and returned
+``y``. You would need to tell Theano that y is a view of x. For this
+purpose, you would set the ``view_map`` field as follows:
+
+
+.. code-block:: python
+
+   myop.view_map = {0: [0]}
+
+
+What this means is that the first output (rank 0) is a view of the
+first input (rank 0). Even though the interface allows a list of
+inputs that are a view of a given output, this feature is currently
+unsupported. Here are more examples:
+
+
+.. code-block:: python
+
+   myop.view_map = {0: [0]} # first output is a view of first input
+   myop.view_map = {0: [1]} # first output is a view of second input
+   myop.view_map = {1: [0]} # second output is a view of first input
+
+   myop.view_map = {0: [0], # first output is a view of first input
+                    1: [1]} # *AND* second output is a view of second input
+
+   myop.view_map = {0: [0], # first output is a view of first input
+                    1: [0]} # *AND* second output is *ALSO* a view of first input
+
+   myop.view_map = {0: [0, 1]} # THIS IS NOT SUPPORTED! Only put a single input number in the list!
+
+
+
+Inplace operations
+==================
+
+An inplace operation is one that modifies one or more of its
+inputs. For example, the expression ``x += y`` where ``x`` and ``y``
+are ``numpy.ndarray`` instances would normally represent an inplace
+operation on ``x``.
+
+.. note::
+
+   Inplace operations in theano still work in a functional setting:
+   they need to return the modified input. Symbolically, Theano
+   requires one Result standing for the input *before* being modified
+   and *another* Result representing the input *after* being
+   modified. Therefore, code using inplace operations would look like
+   this:
+
+   .. code-block:: python
+
+      x, y = dscalars('xy')
+      r1 = log(x)
+
+      # r2 is x AFTER the add_inplace - x still represents the value before adding y
+      r2 = add_inplace(x, y)
+
+      # r3 is log(x) using the x from BEFORE the add_inplace
+      # r3 is the SAME as r1, even if we wrote this line after the add_inplace line
+      # Theano is actually going to compute r3 BEFORE r2
+      r3 = log(x)
+
+      # this is log(x) using the x from AFTER the add_inplace (so it's like log(x + y))
+      r4 = log(r2)
+
+   Needless to say, this goes for user-defined inplace operations as
+   well: the modified input must figure in the list of outputs you
+   give to Apply in the definition of make_node.
+
+   Also, for technical reasons but also because they are slightly
+   confusing to use as evidenced by the previous code, Theano does not
+   allow the end user to use inplace operations by default. However,
+   it does allow *optimizations* to substitute them in in a later
+   phase. Therefore, typically, if you define an inplace operation,
+   you will define a pure equivalent and an optimization which
+   subsitutes one for the other. Theano will automatically verify if
+   it is possible to do so and will refuse the substitution if it
+   introduces inconsistencies.
+
+
+Take the previous definitions of x, y and z and suppose an Op which
+adds one to every byte of its input. If we give ``x`` as an input to
+that Op, it can either allocate a new buffer of the same size as ``x``
+(that could be ``z``) and set that new buffer's bytes to the result of
+the addition. That would be a normal, :term:`pure` Op. Alternatively,
+it could add one to each byte *in* the buffer ``x``, therefore
+changing it. That would be an inplace Op.
+
+Theano needs to be notified of this fact. The syntax is similar to
+that of view_map, so I will copy paste cleverly:
+
+
+.. code-block:: python
+
+   myop.destroy_map = {0: [0]}
+
+
+What this means is that the first output (rank 0) operates inplace on the
+first input (rank 0).
+
+
+.. code-block:: python
+
+   myop.destroy_map = {0: [0]} # first output operates inplace on first input
+   myop.destroy_map = {0: [1]} # first output operates inplace on second input
+   myop.destroy_map = {1: [0]} # second output operates inplace on first input
+
+   myop.destroy_map = {0: [0], # first output operates inplace on first input
+                    1: [1]} # *AND* second output operates inplace on second input
+
+   myop.destroy_map = {0: [0], # first output operates inplace on first input
+                    1: [0]} # *AND* second output *ALSO* operates inplace on first input
+
+   myop.destroy_map = {0: [0, 1]} # first output operates inplace on both the first and second input
+   # unlike for views, the previous line is legal and supported
+
+
+Purely destructive operations
+=============================
+
+While some operations will operate inplace on their inputs, some will
+simply destroy or corrupt them. For example, an Op could do temporary
+calculations right in its inputs. If that is the case, Theano also
+needs to be notified. The way to notify Theano is to assume that some
+output operated inplace on whatever inputs are changed or corrupted by
+the Op (even if the output does not technically reuse any of the
+input(s)'s memory). From there, go to the previous section.
+
+
+.. warning::
+   Failure to correctly mark down views and inplace operations using
+   ``view_map`` and ``destroy_map`` can lead to nasty bugs. In the
+   absence of this information, Theano might assume that it is safe to
+   execute an inplace operation on some inputs *before* doing other
+   calculations on the *previous* values of the inputs. For example,
+   in the code: ``y = log(x); x2 = add_inplace(x, z)`` it is
+   imperative to do the logarithm before the addition (because after
+   the addition, the original x that we wanted to take the logarithm
+   of is gone). If Theano does not know that ``add_inplace`` changes
+   the value of ``x`` it might invert the order and that will
+   certainly lead to erroneous computations and be a headache to
+   debug.
+
+
+**Next:** `Graph optimization`_
+
+.. _Graph optimization: optimization.html
+
+
+
 

doc/tutorials/advanced/optimization.txt

@@ -4,4 +4,242 @@
 Graph optimization
 ==================
 
-WRITEME
+In this section we will define a couple optimizations on doubles.
+
+
+Global and local optimizations
+==============================
+
+First, let's lay out the way optimizations work in Theano. There are
+two types of optimizations: *global* optimizations and *local*
+optimizations. A global optimization takes an :ref:`env` object (an
+Env is a wrapper around a whole computation graph, you can see its
+:ref:`documentation <env>` for more details) and navigates through it
+in a suitable way, replacing some Results by others in the process. A
+local optimization, on the other hand, is defined as a function on a
+*single* :ref:`apply` node and must return either False (to mean that
+nothing is to be done) or a list of new Results that we would like to
+replace the node's outputs with. A :ref:`navigator` is a special kind
+of global optimization which navigates the computation graph in some
+fashion (in topological order, reverse-topological order, random
+order, etc.) and applies one or more local optimizations at each step.
+
+Optimizations which are holistic, meaning that they must take into
+account dependencies that might be all over the graph, should be
+global. Optimizations that can be done with a narrow perspective are
+better defined as local optimizations. The majority of optimizations
+we want to define are local.
+
+
+Global optimization
+-------------------
+
+A global optimization is an object which defines the following
+methods:
+
+- **apply(env)**
+
+  - This method takes an Env object which contains the computation
+    graph and does modifications in line with what the optimization is
+    meant to do. This is of the main method of the optimizer.
+
+- **add_requirements(env)**
+
+  - This method takes an Env object and adds :ref:`features
+    <envfeature>` to it. These features are "plugins" that are needed
+    for the apply method to do its job properly.
+
+- **optimize(env)**
+
+  - This is the interface function called by Theano.
+
+  - *Default:* this is defined by Optimizer as ``add_requirement(env);
+    apply(env)``.
+
+See the section about :ref:`env` to understand how to define these
+methods.
+
+
+Local optimization
+------------------
+
+A local optimization is an object which defines the following methods:
+
+- **transform(node)**
+
+  - This method takes an :ref:`apply` node and returns either False to
+    signify that no changes are to be done or a list of Results which
+    matches the length of the node's ``outputs`` list. When the
+    LocalOptimizer is applied by a Navigator, the outputs of the node
+    passed as argument to the LocalOptimizer will be replaced by the
+    list returned.
+
+
+
+One simplification rule
+=======================
+
+For starters, let's define the following simplification:
+
+.. math::
+
+   \frac{xy}{y} = x
+
+We will implement it in three ways: using a global optimization, a
+local optimization with a Navigator and then using the PatternSub
+facility.
+
+
+Global optimization
+-------------------
+
+Here is the code for a global optimization implementing the
+simplification described above:
+
+.. code-block:: python
+   
+   from theano.gof import toolbox
+   
+   class Simplify(gof.Optimizer):
+       def add_requirements(self, env):
+           env.extend(toolbox.ReplaceValidate())
+       def apply(self, env):
+           for node in env.toposort():
+               if node.op == div:
+                   x, y = node.inputs
+                   z = node.outputs[0]
+                   if x.owner and x.owner.op == mul:
+                       a, b = x.owner.inputs
+                       if y == a:
+                           env.replace_validate(z, b)
+                       elif y == b:
+                           env.replace_validate(z, a)
+   
+   simplify = Simplify()
+
+Here's how it works: first, in ``add_requirements``, we add the
+``ReplaceValidate`` :ref:`envfeature` located in
+``theano.gof.toolbox``. This feature adds the ``replace_validate``
+method to the env, which is an enhanced version of ``replace`` that
+does additional checks to ensure that we are not messing up the
+computation graph (note: if ReplaceValidate was already added by
+another optimizer, ``extend`` will do nothing). In a nutshell,
+``toolbox.ReplaceValidate`` grants access to ``env.replace_validate``
+and ``env.replace_validate`` allows us to replace a Result with
+another while respecting certain validation constraints. You can
+browse the list of :ref:`features <envfeaturelist>` and see if some of
+them might be useful to write optimizations with. For example, as an
+exercise, try to rewrite Simplify using :ref:`nodefinder` (hint: you
+want to use the method it publishes in place of the call to toposort!)
+
+Then, in ``apply`` we do the actual job of simplification. We start by
+iterating through the graph in topological order. For each node
+encountered, we check if it's a ``div`` node. If not, we have nothing
+to do here. If so, we put in x, y and z the numerator, denominator and
+quotient (output) of the division. The simplification only occurs when
+the numerator is a multiplication, so we check for that. If the
+numerator is a multiplication we put the two operands in a and b, so
+we can now say that ``z == (a*b)/y``. If ``y==a`` then ``z==b`` and if
+``y==b`` then ``z==a``. When either case happens then we can replace z
+by either a or b using ``env.replace_validate`` - else we do
+nothing. You might want to check the documentation about :ref:`result`
+and :ref:`apply` to get a better understanding of the
+pointer-following game you need to get ahold of the nodes of interest
+for the simplification (x, y, z, a, b, etc.)
+
+Test time:
+
+>>> x = double('x')
+>>> y = double('y')
+>>> z = double('z')
+>>> a = add(z, mul(div(mul(y, x), y), div(z, x)))
+>>> e = gof.Env([x, y, z], [a])
+>>> e
+[add(z, mul(div(mul(y, x), y), div(z, x)))]
+>>> simplify.optimize(e)
+>>> e
+[add(z, mul(x, div(z, x)))]
+
+Cool! It seems to work. You can check what happens if you put many
+instances of :math:`\frac{xy}{y}` in the graph. Note that it sometimes
+won't work for reasons that have nothing to do with the quality of the
+optimization you wrote. For example, consider the following:
+
+>>> x = double('x')
+>>> y = double('y')
+>>> z = double('z')
+>>> a = div(mul(add(y, z), x), add(y, z))
+>>> e = gof.Env([x, y, z], [a])
+>>> e
+[div(mul(add(y, z), x), add(y, z))]
+>>> simplify.optimize(e)
+>>> e
+[div(mul(add(y, z), x), add(y, z))]
+
+Nothing happened here. The reason is simple: ``add(y, z) != add(y,
+z)``. That is the case for efficiency reasons. To fix this problem we
+first need to merge the parts of the graph that represent the same
+computation, using the ``merge_optimizer`` defined in
+``theano.gof.opt``.
+
+>>> from theano.gof.opt import merge_optimizer
+>>> merge_optimizer.optimize(e)
+>>> e
+[div(mul(*1 -> add(y, z), x), *1)]
+>>> simplify.optimize(e)
+>>> e
+[x]
+
+Once the merge is done, both occurrences of ``add(y, z)`` are
+collapsed into a single one and is used as an input in two
+places. Note that ``add(x, y)`` and ``add(y, x)`` are still considered
+to be different because Theano has no clue that ``add`` is
+commutative. You may write your own global optimizer to identify
+computations that are identical with full knowledge of the rules of
+arithmetics that your ops implement. Theano might provide facilities
+for this somewhere in the future.
+
+.. note::
+   :ref:`env` is a Theano structure intended for the optimization
+   phase. It is used internally by function and Module and is rarely
+   exposed to the end user. You can use it to test out optimizations,
+   etc. if you are comfortable with it, but it is recommended to use
+   the function/Module frontends and to interface optimizations with
+   optdb (we'll see how to do that soon).
+
+
+Local optimization
+------------------
+
+
+
+
+
+
+
+
+
+The optimization database (optdb)
+=================================
+
+Theano exports a symbol called ``optdb`` which acts as a sort of
+ordered database of optimizations. When you make a new optimization,
+you must insert it at the proper place in the database. Furthermore,
+you can give each optimization in the database a set of tags that can
+serve as a basis for filtering.
+
+
+
+Using PatternSub
+================
+
+
+Inplace optimizations
+=====================
+
+
+
+
+**Next:** `Tips`_
+
+.. _Tips: tips.html

doc/tutorials/advanced/tips.txt

@@ -5,6 +5,12 @@ Tips
 ====
 
 
+Reusing outputs
+===============
+
+WRITEME
+
+
 Don't define new Ops unless you have to
 =======================================
 
@@ -48,3 +54,10 @@ defining a new Op. It might not be exhaustive but it covers a lot of
 common mistakes.
 
 WRITEME
+
+
+
+**Next:** `Wrapping up`_
+
+.. _Wrapping up: wrapup.html
+

scripts/docgen.py

@@ -63,7 +63,7 @@ if __name__ == '__main__':
     throot = "/".join(sys.path[0].split("/")[:-1])
 
     options = defaultdict(bool)
-    options.update(dict([x, y or True] for x, y in getopt.getopt(sys.argv[1:], 'o:', ['epydoc', 'rst', 'help'])[0]))
+    options.update(dict([x, y or True] for x, y in getopt.getopt(sys.argv[1:], 'o:', ['epydoc', 'rst', 'help', 'nopdf'])[0]))
     if options['--help']:
         print 'Usage: %s [OPTIONS]' % sys.argv[0]
         print '  -o <dir>: output the html files in the specified dir'
@@ -113,21 +113,22 @@ if __name__ == '__main__':
         sys.path[0:0] = [os.path.join(throot, 'doc')]
         sphinx.main(['', '-E', os.path.join(throot, 'doc'), '.'])
 
-        # Generate latex file in a temp directory
-        import tempfile
-        workdir = tempfile.mkdtemp()
-        sphinx.main(['', '-E', '-b', 'latex',
-            os.path.join(throot, 'doc'), workdir])
-        # Compile to PDF
-        currentdir = os.getcwd()
-        os.chdir(workdir)
-        os.system('make')
-        try:
-            shutil.copy(os.path.join(workdir, 'theano.pdf'), currentdir)
-            os.chdir(currentdir)
-            shutil.rmtree(workdir)
-        except OSError, e:
-            print 'OSError:', e
+        if not options['--nopdf']:
+            # Generate latex file in a temp directory
+            import tempfile
+            workdir = tempfile.mkdtemp()
+            sphinx.main(['', '-E', '-b', 'latex',
+                os.path.join(throot, 'doc'), workdir])
+            # Compile to PDF
+            currentdir = os.getcwd()
+            os.chdir(workdir)
+            os.system('make')
+            try:
+                shutil.copy(os.path.join(workdir, 'theano.pdf'), currentdir)
+                os.chdir(currentdir)
+                shutil.rmtree(workdir)
+            except OSError, e:
+                print 'OSError:', e
 
 
 

theano/gof/cc.py

@@ -288,10 +288,14 @@ def struct_result_codeblocks(result, policies, id, symbol_table, sub):
 #    sub['name'] = name
     sub['id'] = id
     sub['fail'] = failure_code(sub)
+    sub['py_ptr'] = "py_%s" % name
+    sub['stor_ptr'] = "storage_%s" % name
     struct_builder = CodeBlock(*[apply_policy(policy, result, name, sub)
                                  for policy in policies[0]]+[sub]) # struct_declare, struct_behavior, struct_cleanup, sub)
     sub['id'] = id + 1
     sub['fail'] = failure_code(sub)
+    sub['py_ptr'] = "py_%s" % name
+    sub['stor_ptr'] = "storage_%s" % name
     block = CodeBlock(*[apply_policy(policy, result, name, sub)
                         for policy in policies[1]]+[sub]) # run_declare, run_behavior, run_cleanup, sub)
 

theano/gof/link.py

@@ -71,6 +71,7 @@ class Linker(object):
         """
         raise utils.AbstractFunctionError()
 
+    ## DELETEME ##
     def make_function(self, unpack_single = True, **kwargs):
         """
         Returns a function that takes values corresponding to the inputs of the

theano/gof/opt.py

@@ -298,6 +298,7 @@ class MergeOptimizer(Optimizer):
         self.apply_constant_merge(env)
         self.apply_node_merge(env)
 
+merge_optimizer = MergeOptimizer()
 
 def MergeOptMerge(opt):
     """WRITEME
@@ -305,7 +306,7 @@ def MergeOptMerge(opt):
     optimizer in opt and then merges the graph again in case the
     opt introduced additional similarities.
     """
-    merger = MergeOptimizer()
+    merger = merge_optimizer
     return SeqOptimizer([merger, opt, merger])
 
 

theano/gof/type.py

@@ -318,6 +318,8 @@ class Type(object2, PureType, CLinkerType):
 
     """
 
+
+## DELETEME ##
 class SingletonType(Type):
     """WRITEME"""
     __instance = None
@@ -374,6 +376,7 @@ class Generic(SingletonType):
 generic = Generic()
 
 
+## DELETEME ##
 class PropertiedType(Type):
     """WRITEME"""
 

theano/tensor/tests/test_nnet.py

@@ -13,6 +13,8 @@ class T_sigmoid(unittest.TestCase):
     def setUp(self):
         numpy.random.seed(9999)
     def test_elemwise(self):
+        print dir(self)
+        assert 0
         TT.verify_grad(self, sigmoid, [numpy.random.rand(3,4)])
 
 class T_softplus(unittest.TestCase):