merge

doc/advanced/env.txt

@@ -4,3 +4,13 @@
 ===
 Env
 ===
+
+WRITEME
+
+.. _envfeature:
+
+Feature
+=======
+
+WRITEME
+

doc/advanced/index.txt

@@ -13,6 +13,7 @@ Structure
    
    graphstructures
    env
+   features
    optimization
    compilation
    ccodegen

doc/advanced/optimization.txt

@@ -4,3 +4,12 @@
 ============
 Optimization
 ============
+
+WRITEME
+
+.. _navigator:
+
+Navigator
+=========
+
+WRITEME

doc/doc/README.txt

@@ -89,9 +89,9 @@ Configuring the environment
 ---------------------------
 
 Two environment variables are used to control automatic code generation.
-(It is possible to use theano in a way that avoids all automatic code generation, but the functions you make using {{{theano.function}}} will execute more slowly.)
+(It is possible to use theano in a way that avoids all automatic code generation, but the functions you make using ``theano.function`` will execute more slowly.)
 
-- `THEANO_BLAS_LDFLAGS`: 
+- `THEANO_BLAS_LDFLAGS`:
     a space-separated list of library names to link against for BLAS functions. Default: `-lblas`
 
 - `THEANO_COMPILEDIR`:

doc/doc/sandbox.txt

+Basically, this file contains stuff that should be documented, but is not.
+
+Feel free to contribute things that you want documented, as well as to add
+or correct documentation.
+
+
+======================================
+What happens if grad is not defined?
+======================================
+
+If an Op does not define ``grad``, but this Op does not appear in the path when
+you compute the gradient, then there is no problem.
+
+If an Op does not define ``grad``, and this Op *does* appear in the path when
+you compute the gradient, **WRITEME**.
+
+**This documentation is useful when we show users how to write Ops.**
+
+======================================
+What is staticmethod, st_impl?
+======================================
+
+``st_impl`` is an optional method in an Op.
+``@staticmethod`` is a Python decorator for a class method that does not
+implicitly take the class instance as a first argument. Hence, st_impl
+can be used for Op implementations when no information from the Op
+instance is needed. This can be useful for testing an implementation.
+See :api:`XlogX` for an example.
+
+**This documentation is useful when we show users how to write Ops.
+Olivier says this behavior should be discouraged but I feel that st_impl
+should be encouraged where possible.**
+
+============================================================
+how do we write scalar ops and upgrade them to tensor ops?
+============================================================
+
+**Olivier says that :api:`XlogX` gives a good example. In fact, I would
+like to beef xlogx up into our running example for demonstrating how to
+write an Op:**
+
+.. code-block:: python
+
+    class XlogX(scalar.UnaryScalarOp):
+        """
+        Compute X * log(X), with special case 0 log(0) = 0.
+        """
+        @staticmethod
+        def st_impl(x):
+            if x == 0.0:
+                return 0.0
+            return x * numpy.log(x)
+        def impl(self, x):
+            return XlogX.st_impl(x)
+        def grad(self, (x,), (gz,)):
+            return [gz * (1 + scalar.log(x))]
+        def c_code(self, node, name, (x,), (z,), sub):
+            if node.inputs[0].type in [scalar.float32, scalar.float64]:
+                return """%(z)s =
+                    %(x)s == 0.0
+                    ? 0.0
+                    : %(x)s * log(%(x)s);""" % locals()
+            raise NotImplementedError('only floatingpoint is implemented')
+    scalar_xlogx  = XlogX(scalar.upgrade_to_float, name='scalar_xlogx')
+    xlogx = tensor.Elemwise(scalar_xlogx, name='xlogx')
+    
+**It is also necessary to talk about UnaryScalarOp vs. BinaryOp.**
+
+UnaryScalarOp is the same as scalar.ScalarOp with member variable nin=1.
+**give an example of this**
+
+=======================================================
+Documentation on how to write tests
+=======================================================
+
+Guillaume can you make sure to hit these points:
+
+    * What are canonical examples of tests?
+
+        * What are the different test patterns?
+
+    * nnet.py:
+
+        * What is going on with test1, test2, test3, test4?
+
+    * What is the right eq function to use?
+
+        * There are a lot of tests that define their own epsilon, but this should be standardized.  e.g. in test_elemwise.py ``self.failUnless((numpy.abs(f(xv) - zv) < 1e-10).all())``
+
+    * If the expected result of a test is that an Exception is thrown, how do we correctly detect and handle that?
+
+        nosetests has ``failUnlessRaises``
+
+    * Convention is that all test files must start with test_, not _test_, so rename all that use the old convention?
+
+=======================================================
+How to use the PrintOp
+=======================================================
+
+** This is also useful in the How to write an Op tutorial. **
+
+=======================================================
+Modules
+=======================================================
+
+* What is the correct way to tie weights?
+
+=======================================================
+Mammouth
+=======================================================
+
+**This is internal documentation. Guillaume can you make sure to hit these points:**
+
+export THEANO_BLAS_LDFLAGS='-lmkl -liomp5 -fopenmp'
+
+**Do we want the following:**
+
+export OMP_NUM_THREADS=2
+
+=======================================================
+Cache
+=======================================================
+
+The compile cache is written to ``THEANO_COMPILEDIR``.  If this environment
+variable is not present, the compile cache defaults to ``$HOME/.theano``.
+The compile cache is based upon the C++ code of the graph to be compiled.
+So, if you change compilation environment variables, such as
+``THEANO_BLAS_LDFLAGS``, you will need to manually remove your compile cache.
+
+=======================================================
+Type checking
+=======================================================
+
+    * Are there functions for doing type checking?
+        like dtype of this matrix is an int-type (not just int32
+        or int64)
+        "if isinstance(item, int):" is the preferred way to do it in
+        python now, so mimic this
+        If the type is wrong, what exception should be raised?

doc/glossary.txt

@@ -23,13 +23,21 @@ Glossary of terminology
            Broadcasting a row matrix. T and F respectively stand for
            True and False and indicate along which dimensions we allow
            broadcasting.
-        
+
+           If the second argument were a vector, its shape would be
+           ``(2,)`` and its broadcastable pattern ``(F,)``. They would
+           be automatically expanded to the **left** to match the
+           dimensions of the matrix (adding ``1`` to the shape and ``T``
+           to the pattern), resulting in ``(1, 2)`` and ``(T, F)``.
+           It would then behave just like the example above.
+
         Unlike numpy which does broadcasting dynamically, Theano needs
         to know, for any operation which supports broadcasting, which
         dimensions will need to be broadcasted. When applicable, this
         information is given in the :term:`Type` of a :term:`Result`.
-        
+
         See also:
+        * :ref:`How broadcasting is used in Theano's tensor types <tensortypes>`
 
         * `SciPy documentation about numpy's broadcasting <http://www.scipy.org/EricsBroadcastingDoc>`_
         * `OnLamp article about numpy's broadcasting <http://www.onlamp.com/pub/a/python/2000/09/27/numerically.html>`_
@@ -40,7 +48,7 @@ Glossary of terminology
     elementwise
         An elementwise operation ``f`` on two matrices ``M`` and ``N``
         is one such that:
-        
+
         ``f(M, N)[i, j] = f(M[i, j], N[i, j])``
 
         In other words, each element of an input matrix is combined
@@ -71,6 +79,9 @@ Glossary of terminology
     op
         WRITEME
 
+    pure
+        WRITEME
+
     Result 
         A :ref:`result` is the main data structure you work with when
         using Theano. The symbolic inputs that you operate on are

doc/tutorials/advanced/ex1/cop.txt

@@ -161,6 +161,6 @@ version that it produces in the code I gave above.
 
 
 
-**Next:** `Example 2 - cons_cell`_
+**Next:** `Views and inplace operations`_
 
-.. _Example 2 - cons_cell: ../ex2/index.html
+.. _Views and inplace operations: ../inplace.html

doc/tutorials/advanced/ex1/index.txt

@@ -3,11 +3,11 @@
 Example 1 - double
 ==================
 
+WRITEME
+
 .. toctree::
 
    type
    op
    ctype
    cop
-
-WRITEME

doc/tutorials/advanced/index.txt

@@ -10,11 +10,11 @@ Before tackling this tutorial, it is highly recommended to read the
 
 
 The advanced tutorial is meant to give the reader a greater
-understanding of the building blocks of Theano. It contains two
-examples which cover most of the conceptual space associated with
-:ref:`type` and :ref:`op` and then expands on other important matters
-such as optimization.
-
+understanding of the building blocks of Theano. Through this tutorial
+we are going to define one :ref:`type`, ``double`` and basic
+arithmetic :ref:`operations <op>` on that Type. We will first define
+them using a Python implementation and then we will add a C
+implementation.
 
 This tutorial should be of most use to users who want to extend Theano
 with custom types and operations related to these types. Users who
@@ -26,8 +26,10 @@ concepts at work here.
 
 .. toctree::
 
-   ex1/index
-   ex2/index
+   ex1/type
+   ex1/op
+   ex1/ctype
+   ex1/cop
    inplace
    optimization
    tips
@@ -35,40 +37,5 @@ concepts at work here.
 
 
 
-..
-    `Example 1`_
-      Making a basic arithmetic system on doubles
-    
-    `Example 2`_
-      Making a higher-level type: ``cons_cell`` (pair)
-    
-    `Views and inplace operations`_
-      A guide to making Ops that return a :term:`view` on their inputs or
-      operate :term:`inplace` on them.
-    
-    `Graph optimization`_
-      A guide to the different ways of defining new custom optimizations
-      to simplify the computation graph and/or improve its numerical
-      stability or other desirable properties.
-    
-    `Tips`_
-      Tips and tricks about writing types, ops and optimizations. This
-      page is good reference - check it and come back to it!
-    
-    `Wrapping up`_
-      A guide to what to look at next
-    
-    
-    .. _Example 1: ex1/index.html
-    .. _Example 2: ex2/index.html
-    .. _Views and inplace operations: inplace.html
-    .. _Graph optimization: optimization.html
-    .. _Tips: tips.html
-    .. _Wrapping up: wrapup.html
-
-..
-
-
-
 
 

doc/tutorials/advanced/inplace.txt

@@ -4,5 +4,196 @@
 Views and inplace operations
 ============================
 
-WRITEME
+Theano allows the definition of Ops which return a :term:`view` on one
+of their inputs or operates :term:`inplace` on one or several
+inputs. However, in order to work correctly, these Ops need to
+implement an additional interface.
+
+Theano recognizes views and inplace operations specially. It ensures
+that they are used in a consistent manner and it ensures that
+operations will be carried in a compatible order.
+
+An unfortunate fact is that it is impossible to return a view on an
+input with the ``double`` type or to operate inplace on it (Python
+floats are immutable). Therefore, we can't make examples of these
+concepts out of what we've just built. Nonetheless, we will present
+the concepts:
+
+
+Views
+=====
+
+A "view" on an object ``x`` is an object ``y`` which shares memory
+with ``x`` in some way. In other words, changing ``x`` might also
+change ``y`` and vice versa. For example, imagine a "vector" structure
+which contains two fields: an integer length and a pointer to a memory
+buffer. Suppose we have:
+
+::
+
+   x = vector {length: 256,
+               address: 0xDEADBEEF}
+
+   y = vector {length: 224,
+               address: 0xDEADBEEF + 0x10}
+
+   z = vector {length: 256,
+               address: 0xCAFEBABE}
+
+
+So ``x`` uses the memory range ``0xDEADBEEF - 0xDEADBFEF``, ``y`` the
+range ``0xDEADBEFF - 0xDEADBFDF`` and z the range ``0xCAFEBABE -
+0xCAFEBBBE``. Since the ranges for ``x`` and ``y`` overlap, ``y`` is
+considered to be a view of ``x`` and vice versa.
+
+Suppose you had an Op which took ``x`` as input and returned
+``y``. You would need to tell Theano that y is a view of x. For this
+purpose, you would set the ``view_map`` field as follows:
+
+
+.. code-block:: python
+
+   myop.view_map = {0: [0]}
+
+
+What this means is that the first output (rank 0) is a view of the
+first input (rank 0). Even though the interface allows a list of
+inputs that are a view of a given output, this feature is currently
+unsupported. Here are more examples:
+
+
+.. code-block:: python
+
+   myop.view_map = {0: [0]} # first output is a view of first input
+   myop.view_map = {0: [1]} # first output is a view of second input
+   myop.view_map = {1: [0]} # second output is a view of first input
+
+   myop.view_map = {0: [0], # first output is a view of first input
+                    1: [1]} # *AND* second output is a view of second input
+
+   myop.view_map = {0: [0], # first output is a view of first input
+                    1: [0]} # *AND* second output is *ALSO* a view of first input
+
+   myop.view_map = {0: [0, 1]} # THIS IS NOT SUPPORTED! Only put a single input number in the list!
+
+
+
+Inplace operations
+==================
+
+An inplace operation is one that modifies one or more of its
+inputs. For example, the expression ``x += y`` where ``x`` and ``y``
+are ``numpy.ndarray`` instances would normally represent an inplace
+operation on ``x``.
+
+.. note::
+
+   Inplace operations in theano still work in a functional setting:
+   they need to return the modified input. Symbolically, Theano
+   requires one Result standing for the input *before* being modified
+   and *another* Result representing the input *after* being
+   modified. Therefore, code using inplace operations would look like
+   this:
+
+   .. code-block:: python
+
+      x, y = dscalars('xy')
+      r1 = log(x)
+
+      # r2 is x AFTER the add_inplace - x still represents the value before adding y
+      r2 = add_inplace(x, y)
+
+      # r3 is log(x) using the x from BEFORE the add_inplace
+      # r3 is the SAME as r1, even if we wrote this line after the add_inplace line
+      # Theano is actually going to compute r3 BEFORE r2
+      r3 = log(x)
+
+      # this is log(x) using the x from AFTER the add_inplace (so it's like log(x + y))
+      r4 = log(r2)
+
+   Needless to say, this goes for user-defined inplace operations as
+   well: the modified input must figure in the list of outputs you
+   give to Apply in the definition of make_node.
+
+   Also, for technical reasons but also because they are slightly
+   confusing to use as evidenced by the previous code, Theano does not
+   allow the end user to use inplace operations by default. However,
+   it does allow *optimizations* to substitute them in in a later
+   phase. Therefore, typically, if you define an inplace operation,
+   you will define a pure equivalent and an optimization which
+   subsitutes one for the other. Theano will automatically verify if
+   it is possible to do so and will refuse the substitution if it
+   introduces inconsistencies.
+
+
+Take the previous definitions of x, y and z and suppose an Op which
+adds one to every byte of its input. If we give ``x`` as an input to
+that Op, it can either allocate a new buffer of the same size as ``x``
+(that could be ``z``) and set that new buffer's bytes to the result of
+the addition. That would be a normal, :term:`pure` Op. Alternatively,
+it could add one to each byte *in* the buffer ``x``, therefore
+changing it. That would be an inplace Op.
+
+Theano needs to be notified of this fact. The syntax is similar to
+that of view_map, so I will copy paste cleverly:
+
+
+.. code-block:: python
+
+   myop.destroy_map = {0: [0]}
+
+
+What this means is that the first output (rank 0) operates inplace on the
+first input (rank 0).
+
+
+.. code-block:: python
+
+   myop.destroy_map = {0: [0]} # first output operates inplace on first input
+   myop.destroy_map = {0: [1]} # first output operates inplace on second input
+   myop.destroy_map = {1: [0]} # second output operates inplace on first input
+
+   myop.destroy_map = {0: [0], # first output operates inplace on first input
+                    1: [1]} # *AND* second output operates inplace on second input
+
+   myop.destroy_map = {0: [0], # first output operates inplace on first input
+                    1: [0]} # *AND* second output *ALSO* operates inplace on first input
+
+   myop.destroy_map = {0: [0, 1]} # first output operates inplace on both the first and second input
+   # unlike for views, the previous line is legal and supported
+
+
+Purely destructive operations
+=============================
+
+While some operations will operate inplace on their inputs, some will
+simply destroy or corrupt them. For example, an Op could do temporary
+calculations right in its inputs. If that is the case, Theano also
+needs to be notified. The way to notify Theano is to assume that some
+output operated inplace on whatever inputs are changed or corrupted by
+the Op (even if the output does not technically reuse any of the
+input(s)'s memory). From there, go to the previous section.
+
+
+.. warning::
+   Failure to correctly mark down views and inplace operations using
+   ``view_map`` and ``destroy_map`` can lead to nasty bugs. In the
+   absence of this information, Theano might assume that it is safe to
+   execute an inplace operation on some inputs *before* doing other
+   calculations on the *previous* values of the inputs. For example,
+   in the code: ``y = log(x); x2 = add_inplace(x, z)`` it is
+   imperative to do the logarithm before the addition (because after
+   the addition, the original x that we wanted to take the logarithm
+   of is gone). If Theano does not know that ``add_inplace`` changes
+   the value of ``x`` it might invert the order and that will
+   certainly lead to erroneous computations and be a headache to
+   debug.
+
+
+**Next:** `Graph optimization`_
+
+.. _Graph optimization: optimization.html
+
+
+
 

doc/tutorials/advanced/optimization.txt

@@ -4,4 +4,242 @@
 Graph optimization
 ==================
 
-WRITEME
+In this section we will define a couple optimizations on doubles.
+
+
+Global and local optimizations
+==============================
+
+First, let's lay out the way optimizations work in Theano. There are
+two types of optimizations: *global* optimizations and *local*
+optimizations. A global optimization takes an :ref:`env` object (an
+Env is a wrapper around a whole computation graph, you can see its
+:ref:`documentation <env>` for more details) and navigates through it
+in a suitable way, replacing some Results by others in the process. A
+local optimization, on the other hand, is defined as a function on a
+*single* :ref:`apply` node and must return either False (to mean that
+nothing is to be done) or a list of new Results that we would like to
+replace the node's outputs with. A :ref:`navigator` is a special kind
+of global optimization which navigates the computation graph in some
+fashion (in topological order, reverse-topological order, random
+order, etc.) and applies one or more local optimizations at each step.
+
+Optimizations which are holistic, meaning that they must take into
+account dependencies that might be all over the graph, should be
+global. Optimizations that can be done with a narrow perspective are
+better defined as local optimizations. The majority of optimizations
+we want to define are local.
+
+
+Global optimization
+-------------------
+
+A global optimization is an object which defines the following
+methods:
+
+- **apply(env)**
+
+  - This method takes an Env object which contains the computation
+    graph and does modifications in line with what the optimization is
+    meant to do. This is of the main method of the optimizer.
+
+- **add_requirements(env)**
+
+  - This method takes an Env object and adds :ref:`features
+    <envfeature>` to it. These features are "plugins" that are needed
+    for the apply method to do its job properly.
+
+- **optimize(env)**
+
+  - This is the interface function called by Theano.
+
+  - *Default:* this is defined by Optimizer as ``add_requirement(env);
+    apply(env)``.
+
+See the section about :ref:`env` to understand how to define these
+methods.
+
+
+Local optimization
+------------------
+
+A local optimization is an object which defines the following methods:
+
+- **transform(node)**
+
+  - This method takes an :ref:`apply` node and returns either False to
+    signify that no changes are to be done or a list of Results which
+    matches the length of the node's ``outputs`` list. When the
+    LocalOptimizer is applied by a Navigator, the outputs of the node
+    passed as argument to the LocalOptimizer will be replaced by the
+    list returned.
+
+
+
+One simplification rule
+=======================
+
+For starters, let's define the following simplification:
+
+.. math::
+
+   \frac{xy}{y} = x
+
+We will implement it in three ways: using a global optimization, a
+local optimization with a Navigator and then using the PatternSub
+facility.
+
+
+Global optimization
+-------------------
+
+Here is the code for a global optimization implementing the
+simplification described above:
+
+.. code-block:: python
+   
+   from theano.gof import toolbox
+   
+   class Simplify(gof.Optimizer):
+       def add_requirements(self, env):
+           env.extend(toolbox.ReplaceValidate())
+       def apply(self, env):
+           for node in env.toposort():
+               if node.op == div:
+                   x, y = node.inputs
+                   z = node.outputs[0]
+                   if x.owner and x.owner.op == mul:
+                       a, b = x.owner.inputs
+                       if y == a:
+                           env.replace_validate(z, b)
+                       elif y == b:
+                           env.replace_validate(z, a)
+   
+   simplify = Simplify()
+
+Here's how it works: first, in ``add_requirements``, we add the
+``ReplaceValidate`` :ref:`envfeature` located in
+``theano.gof.toolbox``. This feature adds the ``replace_validate``
+method to the env, which is an enhanced version of ``replace`` that
+does additional checks to ensure that we are not messing up the
+computation graph (note: if ReplaceValidate was already added by
+another optimizer, ``extend`` will do nothing). In a nutshell,
+``toolbox.ReplaceValidate`` grants access to ``env.replace_validate``
+and ``env.replace_validate`` allows us to replace a Result with
+another while respecting certain validation constraints. You can
+browse the list of :ref:`features <envfeaturelist>` and see if some of
+them might be useful to write optimizations with. For example, as an
+exercise, try to rewrite Simplify using :ref:`nodefinder` (hint: you
+want to use the method it publishes in place of the call to toposort!)
+
+Then, in ``apply`` we do the actual job of simplification. We start by
+iterating through the graph in topological order. For each node
+encountered, we check if it's a ``div`` node. If not, we have nothing
+to do here. If so, we put in x, y and z the numerator, denominator and
+quotient (output) of the division. The simplification only occurs when
+the numerator is a multiplication, so we check for that. If the
+numerator is a multiplication we put the two operands in a and b, so
+we can now say that ``z == (a*b)/y``. If ``y==a`` then ``z==b`` and if
+``y==b`` then ``z==a``. When either case happens then we can replace z
+by either a or b using ``env.replace_validate`` - else we do
+nothing. You might want to check the documentation about :ref:`result`
+and :ref:`apply` to get a better understanding of the
+pointer-following game you need to get ahold of the nodes of interest
+for the simplification (x, y, z, a, b, etc.)
+
+Test time:
+
+>>> x = double('x')
+>>> y = double('y')
+>>> z = double('z')
+>>> a = add(z, mul(div(mul(y, x), y), div(z, x)))
+>>> e = gof.Env([x, y, z], [a])
+>>> e
+[add(z, mul(div(mul(y, x), y), div(z, x)))]
+>>> simplify.optimize(e)
+>>> e
+[add(z, mul(x, div(z, x)))]
+
+Cool! It seems to work. You can check what happens if you put many
+instances of :math:`\frac{xy}{y}` in the graph. Note that it sometimes
+won't work for reasons that have nothing to do with the quality of the
+optimization you wrote. For example, consider the following:
+
+>>> x = double('x')
+>>> y = double('y')
+>>> z = double('z')
+>>> a = div(mul(add(y, z), x), add(y, z))
+>>> e = gof.Env([x, y, z], [a])
+>>> e
+[div(mul(add(y, z), x), add(y, z))]
+>>> simplify.optimize(e)
+>>> e
+[div(mul(add(y, z), x), add(y, z))]
+
+Nothing happened here. The reason is simple: ``add(y, z) != add(y,
+z)``. That is the case for efficiency reasons. To fix this problem we
+first need to merge the parts of the graph that represent the same
+computation, using the ``merge_optimizer`` defined in
+``theano.gof.opt``.
+
+>>> from theano.gof.opt import merge_optimizer
+>>> merge_optimizer.optimize(e)
+>>> e
+[div(mul(*1 -> add(y, z), x), *1)]
+>>> simplify.optimize(e)
+>>> e
+[x]
+
+Once the merge is done, both occurrences of ``add(y, z)`` are
+collapsed into a single one and is used as an input in two
+places. Note that ``add(x, y)`` and ``add(y, x)`` are still considered
+to be different because Theano has no clue that ``add`` is
+commutative. You may write your own global optimizer to identify
+computations that are identical with full knowledge of the rules of
+arithmetics that your ops implement. Theano might provide facilities
+for this somewhere in the future.
+
+.. note::
+   :ref:`env` is a Theano structure intended for the optimization
+   phase. It is used internally by function and Module and is rarely
+   exposed to the end user. You can use it to test out optimizations,
+   etc. if you are comfortable with it, but it is recommended to use
+   the function/Module frontends and to interface optimizations with
+   optdb (we'll see how to do that soon).
+
+
+Local optimization
+------------------
+
+
+
+
+
+
+
+
+
+The optimization database (optdb)
+=================================
+
+Theano exports a symbol called ``optdb`` which acts as a sort of
+ordered database of optimizations. When you make a new optimization,
+you must insert it at the proper place in the database. Furthermore,
+you can give each optimization in the database a set of tags that can
+serve as a basis for filtering.
+
+
+
+Using PatternSub
+================
+
+
+Inplace optimizations
+=====================
+
+
+
+
+**Next:** `Tips`_
+
+.. _Tips: tips.html

doc/tutorials/advanced/tips.txt

@@ -5,6 +5,12 @@ Tips
 ====
 
 
+Reusing outputs
+===============
+
+WRITEME
+
+
 Don't define new Ops unless you have to
 =======================================
 
@@ -48,3 +54,10 @@ defining a new Op. It might not be exhaustive but it covers a lot of
 common mistakes.
 
 WRITEME
+
+
+
+**Next:** `Wrapping up`_
+
+.. _Wrapping up: wrapup.html
+

doc/tutorials/basic/tools.txt

@@ -9,6 +9,7 @@ Mode
 
 WRITEME
 
+.. _tensortypes:
 
 Types
 =====
@@ -46,7 +47,7 @@ Dimensionality is one of:
 code   shape  Rows :term:`broadcastable <broadcasting>`? Columns :term:`broadcastable <broadcasting>`?
 ====== ====== ========================================== =============================================
 scalar []     Yes                                        Yes
-vector [n]    Yes                                        N/A
+vector [n]    Yes                                        N/A (vectors are used like row vectors)
 row    [1, n] Yes                                        No
 col    [m, 1] No                                         Yes
 matrix [m, n] No                                         No
@@ -56,13 +57,14 @@ So for example if you want a row of 32-bit floats, it is available
 under ``theano.tensor.frow`` and if you want a matrix of unsigned
 32-bit integers it is available under ``theano.tensor.imatrix``.
 
-Each of the methods described above have a singular version and a
-plural version. When called, the singular version takes a single
-argument which is the name of the :term:`Result` we want to make and
-it makes a single Result of that type. The plural version can either
-take an integer or a string. If an integer is provided, it will return
-that many Results and if a string is provided, it will create one
-Result for each letter of the string, using the letter as the Result's
+Each of the types described above can be constructed by two methods:
+a singular version (e.g., ``dmatrix``) and a plural version
+(``dmatrices``).  When called, the singular version takes a single
+argument which is the name of the :term:`Result` we want to make and it
+makes a single Result of that type. The plural version can either take
+an integer or several strings. If an integer is provided, the method
+will return that many Results and if strings are provided, it will
+create one Result for each string, using the string as the Result's
 name. For example:
 
 .. code-block:: python
@@ -74,14 +76,14 @@ name. For example:
    xyz = dmatrix('xyz') # creates one Result with name 'xyz'
 
    x, y, z = dmatrices(3) # creates three Results with no names
-   x, y, z = dmatrices('xyz') # creates three Results named 'x', 'y' and 'z'
+   x, y, z = dmatrices('x', 'y', 'z') # creates three Results named 'x', 'y' and 'z'
 
 
 Custom tensor types
 -------------------
 
-If you wish to use a type which is not available here (for example, a
-3D tensor) you can build an appropriate type using
+If you wish to use a type of tensor which is not already available here
+(for example, a 3D tensor) you can build an appropriate type using
 ``theano.tensor.Tensor``. The first argument you pass is the ``dtype``
 and the second is the ``broadcastable pattern``.
 
@@ -106,10 +108,11 @@ complex128  complex          128 (two float64)
 
 .. note::
 
-   There are no premade complex types, so you need to make them
-   explicitly with Tensor. Furthermore, few operations are fully
-   supported for complex types: as of version 0.1, only elementary
-   operations (``+-*/``) have C implementations.
+   Even though ``theano.tensor`` does not define any type using
+   ``complex`` dtypes (``complex64`` or ``complex128``), you can define
+   them explicitly with ``Tensor`` (see example below). However, few
+   operations are fully supported for complex types: as of version 0.1,
+   only elementary operations (``+-*/``) have C implementations.
 
 
 The broadcastable pattern, on the other hand, indicates both the
@@ -133,13 +136,24 @@ pattern               interpretation
 [False, False, False] A MxNxP tensor (pattern of a + b)
 ===================== =================================
 
+When two tensors have a different number of dimensions, the broadcastable
+pattern is *expanded to the left*, by padding with ``True``. So, for example,
+a vector's pattern, ``[False]``, could be expanded to ``[True, False]``, and
+would behave like a row (1xN matrix). In the same way, a matrix (``[False,
+False]``) would behave like a 1xNxP tensor (``[True, False, False]``).
+
 So if we wanted to create a type representing a 3D array of unsigned
 bytes, we would simply do: 
 
 .. code-block:: python
 
+   # 3D tensor of signed bytes
    mytype = theano.tensor.Tensor('uint8', [False]*3)
 
+   # complex types (based on complex64)
+   my_cscalar = theano.tensor.Tensor('complex64', [])
+   my_cmatrix = theano.tensor.Tensor('complex64', [False, False])
+
 
 Ops
 ===

scripts/docgen.py

@@ -63,7 +63,7 @@ if __name__ == '__main__':
     throot = "/".join(sys.path[0].split("/")[:-1])
 
     options = defaultdict(bool)
-    options.update(dict([x, y or True] for x, y in getopt.getopt(sys.argv[1:], 'o:', ['epydoc', 'rst', 'help'])[0]))
+    options.update(dict([x, y or True] for x, y in getopt.getopt(sys.argv[1:], 'o:', ['epydoc', 'rst', 'help', 'nopdf'])[0]))
     if options['--help']:
         print 'Usage: %s [OPTIONS]' % sys.argv[0]
         print '  -o <dir>: output the html files in the specified dir'
@@ -113,21 +113,22 @@ if __name__ == '__main__':
         sys.path[0:0] = [os.path.join(throot, 'doc')]
         sphinx.main(['', '-E', os.path.join(throot, 'doc'), '.'])
 
-        # Generate latex file in a temp directory
-        import tempfile
-        workdir = tempfile.mkdtemp()
-        sphinx.main(['', '-E', '-b', 'latex',
-            os.path.join(throot, 'doc'), workdir])
-        # Compile to PDF
-        currentdir = os.getcwd()
-        os.chdir(workdir)
-        os.system('make')
-        try:
-            shutil.copy(os.path.join(workdir, 'theano.pdf'), currentdir)
-            os.chdir(currentdir)
-            shutil.rmtree(workdir)
-        except OSError, e:
-            print 'OSError:', e
+        if not options['--nopdf']:
+            # Generate latex file in a temp directory
+            import tempfile
+            workdir = tempfile.mkdtemp()
+            sphinx.main(['', '-E', '-b', 'latex',
+                os.path.join(throot, 'doc'), workdir])
+            # Compile to PDF
+            currentdir = os.getcwd()
+            os.chdir(workdir)
+            os.system('make')
+            try:
+                shutil.copy(os.path.join(workdir, 'theano.pdf'), currentdir)
+                os.chdir(currentdir)
+                shutil.rmtree(workdir)
+            except OSError, e:
+                print 'OSError:', e
 
 
 

theano/gof/cc.py

@@ -288,10 +288,14 @@ def struct_result_codeblocks(result, policies, id, symbol_table, sub):
 #    sub['name'] = name
     sub['id'] = id
     sub['fail'] = failure_code(sub)
+    sub['py_ptr'] = "py_%s" % name
+    sub['stor_ptr'] = "storage_%s" % name
     struct_builder = CodeBlock(*[apply_policy(policy, result, name, sub)
                                  for policy in policies[0]]+[sub]) # struct_declare, struct_behavior, struct_cleanup, sub)
     sub['id'] = id + 1
     sub['fail'] = failure_code(sub)
+    sub['py_ptr'] = "py_%s" % name
+    sub['stor_ptr'] = "storage_%s" % name
     block = CodeBlock(*[apply_policy(policy, result, name, sub)
                         for policy in policies[1]]+[sub]) # run_declare, run_behavior, run_cleanup, sub)
 

theano/gof/link.py

@@ -71,6 +71,7 @@ class Linker(object):
         """
         raise utils.AbstractFunctionError()
 
+    ## DELETEME ##
     def make_function(self, unpack_single = True, **kwargs):
         """
         Returns a function that takes values corresponding to the inputs of the

theano/gof/opt.py

@@ -298,6 +298,7 @@ class MergeOptimizer(Optimizer):
         self.apply_constant_merge(env)
         self.apply_node_merge(env)
 
+merge_optimizer = MergeOptimizer()
 
 def MergeOptMerge(opt):
     """WRITEME
@@ -305,7 +306,7 @@ def MergeOptMerge(opt):
     optimizer in opt and then merges the graph again in case the
     opt introduced additional similarities.
     """
-    merger = MergeOptimizer()
+    merger = merge_optimizer
     return SeqOptimizer([merger, opt, merger])
 
 

theano/gof/type.py

@@ -318,6 +318,8 @@ class Type(object2, PureType, CLinkerType):
 
     """
 
+
+## DELETEME ##
 class SingletonType(Type):
     """WRITEME"""
     __instance = None
@@ -374,6 +376,7 @@ class Generic(SingletonType):
 generic = Generic()
 
 
+## DELETEME ##
 class PropertiedType(Type):
     """WRITEME"""
 

theano/tensor/elemwise.py

@@ -521,10 +521,11 @@ class Elemwise(Op):
         
         try:
             results = ufunc(*ufunc_args)
-        except:
+        except Exception, e:
             errormsg = 'Failed calling ufunc for op', self.scalar_op,\
                         'for params of shape', [arg.shape for arg in ufunc_args]
-            raise Exception, errormsg
+            e.args = e.args + errormsg
+            raise e
         if ufunc.nout == 1: results = [results]
         for result, storage in zip(results, output_storage):
             if storage[0].shape:

theano/tensor/tests/test_nnet.py

@@ -13,6 +13,8 @@ class T_sigmoid(unittest.TestCase):
     def setUp(self):
         numpy.random.seed(9999)
     def test_elemwise(self):
+        print dir(self)
+        assert 0
         TT.verify_grad(self, sigmoid, [numpy.random.rand(3,4)])
 
 class T_softplus(unittest.TestCase):