reading through advanced tutorial

doc/advanced_tutorial/ex1/op.txt

@@ -12,7 +12,7 @@ computations. We'll start by defining multiplication.
 Op's contract
 =============
 
-An Op is any object which defines the following methods:
+An Op (:api:`gof.op.Op`) is any object which defines the following methods:
 
 - **make_node(*inputs)**
     
@@ -99,7 +99,7 @@ An Op is any object which defines the following methods:
     value.
       
   - Equally important, this hash value must not change during the lifetime of
-    self.
+    self.  Op instances should be immutable in this sense.
 
 - **__ne__(self, other)**
 
@@ -128,7 +128,7 @@ An Op is any object which defines the following methods:
   - For more information on the use of this method, see ``grad``.
 
 
-For each method, the *default* is what :api:`theano.gof.Op` defines
+For each method, the *default* is what :api:`theano.gof.op.Op` defines
 for you.  At a bare minimum, a new Op must define ``make_node`` and
 ``perform``, which have no defaults.
 
@@ -149,17 +149,21 @@ Use this list to make sure that you defined everything you need for your Op:
       * Consider making pre-made instances for common parameters. This will simplify usage.
     * No? (usual case for simple Ops)
 
-      * Consider making a singleton of your Op (this can be as simple as ``my_op = MyOp()``). It will simplify usage. [*What is the benefit of using the singleton? How does it simplify usage? We __shouldn't__ use singletons when there __are__ parameters?*]
-      * All instances should compare equal (which is trivial if there is only one of them). [*How do we make sure this is true? Because this checklist should be a list of instructions. Do you describe later on?*]
+      * Consider making a singleton of your Op (this can be as simple as 
+        ``my_op = MyOp()``). This will save you from having to implement __eq__
+        and company.  The singleton approach does not work when an Op instance
+        has parameters (Did you pass anything to __init__?)
   * Always define *make_node* (see make_node section below).
   * Always define *perform* (see perform section below).
   * Do you need performance only C can offer?
 
     * Define *c_code* and *c_code_cleanup* (see HowtoMakeCeeOps)
     * Remember to use the 'c' or 'c|py' linker on graphs using your Op! [*This is described where?*]
-  * Is your Op differentiable?
+  * Is your Op differentiable?  Do you want to use it in differentiable
+    expressions?
+
+    * Define *grad* (see grad section below) 
 
-    * Define *grad* (see grad section below) [*If not, and you don't define *grad*, what will happen if you try to differentiate it?*]
   * Does your Op modify any of its inputs?
 
     * *IMPORTANT:* read the destroyers and viewers section.
@@ -173,8 +177,8 @@ Use this list to make sure that you defined everything you need for your Op:
 [*Consider changing the order of the checklist above and the sections below such that the stuff you ALWAYS have to do, which is the most basic stuff anyhow, goes towards the top.*]
 
 
-Defining mul
-============
+Defining an Op: ``mul``
+=======================
 
 We'll define multiplication as a *binary* operation, even though a
 multiplication Op could take an arbitrary number of arguments.
@@ -220,9 +224,7 @@ node representing the application of Op ``mul`` to inputs ``x`` and
    Theano relies on the fact that if you call the ``make_node`` method
    of Apply's first argument on the inputs passed as the Apply's
    second argument, the call will not fail and the returned Apply
-   instance will be equivalent. We can see that this is trivially true
-   here.
-
+   instance will be equivalent.  This is how graphs are copied.
 
 **perform**
 
@@ -256,6 +258,28 @@ Here, ``z`` is a list of one element. By default, ``z == [None]``.
    that a Python ``float`` must be put there. You should not put, say, an
    ``int`` in ``z[0]`` because Theano assumes Ops handle typing properly.
 
+**eq** and **hash**
+
+Correct implementations of eq and hash permit Theano to recognize one 
+of the most obvious opportunities
+for optimization: not repeatedly computing the same thing.
+
+.. code-block:: python
+
+   def __eq__(self, other):
+       return type(self) == type(other) and (self.name == other.name) and (self.fn == other.fn)
+
+   def __hash__(self):
+       return hash(type(self)) ^ hash(self.name) ^ hash(self.fn)
+
+When theano compiles a graph, most Modes first :term:`merge` the graph (this is
+done by the :api:`MergeOptimizer`.) The principle of merging is that if the
+inputs to two different :ref:`Applies <apply>` are identical and the :ref:`op`s
+applied to them compare equal, then those two Apply instances are guaranteed to
+produce the same outputs.
+So Theano will only compute one of them.
+
+
 
 Trying out our new Op
 =====================
@@ -282,6 +306,9 @@ Traceback (most recent call last):
   File "<stdin>", line 2, in make_node
 AttributeError: 'int' object has no attribute 'type'
 
+Automatic Constant Wrapping
+---------------------------
+
 Well, ok. We'd like our Op to be a bit more flexible. This can be done
 by modifying ``make_node`` to accept Python ``int`` or ``float`` as
 ``x`` and/or ``y``:
@@ -312,6 +339,14 @@ is a :ref:`variable` we statically know the value of.
 
 Now the code works the way we want it to.
 
+.. note:: 
+   Most Theano Ops follow this convention of up-casting literal 
+   make_node arguments to Constants.
+   This makes typing expressions more natural.  If you do 
+   not want a constant somewhere in your graph, you have to pass a Variable
+   (like ``double('x')`` here).
+
+
 
 Final version
 =============
@@ -385,3 +420,4 @@ your disposal to create these objects as efficiently as possible.
 
 **Exercise**: Make a generic DoubleOp, where the number of
 arguments can also be given as a parameter.
+

doc/advanced_tutorial/ex1/type.txt

@@ -10,11 +10,12 @@ Making the double type
 Type's contract
 ===============
 
-In Theano's framework, a Type is any object which defines the following
+In Theano's framework, a ``Type`` (:api:`gof.type.Type`)
+is any object which defines the following
 methods. To obtain the default methods described below, the Type should
-be an instance of :api:`theano.gof.Type` or should be an instance of a
-subclass of :api:`theano.gof.Type`. If you will write all methods yourself,
-you need not use an instance of :api:`theano.gof.Type`.
+be an instance of ``Type`` or should be an instance of a
+subclass of ``Type``. If you will write all methods yourself,
+you need not use an instance of ``Type``.
 
 Methods with default arguments must be defined with the same signature,
 i.e.  the same default argument names and values. If you wish to add
@@ -72,10 +73,22 @@ default values.
 
   - *Default*: ``make_variable``
 
+- **__eq__(self, other)**:
 
-For each method, the *default* is what :api:`theano.gof.Type` defines
-for you. So, if you create an instance of :api:`theano.gof.Type` or an
-instance of a subclass of :api:`theano.gof.Type`, you
+  - Used to compare Type instances themselves
+
+  - *Default*: ``object.__eq__``
+
+- **__hash__(self)**:
+
+  - Types should not be mutable, so it should be Ok to define a hash function.
+    Typically this function should hash all of the terms involved in ``__eq__``.
+
+  - *Default*: ``id(self)``
+
+For each method, the *default* is what ``Type`` defines
+for you. So, if you create an instance of ``Type`` or an
+instance of a subclass of ``Type``, you
 must define ``filter``. You might want to override ``values_eq_approx``,
 as well as ``values_eq``. The other defaults generally need not be
 overridden.
@@ -185,13 +198,11 @@ instances of ``Double`` are technically the same Type. However, different
 >>> double1 == double2
 False
 
-Theano compares Types using ``==`` to see if they are the same. If
-the inputs of two different :ref:`Applies <apply>` are the same
-and two :ref:`op`s applied to them compare equal, then only one of those ops
-must be evaluated.  (This is sometimes called a :term:`merging <merge>` and it
-is done by the :api:`MergeOptimizer`.)
+Theano compares Types using ``==`` to see if they are the same.
+This happens in DebugMode.  Also, ops can (and should) ensure that their inputs
+have the expected Type by checking something like ``if x.type == lvector``.
 
-There are several ways to make sure graphs are merged properly:
+There are several ways to make sure that equality testing works properly:
 
  #. Define ``Double.__eq__`` so that instances of type Double
     are equal. For example:
@@ -221,7 +232,8 @@ it is can be confusing what these represent semantically. Here is an
 attempt to clear up the confusion:
 
 
-* An **instance of Type** is a set of constraints on real data. It is
+* An **instance of Type** (or an instance of a subclass)
+  is a set of constraints on real data. It is
   akin to a primitive type or class in C. It is a *static*
   annotation.
 
@@ -231,12 +243,15 @@ attempt to clear up the confusion:
   that Type instance. If you were to parse the C expression ``c = a +
   b;``, ``a``, ``b`` and ``c`` would all be Variable instances.
 
-* A **subclass of Type** represents a set of Type instances that share
+* A **subclass of Type** is a way of implementing 
+  a set of Type instances that share
   structural similarities. In the ``double`` example that we are doing,
   there is actually only one Type in that set, therefore the subclass
   doesn't represent anything that one of its instances doesn't. In this
-  case it is a singleton, a set with one element. However, the TensorType
-  class which is a subclass of Type represents a set of types of tensors
+  case it is a singleton, a set with one element. However, the
+  :api:`TensorType`
+  class in Theano (which is a subclass of Type)
+  represents a set of types of tensors
   parametrized by their data type or number of dimensions. We could say
   that subclassing Type builds a hierarchy of Types which is based upon
   structural similarity rather than compatibility.
@@ -266,4 +281,5 @@ Final version
 
 
 We add one utility function, ``__str__``. That way, when we print
-``double``, it will print out something sensible.
+``double``, it will print out something intelligible.
+

doc/advanced_tutorial/inplace.txt

@@ -7,7 +7,9 @@ Views and inplace operations
 
 Theano allows the definition of Ops which return a :term:`view` on one
 of their inputs or operates :term:`inplace` on one or several
-inputs. However, in order to work correctly, these Ops need to
+inputs. This allows more efficient operations on numpy's ndarray data type than
+would be possible otherwise.
+However, in order to work correctly, these Ops need to
 implement an additional interface.
 
 Theano recognizes views and inplace operations specially. It ensures
@@ -58,8 +60,8 @@ purpose, you would set the ``view_map`` field as follows:
    myop.view_map = {0: [0]}
 
 
-What this means is that the first output (rank 0) is a view of the
-first input (rank 0). Even though the interface allows a list of
+What this means is that the first output (position 0) is a view of the
+first input (position 0). Even though the interface allows a list of
 inputs that are a view of a given output, this feature is currently
 unsupported. Here are more examples:
 
@@ -76,7 +78,7 @@ unsupported. Here are more examples:
    myop.view_map = {0: [0], # first output is a view of first input
                     1: [0]} # *AND* second output is *ALSO* a view of first input
 
-   myop.view_map = {0: [0, 1]} # THIS IS NOT SUPPORTED! Only put a single input number in the list!
+   myop.view_map = {0: [0, 1]} # THIS IS NOT SUPPORTED YET! Only put a single input number in the list!
 
 
 .. _inplace:
@@ -139,7 +141,7 @@ it could add one to each byte *in* the buffer ``x``, therefore
 changing it. That would be an inplace Op.
 
 Theano needs to be notified of this fact. The syntax is similar to
-that of view_map, so I will copy paste cleverly:
+that of view_map:
 
 
 .. code-block:: python
@@ -147,8 +149,8 @@ that of view_map, so I will copy paste cleverly:
    myop.destroy_map = {0: [0]}
 
 
-What this means is that the first output (rank 0) operates inplace on the
-first input (rank 0).
+What this means is that the first output (position 0) operates inplace on the
+first input (position 0).
 
 
 .. code-block:: python
@@ -167,8 +169,8 @@ first input (rank 0).
    # unlike for views, the previous line is legal and supported
 
 
-Purely destructive operations
-=============================
+Destructive Operations
+======================
 
 While some operations will operate inplace on their inputs, some might
 simply destroy or corrupt them. For example, an Op could do temporary