Note that the ability to change the seed from one nosetest to another, is incompatible with the method of hard-coding the baseline results (against which we compare the theano outputs). These must then be determined "algorithmically". Although this represents more work, the test suite will be better because of it.
Note that the ability to change the seed from one nosetest to another, is incompatible with the method of hard-coding the baseline variables (against which we compare the theano outputs). These must then be determined "algorithmically". Although this represents more work, the test suite will be better because of it.
@@ -26,17 +26,17 @@ The idea here is that we've compiled the symbolic graph (``2*x``) into a functio
...
@@ -26,17 +26,17 @@ The idea here is that we've compiled the symbolic graph (``2*x``) into a functio
Inputs
Inputs
======
======
The ``inputs`` argument to ``theano.function`` is a list, containing the ``Result`` instances for which values will be specified at the time of the function call. But inputs can be more than just Results.
The ``inputs`` argument to ``theano.function`` is a list, containing the ``Variable`` instances for which values will be specified at the time of the function call. But inputs can be more than just Variables.
``In`` instances let us attach properties to ``Results`` to tell function more about how to use them.
``In`` instances let us attach properties to ``Variables`` to tell function more about how to use them.
**In(result, name=None, value=None, update=None, mutable=False)** returns an ``In`` instance:
**In(variable, name=None, value=None, update=None, mutable=False)** returns an ``In`` instance:
- ``result``: a Result instance.
- ``variable``: a Variable instance.
This will be assigned a value before running the function,
This will be assigned a value before running the function,
not computed from its owner.
not computed from its owner.
- ``name``: Any type. (If autoname_input=True, defaults to result.name).
- ``name``: Any type. (If autoname_input=True, defaults to variable.name).
If name is a valid Python identifier, this input can be set by
If name is a valid Python identifier, this input can be set by
``kwarg``, and its value can be accessed by ``self.<name>``.
``kwarg``, and its value can be accessed by ``self.<name>``.
...
@@ -49,9 +49,9 @@ The ``inputs`` argument to ``theano.function`` is a list, containing the ``Resul
...
@@ -49,9 +49,9 @@ The ``inputs`` argument to ``theano.function`` is a list, containing the ``Resul
Default: ``None``
Default: ``None``
- ``update``: Result instance
- ``update``: Variable instance
This expression Result will replace ``value`` after each function call.
This expression Variable will replace ``value`` after each function call.
Default: ``None``
Default: ``None``
...
@@ -63,7 +63,7 @@ The ``inputs`` argument to ``theano.function`` is a list, containing the ``Resul
...
@@ -63,7 +63,7 @@ The ``inputs`` argument to ``theano.function`` is a list, containing the ``Resul
- ``autoname``: Bool
- ``autoname``: Bool
``True``: if ``name`` is None and the Result has a name, it will be taken
``True``: if ``name`` is None and the Variable has a name, it will be taken
as the input's name.
as the input's name.
``False``: the name is the exact value passed as the name parameter
``False``: the name is the exact value passed as the name parameter
...
@@ -121,7 +121,7 @@ Advanced: Sharing Storage Between Functions
...
@@ -121,7 +121,7 @@ Advanced: Sharing Storage Between Functions
-------------------------------------------
-------------------------------------------
``value`` can be a :api:`theano.gof.Container` as well as a literal.
``value`` can be a :api:`theano.gof.Container` as well as a literal.
This permits linking a value of a Result in one function to the value of a Result in another function.
This permits linking a value of a Variable in one function to the value of a Variable in another function.
By using a ``Container`` as a value we can implement shared variables between functions.
By using a ``Container`` as a value we can implement shared variables between functions.
For example, consider the following program.
For example, consider the following program.
...
@@ -141,7 +141,7 @@ For example, consider the following program.
...
@@ -141,7 +141,7 @@ For example, consider the following program.
The functions ``inc`` and ``dec`` operate on a shared internal value for ``s``.
The functions ``inc`` and ``dec`` operate on a shared internal value for ``s``.
Theano's Module system uses this mechanism to share storage between Methods.
Theano's Module system uses this mechanism to share storage between Methods.
The container being shared doesn't have to correspond to the same Result in both functions,
The container being shared doesn't have to correspond to the same Variable in both functions,
but that's usually how this mechanism is used.
but that's usually how this mechanism is used.
Input Argument Restrictions
Input Argument Restrictions
...
@@ -161,11 +161,11 @@ The following restrictions apply to the inputs to ``theano.function``:
...
@@ -161,11 +161,11 @@ The following restrictions apply to the inputs to ``theano.function``:
have the same name, then the function will raise an exception. [***Which
have the same name, then the function will raise an exception. [***Which
exception?**]
exception?**]
- Two ``In`` instances may not name the same Result. I.e. you cannot
- Two ``In`` instances may not name the same Variable. I.e. you cannot
give the same parameter multiple times.
give the same parameter multiple times.
If no name is specified explicitly for an In instance, then its name
If no name is specified explicitly for an In instance, then its name
will be taken from the Result's name. Note that this feature can cause
will be taken from the Variable's name. Note that this feature can cause
harmless-looking input lists to not satisfy the two conditions above.
harmless-looking input lists to not satisfy the two conditions above.
In such cases, Inputs should be named explicitly to avoid problems
In such cases, Inputs should be named explicitly to avoid problems
such as duplicate names, and named arguments preceding unnamed ones.
such as duplicate names, and named arguments preceding unnamed ones.
...
@@ -198,7 +198,7 @@ Both ``value`` and ``container`` properties provide dictionary-like access based
...
@@ -198,7 +198,7 @@ Both ``value`` and ``container`` properties provide dictionary-like access based
- integer keys: you can look up a value/container by its position in the input list;
- integer keys: you can look up a value/container by its position in the input list;
- name keys: you can look up a value/container by its name;
- name keys: you can look up a value/container by its name;
- Result keys: you can look up a value/container by the Result it corresponds to.
- Variable keys: you can look up a value/container by the Variable it corresponds to.
In addition to these access mechanisms, there is an even more convenient
In addition to these access mechanisms, there is an even more convenient
method to access values by indexing a Function directly by typing
method to access values by indexing a Function directly by typing
...
@@ -234,7 +234,7 @@ Input Shortcuts
...
@@ -234,7 +234,7 @@ Input Shortcuts
Every element of the inputs list will be upgraded to an In instance if necessary.
Every element of the inputs list will be upgraded to an In instance if necessary.
- a Result instance ``r`` will be upgraded like ``In(r)``
- a Variable instance ``r`` will be upgraded like ``In(r)``
- a tuple ``(name, r)`` will be ``In(r, name=name)``
- a tuple ``(name, r)`` will be ``In(r, name=name)``
...
@@ -285,13 +285,13 @@ Outputs
...
@@ -285,13 +285,13 @@ Outputs
The ``outputs`` argument to function can be one of
The ``outputs`` argument to function can be one of
- ``None``, or
- ``None``, or
- a Result or ``Out`` instance, or
- a Variable or ``Out`` instance, or
- a list of Results or ``Out`` instances.
- a list of Variables or ``Out`` instances.
An ``Out`` instance is a structure that lets us attach options to individual output ``Result`` instances,
An ``Out`` instance is a structure that lets us attach options to individual output ``Variable`` instances,
similarly to how ``In`` lets us attach options to individual input ``Result`` instances.
similarly to how ``In`` lets us attach options to individual input ``Variable`` instances.
**Out(result, borrow=False)** returns an ``Out`` instance:
**Out(variable, borrow=False)** returns an ``Out`` instance:
* ``borrow``
* ``borrow``
...
@@ -304,9 +304,9 @@ similarly to how ``In`` lets us attach options to individual input ``Result`` in
...
@@ -304,9 +304,9 @@ similarly to how ``In`` lets us attach options to individual input ``Result`` in
If a single ``Result`` or ``Out`` instance is given as argument, then the compiled function will return a single value.
If a single ``Variable`` or ``Out`` instance is given as argument, then the compiled function will return a single value.
If a list of ``Result`` or ``Out`` instances is given as argument, then the compiled function will return a list of their values.
If a list of ``Variable`` or ``Out`` instances is given as argument, then the compiled function will return a list of their values.
@@ -18,7 +18,7 @@ Examples of parameterized Ops in theano:
...
@@ -18,7 +18,7 @@ Examples of parameterized Ops in theano:
``Reduce(<scalar op>, <axes>)``
``Reduce(<scalar op>, <axes>)``
reduces the specified axes using the provided scalar op.
reduces the specified axes using the provided scalar op.
``Add(<output type inferrer>)``
``Add(<output type inferrer>)``
adds scalars and puts the result in a scalar whose type is inferred from the input types using ``output_type_inferrer(*inputs)``
adds scalars and puts the variable in a scalar whose type is inferred from the input types using ``output_type_inferrer(*inputs)``
``Composite(<graph>)``
``Composite(<graph>)``
makes a single Op out of a graph of scalar operations.
makes a single Op out of a graph of scalar operations.
...
@@ -46,14 +46,14 @@ The ``make_node`` method is expected to have the following signature:
...
@@ -46,14 +46,14 @@ The ``make_node`` method is expected to have the following signature:
make_node(self, *inputs)
make_node(self, *inputs)
``inputs`` may be a list of anything that the user wants to provide as symbolic input (symbolic: standing for the actual values that will be passed when the graph is compiled into an executable function). [*The Theano intro should describe symbolic in greater depth, and we should link to that from here.*] This may or may not include Result instances (but if you want the inputs of this Op to sometimes be outputs of another Op, then the inputs should be Result instances). [*What else could they be? Constant, Values, ...*] The return value should be an instance of [GraphStructures Apply] (see the example below). Here are the tasks typically handled in ``make_node``.
``inputs`` may be a list of anything that the user wants to provide as symbolic input (symbolic: standing for the actual values that will be passed when the graph is compiled into an executable function). [*The Theano intro should describe symbolic in greater depth, and we should link to that from here.*] This may or may not include Variable instances (but if you want the inputs of this Op to sometimes be outputs of another Op, then the inputs should be Variable instances). [*What else could they be? Constant, Values, ...*] The return value should be an instance of [GraphStructures Apply] (see the example below). Here are the tasks typically handled in ``make_node``.
* Check that the inputs are valid (type checking, etc.). [*Since we don't actually have values, what can we do besides type checking?*]
* Check that the inputs are valid (type checking, etc.). [*Since we don't actually have values, what can we do besides type checking?*]
* If needed, wrap the inputs in Result instances with the proper type.
* If needed, wrap the inputs in Variable instances with the proper type.
* Make the Result instances that will serve as the outputs of the node.
* Make the Variable instances that will serve as the outputs of the node.
The ``inputs`` and ``outputs`` arguments to ``Apply`` must be lists of ``Result`` instances (or instances of subclasses of ``Result``). The inputs given to ``Apply`` do not have to be the same as the inputs passed to ``make_node``, but it is recommended that the order corresponds. [*why?*] The behavior of ``make_node`` should not depend on the structure of the graph of [*or?*] its inputs: it may look at the type and type fields of its inputs, but not at their owner field, because modifications to the graph structure do not use ``make_node``. [*???*]
The ``inputs`` and ``outputs`` arguments to ``Apply`` must be lists of ``Variable`` instances (or instances of subclasses of ``Variable``). The inputs given to ``Apply`` do not have to be the same as the inputs passed to ``make_node``, but it is recommended that the order corresponds. [*why?*] The behavior of ``make_node`` should not depend on the structure of the graph of [*or?*] its inputs: it may look at the type and type fields of its inputs, but not at their owner field, because modifications to the graph structure do not use ``make_node``. [*???*]
Example:
Example:
...
@@ -66,14 +66,14 @@ Example:
...
@@ -66,14 +66,14 @@ Example:
def make_node(self, x, y):
def make_node(self, x, y):
# note 1: constant, int64 and Scalar are defined in theano.scalar
# note 1: constant, int64 and Scalar are defined in theano.scalar
# note 2: constant(x) is equivalent to Constant(type = int64, data = x)
# note 2: constant(x) is equivalent to Constant(type = int64, data = x)
# note 3: the call int64() is equivalent to Result(type = int64) or Result(type = Scalar(dtype = 'int64'))
# note 3: the call int64() is equivalent to Variable(type = int64) or Variable(type = Scalar(dtype = 'int64'))
if isinstance(x, int):
if isinstance(x, int):
x = constant(x)
x = constant(x)
elif not isinstance(x, Result) or not x.type == int64:
elif not isinstance(x, Variable) or not x.type == int64:
raise TypeError("expected an int64 Scalar")
raise TypeError("expected an int64 Scalar")
if isinstance(y, int):
if isinstance(y, int):
y = constant(y)
y = constant(y)
elif not isinstance(y, Result) or not x.type == int64:
elif not isinstance(y, Variable) or not x.type == int64:
raise TypeError("expected an int64 Scalar")
raise TypeError("expected an int64 Scalar")
inputs = [x, y]
inputs = [x, y]
outputs = [int64()]
outputs = [int64()]
...
@@ -82,12 +82,12 @@ Example:
...
@@ -82,12 +82,12 @@ Example:
#...
#...
add = Add() # I make an instance of Add
add = Add() # I make an instance of Add
node1 = add.make_node(int64(), int64()) # I make a node with two Result inputs
node1 = add.make_node(int64(), int64()) # I make a node with two Variable inputs
node2 = add.make_node(1, 2) # this works too
node2 = add.make_node(1, 2) # this works too
node3 = add.make_node(int64(), 79) # this works three
node3 = add.make_node(int64(), 79) # this works three
node4 = add.make_node(float64(), int64()) # this raises a TypeError
node4 = add.make_node(float64(), int64()) # this raises a TypeError
[*What type is an instance of Add? It's an Apply? But that's not a Result, and cannot be used as input for another Op.*]
[*What type is an instance of Add? It's an Apply? But that's not a Variable, and cannot be used as input for another Op.*]
Two Apply nodes ``node1`` and ``node2`` are *assumed* by the compiler to represent the same behavior if:
Two Apply nodes ``node1`` and ``node2`` are *assumed* by the compiler to represent the same behavior if:
1. ``node1.op == node2.op``
1. ``node1.op == node2.op``
...
@@ -99,7 +99,7 @@ It is considered an *error* to have conditions 1 and 2 but not condition 3. A co
...
@@ -99,7 +99,7 @@ It is considered an *error* to have conditions 1 and 2 but not condition 3. A co
``__call__``
``__call__``
----------------
----------------
In ``Op``, ``__call__`` is defined in terms of ``make_node``. Instead of returning a node, it returns the output Results directly, which is practical from a UI standpoint. Here is pseudocode:
In ``Op``, ``__call__`` is defined in terms of ``make_node``. Instead of returning a node, it returns the output Variables directly, which is practical from a UI standpoint. Here is pseudocode:
* *node*: a pointer to an Apply instance - ``node`` is assumed to be produced by a previous call to ``self.make_node``.
* *node*: a pointer to an Apply instance - ``node`` is assumed to be produced by a previous call to ``self.make_node``.
* *inputs*: *not* the same as ``node.inputs`` - it is a list of values. [*i.e. actually data, not just symbolic stuff?*]
* *inputs*: *not* the same as ``node.inputs`` - it is a list of values. [*i.e. actually data, not just symbolic stuff?*]
* *output_storage*: *not* the same as ``node.outputs`` - it is a list of lists of length 1 where the results of the computation must be put.
* *output_storage*: *not* the same as ``node.outputs`` - it is a list of lists of length 1 where the variables of the computation must be put.
[*Can you explain better how inputs is not node.inputs and output_storage is not node.outputs?*]
[*Can you explain better how inputs is not node.inputs and output_storage is not node.outputs?*]
...
@@ -138,7 +138,7 @@ Here is an example of a properly defined ``perform``:
...
@@ -138,7 +138,7 @@ Here is an example of a properly defined ``perform``:
# this does z = x + y
# this does z = x + y
x, y = inputs # extract the two inputs
x, y = inputs # extract the two inputs
z, = output_storage # extract the one storage (the comma after z is not optional)
z, = output_storage # extract the one storage (the comma after z is not optional)
z[0] = x + y # we must put the result in z[0]
z[0] = x + y # we must put the variable in z[0]
...
...
add = Add() # I make an instance of Add
add = Add() # I make an instance of Add
...
@@ -175,8 +175,8 @@ grad
...
@@ -175,8 +175,8 @@ grad
where:
where:
* ``inputs`` is a list of Result instances. It is assumed to be the ``inputs`` field of a node produced by ``make_node``.
* ``inputs`` is a list of Variable instances. It is assumed to be the ``inputs`` field of a node produced by ``make_node``.
* ``output_gradients`` is a list of Result instances. They have the same properties as the outputs of the node, but are filled with gradient values.
* ``output_gradients`` is a list of Variable instances. They have the same properties as the outputs of the node, but are filled with gradient values.
Essentially, the semantics are:
Essentially, the semantics are:
...
@@ -192,7 +192,7 @@ Essentially, the semantics are:
...
@@ -192,7 +192,7 @@ Essentially, the semantics are:
return gz*dz/dx + gw*dw/dx, gz*dz/dy + gw*dw/dy
return gz*dz/dx + gw*dw/dx, gz*dz/dy + gw*dw/dy
More specifically,
More specifically,
``grad`` must return a list or tuple of input gradients, as many as there are inputs. Let C be a Result (currently assumed to be a scalar) that depends through a theano symbolic expression on the node outputs. Then each output_gradients[i] represents symbolically dC/doutputs[i]. The returned input gradients should represent symbolically dC/dinputs[i].
``grad`` must return a list or tuple of input gradients, as many as there are inputs. Let C be a Variable (currently assumed to be a scalar) that depends through a theano symbolic expression on the node outputs. Then each output_gradients[i] represents symbolically dC/doutputs[i]. The returned input gradients should represent symbolically dC/dinputs[i].
Example:
Example:
...
@@ -253,7 +253,7 @@ Example: if we expect to call the op repeatedly on incrementally bigger inputs,
...
@@ -253,7 +253,7 @@ Example: if we expect to call the op repeatedly on incrementally bigger inputs,
A ``Member`` represents a state variable (i.e., whose value remains after a ``Method`` is called). It will be named automatically after that field and it will be an implicit input of all ``Methods`` of the ``Module``. Its storage (i.e. where the value is stored) will be shared by all ``Methods`` of the ``Module``.
A ``Member`` represents a state variable (i.e., whose value remains after a ``Method`` is called). It will be named automatically after that field and it will be an implicit input of all ``Methods`` of the ``Module``. Its storage (i.e. where the value is stored) will be shared by all ``Methods`` of the ``Module``.
A ``Result`` which is the result of a previous computation (by opposition to being ``updated``) is not a ``Member``. Internally this is called an External. You should not need to care about this.
A ``Variable`` which is the variable of a previous computation (by opposition to being ``updated``) is not a ``Member``. Internally this is called an External. You should not need to care about this.
For sharing state between modules, see ``Inner Module`` section.
For sharing state between modules, see ``Inner Module`` section.
...
@@ -100,7 +100,7 @@ Module Interface
...
@@ -100,7 +100,7 @@ Module Interface
def resolve(self, symbol, filter = None)
def resolve(self, symbol, filter = None)
Resolves a symbol in this module. The symbol can be a string or a ``Result``. If the string contains dots (eg ``"x.y"``), the module will resolve the symbol hierarchically in its inner modules. The filter argument is None or a class and it can be used to restrict the search to ``Member`` or ``Method`` instances for example.
Resolves a symbol in this module. The symbol can be a string or a ``Variable``. If the string contains dots (eg ``"x.y"``), the module will resolve the symbol hierarchically in its inner modules. The filter argument is None or a class and it can be used to restrict the search to ``Member`` or ``Method`` instances for example.
currently, there is no way for a grad() method to distinguish between cases 3
currently, there is no way for a grad() method to distinguish between cases 3
...
@@ -123,7 +123,7 @@ Guillaume can you make sure to hit these points:
...
@@ -123,7 +123,7 @@ Guillaume can you make sure to hit these points:
* There are a lot of tests that define their own epsilon, but this should be standardized. e.g. in test_elemwise.py ``self.failUnless((numpy.abs(f(xv) - zv) < 1e-10).all())``
* There are a lot of tests that define their own epsilon, but this should be standardized. e.g. in test_elemwise.py ``self.failUnless((numpy.abs(f(xv) - zv) < 1e-10).all())``
* If the expected result of a test is that an Exception is thrown, how do we correctly detect and handle that?
* If the expected variable of a test is that an Exception is thrown, how do we correctly detect and handle that?