Skip to content

P
pytensor
提交 09bce224 authored 作者: Olivier Breuleux's avatar Olivier Breuleux
浏览文件
操作

merge

上级 bcea84cf 24f4ad19
隐藏空白字符变更
内嵌 并排
正在显示 包含 831 行增加 和 348 行删除
doc/doc/sandbox.txt
浏览文件 @ 09bce224
......@@ -193,3 +193,9 @@ How to reuse (overwrite) a storage tensor
``theano.compile.io.Out(gw1, borrow = True)`` for that value in
``compile.function``
=========================================
ProfileMode
=========================================
*** write up how to use it ***
doc/index.txt
浏览文件 @ 09bce224
......@@ -5,43 +5,49 @@
Theano
======
Theano is a Python library aiming to allow definition, optimization
and efficient evaluation of mathematical expressions involving
multi-dimensional arrays (though it may be extended to support many
other types). Theano melds some aspects of a computer algebra system
(CAS) with aspects of an optimizing compiler. This is particularly
useful in fields such as machine learning where complicated algorithms
must be run over large amounts of data.
Theano supports a wide range of numerical types in multiple
dimensions, a rapidly growing number of well-tested operations as well
as utilities to compute the gradient of an expression with respect to
another. Symbolic expressions may be compiled into functions, which
work merrily on the same data structures as numpy_, allowing for easy
interoperability.
Theano's compiler applies many optimizations of varying
complexity. These optimizations include, but are not limited to
constant folding, merging of similar subgraphs (to avoid calculating
the same values more than once), simple arithmetic simplification
(``x*y/x -> y``), inserting efficient BLAS_ operations and using
inplace operations wherever it is safe to do so. Theano also defines
several optimizations which improve the numerical stability of
computations and it provides a framework to add and test new
optimizers.
Theano was written at the LISA_ to support the development of
Theano is a Python library that allows you to definite, optimize, and
efficiently evaluate mathematical expressions involving multi-dimensional
arrays. It can be extended to support other types. Theano melds some
aspects of a computer algebra system (CAS) with aspects of an optimizing
compiler. It can even transform some or all of the expression into C code
and compile it into native machine instructions. This combination of CAS
with optimizing compilation is particularly useful for computational
fields in which complicated mathematical expressions are evaluated
numerous times over large data sets.
Theano was written at the LISA_ lab to support the development of
efficient machine learning algorithms while minimizing human
time. Theano was named after the `Greek mathematician`_ who may have
been Pythagoras' wife.
time. We use it especially in gradient-based learning techniques.
Theano supports a range of numerical types in multiple dimensions and
a number of well-tested operations. It also allows you to compute the
gradient of an expression with respect to another. Symbolic expressions
may be compiled into functions, which work on the same data structures
as numpy_, allowing for easy interoperability.
Theano's compiler applies many optimizations of varying complexity
to these symbolic expressions. These optimizations include, but are
not limited to:
* constant folding
* merging of similar subgraphs, to avoid calculating the same values more than once
* simple arithmetic simplification (``x*y/x -> y``)
* inserting efficient BLAS_ operations
* using inplace operations wherever it is safe to do so.
Theano defines several optimizations which improve the numerical
stability of computations. It also provides a framework to add and test
new optimizers.
Theano was named after the `Greek mathematician`_, who may have
been Pythagoras' wife.
Theano is released under a BSD license (:ref:`link <license>`)
Sneak peek
==========
Here's a very simple example of how to use Theano. It doesn't show
Here is a simple example of how to use Theano. It doesn't show
off many of Theano's features, but it illustrates concretely what
Theano is.
......@@ -66,9 +72,8 @@ Theano is.
Theano is not a programming language in the normal sense because you
write a program in Python that builds expressions for Theano. Still
it is like a programming language in the sense that to use theano, you
have to
write a program in Python that builds expressions for Theano. Still
it is like a programming language in the sense that you have to
- declare variables (``a,b``) and give their types
......@@ -77,8 +82,8 @@ have to
- compile expression graphs to functions in order to use them for computation.
It is good to think of ``theano.function`` as the interface to a
compiler which builds a callable object from a purely symbolic graph;
one of theano's most important features is that ``theano.function``
compiler which builds a callable object from a purely symbolic graph.
One of theano's most important features is that ``theano.function``
can optimize a graph and even compile some or all of it into native
machine instructions.
......@@ -95,18 +100,18 @@ package, so what does Theano do that Python and numpy do not?
parts your expression graph into native machine code, which runs
much faster than python.
- *symbolic differentiation*: Theano can convert a symbolic graph
build symbolic graphs for computing gradients.
- *symbolic differentiation*: Theano can automatic build symbolic graphs
for computing gradients.
- *stability optimizations*: Theano can recognize numerically unstable
expressions and compute them with more stable algorithms.
There also exists symbolic packages in Python, namely sympy_. Theano
is different from them in the sense that while it allows symbolic
manipulation it puts more emphasis on the evaluation of these
expressions and being able to repeatedly evaluate them on many
different sets of inputs. It is also better suited to handling very
large tensors which have no assumed structures.
There exist another symbolic package in Python, namely sympy_. Theano
is different from sympy in the sense that while Theano allows symbolic
manipulation it puts more emphasis on the evaluation of these expressions
and being able to repeatedly evaluate them on many different inputs. Theano
is also better suited to handling very large tensors which have no
assumed structures.
If numpy_ is to be compared to MATLAB_ and sympy_ to Mathematica_,
Theano is a sort of hybrid of the two which tries to make the best of
......@@ -145,10 +150,9 @@ issues that concern the end users.
Questions, comments, praise, criticism as well as bug reports should
be submitted to these mailing lists.
We welcome all kinds of contributions. Our `task list`_ is full of
interesting ideas awaiting a champion. If you have any questions
regarding how to extend Theano, please feel free to ask on the
theano-dev_ mailing list.
We welcome all kinds of contributions. If you have any questions
regarding how to extend Theano, please feel free to ask on the theano-dev_
mailing list.
......
doc/install.txt
浏览文件 @ 09bce224
......@@ -12,15 +12,31 @@ Requirements
In order to use Theano, the following libraries and software will need
to be installed:
- linux or OS-X operating system
- python >=2.5
- numpy >=1.2 (earlier versions have memory leaks)
- SciPy (specifically numpy, sparse, weave). We recommend scipy >=0.7 if you are using sparse matrices, because scipy.sparse is buggy in 0.6. (scipy.csc_matrix dot has a bug with singleton dimensions. There may be more bugs.)
- g++, python-dev (optional but highly recommended, to compile generated C code)
- sphinx >=0.5.1, pygments (optional, to build documentation) (also latex and dvipng if you want math to show up as images...)
- mercurial (optional, to download the source)
- nose (nosetests) (optional, for testing)
Linux or OS-X operating system
We develop mainly on 64-bit Linux machines. 32-bit architectures are
not well-tested.
python >= 2.5
`numpy <http://numpy.scipy.org/>`_ >= 1.2
Earlier versions have memory leaks.
`SciPy <http://scipy.org>`_
Specifically numpy, sparse, and weave. We recommend scipy
>=0.7 if you are using sparse matrices, because scipy.sparse
is buggy in 0.6. (scipy.csc_matrix dot has a bug with singleton
dimensions. There may be more bugs.)
The following libraries and software are optional:
g++, python-dev
Highly recommended, to compile generated C code.
`nose <http://somethingaboutorange.com/mrl/projects/nose/>`_
Recommended, to run Theano's test-suite.
`sphinx <http://sphinx.pocoo.org/>`_ >=0.5.1, `pygments <http://pygments.org/>`_
Used to build documentation. latex and dvipng
are also necessary for math to show up as images.
`mercurial <http://www.selenic.com/mercurial/>`_
To download bleeding-edge
------------
......
doc/tutorials/basic/adding.txt
浏览文件 @ 09bce224
......@@ -37,19 +37,15 @@ objects).
>>> x = T.dscalar('x')
>>> y = T.dscalar('y')
In Theano, all symbols must be typed. In particular, ``T.dscalar`` is
the type we assign to "0-dimensional arrays of doubles". It is a
Theano :term:`Type`. Therefore, you can guess that by calling
``T.dscalar`` with a string argument, you create a :term:`Result`
representing a floating-point scalar quantity with the given name (if
you provide no argument, the symbol will be unnamed, which can cause
difficulties in debugging).
Note that ``dscalar`` is not a class and that therefore neither ``x``
nor ``y`` are actually instances of ``dscalar``. They are instances of
:api:`TensorResult <theano.tensor.basic.TensorResult>`. It is however
assigned the theano Type ``dscalar`` in its ``type`` field, as you can
see here:
In Theano, all symbols must be typed. In particular, ``T.dscalar``
is the type we assign to "0-dimensional arrays (`scalar`) of doubles
(`d`)". It is a Theano :term:`Type`.
``dscalar`` is not a class. Therefore, neither ``x`` nor ``y``
are actually instances of ``dscalar``. They are instances of
:api:`TensorResult <theano.tensor.basic.TensorResult>`. ``x`` and ``y``
are, however, assigned the theano Type ``dscalar`` in their ``type``
field, as you can see here:
>>> type(x)
<class 'theano.tensor.basic.TensorResult'>
......@@ -60,9 +56,14 @@ Tensor(float64, scalar)
>>> x.type == T.dscalar
True
Ditto for ``y``. You may learn more about the structures in Theano in
You can learn more about the structures in Theano in
the :ref:`advtutorial` and in :ref:`graphstructures`.
By calling ``T.dscalar`` with a string argument, you create a
:term:`Result` representing a floating-point scalar quantity with the
given name. If you provide no argument, the symbol will be unnamed. Names
are not require, but they can aid debugging.
-------------------------------------------
**Step 2**
......@@ -83,14 +84,14 @@ x + y
**Step 3**
The last step is to create a function taking ``x`` and ``y`` as inputs
and giving out ``z`` as output:
and giving ``z`` as output:
>>> f = function([x, y], z)
The first argument to ``function`` is a list of :term:`Results
<Result>` that will be provided as inputs to the function. The second
argument is a single Result that we want to see as output *or* a list
of output results.
The first argument to ``function`` is a list of :term:`Results <Result>`
that will be provided as inputs to the function. The second argument
is a single Result *or* a list of Results. For either case, the second
argument is what we want to see as output when we apply the function.
``f`` may then be used like a normal Python function.
......
doc/tutorials/basic/index.txt
浏览文件 @ 09bce224
......@@ -17,7 +17,7 @@ installed:
>>> from theano import *
Many of symbols you will need to use lie in the ``tensor`` subpackage
Many of symbols you will need to use are in the ``tensor`` subpackage
of theano. Let's import that subpackage under a handy name. I like
``T``.
......
theano/compile/debugmode.py
浏览文件 @ 09bce224
......@@ -195,8 +195,10 @@ def _optcheck_env(input_specs, output_specs, accept_inplace = False):
inputs, outputs = gof.graph.clone(orig_inputs, orig_outputs)
equivalence_tracker = _ResultEquivalenceTracker()
env = gof.env.Env(inputs, outputs,
features=[equivalence_tracker,
gof.DestroyHandler(do_imports_on_attach=False)])
#DestroyHandler is not needed because it is actually installed by an optimization
# after canonicalization. This results in a big speed gain.
#features=[equivalence_tracker, gof.DestroyHandler(do_imports_on_attach=False)])
features=[equivalence_tracker])
if not accept_inplace:
for node in env.nodes:
......
theano/compile/function_module.py
浏览文件 @ 09bce224
"""Driver of graph construction, optimization, and linking.
"""
__docformat__ = "restructuredtext en"
import copy_reg
import cPickle
......
theano/compile/module.py
浏览文件 @ 09bce224
"""Classes implementing Theano's Module system.
Rationale
=========
Functions in theano can share containers, when the `value` argument to `In` is a Container
instance. This feature makes it possible for multiple functions to use (and update) the same
inputs.
......@@ -17,9 +20,56 @@ have become `ModuleInstances`, Members have become `Container`s, and Methods hav
`Function`s.
This structure contains numbers and functions, and is ready for computation.
Design Documentation
====================
Module Graph
------------
Components form a tree structure. Each component may have a _parent_ to which it is _bound_.
When we call `make`, this tree structure is replicated with ComponentInstances instead of
Components. Wheras Components are primarily symbolic, ComponentInstances are sparse matrices,
ndarrays, callable functions, etc.
Compilation via make
--------------------
Conversion from a Component graph to a ComponentInstance graph is performed by `Component.make`.
This method traverses the Component graph in two passes.
In the first pass (the allocate pass), it creates storage for all Results that are contained in the graph (see
`Component.allocate`). These are the module variables.
In the second pass (the build pass), it creates functions that (in general) operate on these module variables.
This pass also serves to construct all ComponentInstance-derived instances as well, such as
`ModuleInstance`s. The objects that are returned from this second pass are the return value of
`Component.make`.
In the third pass (the initialize pass), is optional and not necessarily recursive through the
graph.
The purpose of the third pass is to call the initialize method of the ComponentInstances built
during the second pass.
During this pass the ComponentInstance graph is complete. It is a good time to fill storage
allocated in phase 1 with sensible values.
Class Structure
---------------
The most important classes for the user API here are `Module`, `ModuleInstance`, and `Method`.
Several other classes are defined to factorize functionality.
- `Component`: WRITEME: what properties make something a Component?
- `_RComponent`: WRITEME: what properties make something a Component?
- `External`: WRITEME: what properties hold? What
- `Member`: WRITEME: what properties hold? What do they do?
"""
__doc__='restructuredtext en'
__docformat__ = "restructuredtext en"
from theano import gof
from theano.printing import pprint
......@@ -27,19 +77,19 @@ from collections import defaultdict
from itertools import chain
from functools import partial
from copy import copy
import io
import io, sys
import function_module as F
from mode import default_mode
def join(*args):
def name_join(*args):
"""
Creates a string representation for the given names:
join('a', 'b', 'c') => 'a.b.c'
"""
return ".".join(arg for arg in args if arg)
def split(sym, n=-1):
def name_split(sym, n=-1):
"""
Gets the names from their joined representation
split('a.b.c') => ['a', 'b', 'c']
......@@ -55,7 +105,7 @@ def canonicalize(name):
[Fred: why we return the right type? Why int only?]
"""
if isinstance(name, str):
name = split(name)
name = name_split(name)
def convert(x):
try:
return int(x)
......@@ -63,7 +113,6 @@ def canonicalize(name):
return x
return map(convert, name)
class AllocationError(Exception):
"""
Exception raised when a Result has no associated storage.
......@@ -116,7 +165,7 @@ class Component(object):
else:
raise BindError("%s is already bound to %s as %s" % (self, self.parent, self.name))
self.parent = parent
self.name = join(parent.name, name)
self.name = name_join(parent.name, name)
return self
def bound(self):
......@@ -292,8 +341,12 @@ class Member(_RComponent):
r = self.r
if memo and r in memo:
return memo[r]
rval = gof.Container(r, storage = [getattr(r, 'data', None)])
memo[r] = io.In(result = r, value = rval, mutable = False)
assert isinstance(r, gof.Result)
rval = gof.Container(r, storage = [getattr(r, 'data', None)],
readonly=isinstance(r, gof.Constant))
memo[r] = io.In(result=r,
value=rval,
mutable=False)
return memo[r]
def build(self, mode, memo):
......@@ -302,41 +355,95 @@ class Member(_RComponent):
"""
return memo[self.r].value
class Method(Component):
"""
Method is a declaration of a function. It contains inputs,
outputs and updates. If the Method is part of a Composite
which holds references to Members, the Method may use them
without declaring them in the inputs, outputs or updates list.
inputs, outputs or updates may be strings. In that case, they
will be resolved in the Composite which is the parent of this
Method.
Method builds a Function (same structure as a call to
theano.function)
"""
class Method(Component):
inputs = []
"""function inputs (see `compile.function`)
def __init__(self, inputs, outputs, updates = {}, kits = [], **kwupdates):
"""
Method is a declaration of a function. It contains inputs,
outputs and updates. If the Method is part of a Composite
which holds references to Members, the Method may use them
without declaring them in the inputs, outputs or updates list.
If Module members are named explicitly in this list, then they will not use shared storage.
Storage must be provided either via an `io.In` value argument, or at the point of the
function call.
"""
[TODO: remove references to kits, for they are not really
needed anymore]
outputs=None
"""function outputs (see `compile.function`)"""
inputs, outputs or updates may be strings. In that case, they
will be resolved in the Composite which is the parent of this
Method.
updates = {}
"""update expressions for module members
If this method should update the shared storage value for a Module member, then the
update expression must be given in this dictionary.
Keys in this dictionary must be members of the module graph--results for which this Method
will use the shared storage.
The value associated with each key should be a Result (or a string that can be resolved to
a Result) representing the computation of a new value for this shared storage after
each function call.
"""
mode=None
"""This will override the Module compilation mode for this Method"""
def __init__(self, inputs, outputs, updates = {}, mode=None, **kwupdates):
"""Initialize attributes
:param inputs: value for `Method.inputs`
:param outputs: value for `Method.outputs`
:param updates: value for `Method.updates`
:param kwupdates: additions to `updates`
:param mode: value for `Method.mode`
:type inputs: list of (str or `Result` or `io.In`)
:type outputs: None or str or `Result` or `io.Out` or list of (str or `Result` or
`io.Out`)
:type updates: dict of `Result` or str -> `Result` or str
:type kwupdates: extra updates
:type mode: None or any mode accepted by `compile.function`
Method builds a Function (same structure as a call to
theano.function)
"""
super(Method, self).__init__()
self.inputs = inputs
self.outputs = outputs
self.updates = dict(updates, **kwupdates)
self.kits = list(kits)
self.mode = mode
def bind(self, parent, name, dup_ok=True):
"""Implement`Component.bind`"""
rval = super(Method, self).bind(parent, name, dup_ok=dup_ok)
rval.resolve_all()
return rval
def resolve(self, name):
"""
Resolves the name of an input or output in the parent.
"""Return the Result corresponding to a given name
:param name: the name of a Result in the Module to which this Method is bound
:type name: str
:rtype: `Result`
"""
if not self.bound():
raise ValueError('Trying to resolve a name on an unbound Method.')
......@@ -345,34 +452,47 @@ class Method(Component):
raise TypeError('Expected a Component with subtype Member or External.')
return result
def resolve_result(self, x):
if isinstance(x, gof.Result):
return x
elif isinstance(x, _RComponent):
return x.r
else:
return self.resolve(x).r
def resolve_all(self):
"""Convert all inputs, outputs, and updates specified as strings to Results.
This works by searching the attribute list of the Module to which this Method is bound.
"""
Resolves all inputs, outputs and updates that were given as
strings so that the fields contain the corresponding Result
instances instead.
"""
if isinstance(self.inputs, (gof.Result, str)):
inputs = [self.inputs]
else:
inputs = list(self.inputs)
self.inputs = [self.resolve_result(input) for input in inputs]
if isinstance(self.outputs, (list, tuple, ComponentList)):
self.outputs = [self.resolve_result(output) for output in self.outputs]
else:
self.outputs = self.resolve_result(self.outputs)
updates = self.updates
self.updates = {}
for k, v in updates.iteritems():
k, v = self.resolve_result(k), self.resolve_result(v)
self.updates[k] = v
def resolve_result(x, passthrough=(gof.Result)):
if isinstance(x, passthrough):
return x
elif isinstance(x, _RComponent):
return x.r
else:
return self.resolve(x).r
def resolve_inputs():
if isinstance(self.inputs, (io.In, gof.Result, str)):
inputs = [self.inputs]
else:
inputs = list(self.inputs)
self.inputs = [resolve_result(input,
passthrough=(gof.Result, io.In)) for input in inputs]
def resolve_outputs():
if isinstance(self.outputs, (io.Out, gof.Result, str, type(None))):
output = self.outputs
self.outputs = resolve_result(output,
passthrough=(gof.Result, io.Out, type(None)))
else:
outputs = list(self.outputs)
self.outputs = [resolve_result(output,
passthrough=(gof.Result, io.Out)) for output in outputs]
def resolve_updates():
updates = self.updates
self.updates = {}
for k, v in updates.iteritems():
k, v = resolve_result(k), resolve_result(v)
self.updates[k] = v
resolve_inputs()
resolve_outputs()
resolve_updates()
def allocate(self, memo):
"""
......@@ -381,13 +501,21 @@ class Method(Component):
return None
def build(self, mode, memo, allocate_all = False):
"""
Produces a function. If allocate_all is True, storage will be
allocated for all needed Results, even if there is no
"""Compile a function for this Method.
:param allocate_all: if True, storage will be
allocated for all needed Results even if there is no
associated storage for them in the memo. If allocate_all is
False, storage will only be allocated for Results that are
reachable from the inputs list.
:returns: a function that implements this method
:rtype: `Function` instance
"""
if self in memo:
return memo[self]
self.resolve_all() # resolve all so we don't have to mess with strings
def get_storage(r, require = False):
# If require is True, we can only get storage from the memo.
......@@ -399,37 +527,86 @@ class Method(Component):
' Verify that it is indeed a Member of the'
' enclosing module or of one of its submodules.' % (r, self.name, self))
else:
return io.In(result = r, value = gof.Container(r, storage = [None]), mutable = False)
# Wrap the inputs in In instances. TODO: allow the inputs to _be_ In instances
return io.In(result=r,
value=gof.Container(r,
storage=[getattr(r, 'data', None)],
readonly=(isinstance(r, gof.Constant))),
mutable=False)
inputs = self.inputs
inputs = [io.In(result = input,
value = get_storage(input).value,
mutable = False)
for input in inputs]
# Add the members to update to the inputs. TODO: see above
inputs += [io.In(result = k,
update = v,
value = get_storage(k, not allocate_all).value,
mutable = True,
strict = True)
for k, v in self.updates.iteritems()]
# Deal with explicit inputs
inputs = []
for input in self.inputs:
if type(input) is io.In:
inputs.append(input)
elif isinstance(input, gof.Result):
input_in = io.In(
result=input,
mutable=False)
inputs.append(input_in)
else:
raise TypeError(input, type(input))
# Deal with updates to shared storage
for k, v in self.updates.iteritems():
assert isinstance(k, gof.Result)
if isinstance(k, gof.Constant):
raise TypeError('Module Constants cannot be updated', k)
assert isinstance(v, gof.Result)
#identify an input for result k
input_k = None
for input in inputs:
if input.result == k:
input_k = input
#print 'METHOD UPDATE', k, v, input_k
if input_k is None:
# this is an implicit input,
# use shared storage
input_k = io.In(
result=k,
update=v,
value=get_storage(k, not allocate_all).value,
mutable=True)
inputs.append(input_k)
else:
raise ValueError(('Result listed in both inputs and updates.'
' Use inputs to use your own storage, use updates to '
'work on module-shared storage'), k)
# Deal with module inputs that are not updated
outputs = self.outputs
_inputs = [x.result for x in inputs]
# Grab the results that are not accessible from either the inputs or the updates.
for input in gof.graph.inputs((list(outputs) if isinstance(outputs, (list, tuple)) else [outputs])
outputs_list = list(outputs) if isinstance(outputs, (list, tuple)) else [outputs]
outputs_result_list = [o.result if isinstance(o, io.Out) else o for o in outputs_list]
for input in gof.graph.inputs(outputs_result_list
+ [x.update for x in inputs if getattr(x, 'update', False)],
blockers = _inputs):
if input not in _inputs:
# Add this input to the inputs; we require that storage already exists for them,
# but otherwise they are immutable.
if isinstance(input, gof.Value): # and not isinstance(input, gof.Constant):
#input might be Value or Constant
storage = get_storage(input)
storage.value = input.data
assert type(storage) is io.In
container = storage.value
#the user is allowed to change this value between function calls if it isn't a constant
assert container.readonly == (isinstance(input, gof.Constant))
#the function is not allowed to change this value
assert storage.mutable == False
else:
storage = get_storage(input, not allocate_all)
assert type(storage) is io.In
inputs.append(storage)
return F.function(inputs, outputs, mode)
effective_mode = mode if self.mode is None else self.mode
rval = F.function(inputs, outputs, effective_mode)
memo[self] = rval
return rval
def pretty(self, **kwargs):
self.resolve_all()
......@@ -458,17 +635,15 @@ class Method(Component):
def dup(self):
self.resolve_all()
return self.__class__(list(self.inputs),
list(self.outputs) if isinstance(self.outputs, list) else self.outputs,
dict(self.updates),
list(self.kits))
return self.__class__(inputs=list(self.inputs),
outputs=list(self.outputs) if isinstance(self.outputs, list) else self.outputs,
updates=dict(self.updates),
mode=self.mode)
def __call__(self, *args, **kwargs):
raise TypeError("'Method' object is not callable"
" (Hint: compile your module first. See Component.make())")
class CompositeInstance(object):
"""
Generic type which various Composite subclasses are intended to
......@@ -579,6 +754,7 @@ class Composite(Component):
def __getitem__(self, item):
# Uses get() internally
print 'COMPOSITE GETITEM', item
x = self.get(item)
if isinstance(x, (External, Member)):
return x.r
......@@ -617,6 +793,8 @@ class ComponentList(Composite):
_components = _components[0]
self._components = []
for c in _components:
if not isinstance(c, Component):
raise TypeError(c, type(c))
self.append(c)
def resolve(self, name):
......@@ -713,18 +891,15 @@ def default_initialize(self, init = {}, **kwinit):
for k, initv in dict(init, **kwinit).iteritems():
self[k] = initv
class ComponentDictInstance(CompositeInstance):
"""
ComponentDictInstance is meant to be instantiated by ComponentDict.
"""
class ComponentDictInstanceNoInit(CompositeInstance):
"""Component Instance that allows new items to be added"""
def __setitem__(self, item, value):
if item not in self.__items__:
# Set it if it's not there
# TODO: is this needed here? move to ModuleInstance?
self.__items__[item] = value
return
super(ComponentDictInstance, self).__setitem__(item, value)
else:
super(ComponentDictInstanceNoInit, self).__setitem__(item, value)
def __str__(self):
strings = []
......@@ -737,14 +912,30 @@ class ComponentDictInstance(CompositeInstance):
return '{%s}' % '\n'.join(strings).replace('\n', '\n ')
class ComponentDictInstance(ComponentDictInstanceNoInit):
"""
ComponentDictInstance is meant to be instantiated by ComponentDict.
"""
def initialize(self, init={}, **kwinit):
for k, initv in dict(init, **kwinit).iteritems():
self[k] = initv
class ComponentDict(Composite):
InstanceType = ComponentDictInstance # Type used by build() to make the instance
def __init__(self, components = {}, **kwcomponents):
super(ComponentDict, self).__init__()
components = dict(components, **kwcomponents)
for val in components.itervalues():
if not isinstance(val, Component):
raise TypeError(val, type(val))
self.__dict__['_components'] = components
def resolve(self, name):
name = canonicalize(name)
item = self.get(name[0])
......@@ -804,22 +995,35 @@ __autowrappers = []
def register_wrapper(condition, wrapper):
__autowrappers.append((condition, wrapper))
def wrapper(x):
"""Returns a wrapper function appropriate for `x`
Returns None if not appropriate wrapper is found
"""
for condition, wrap_fn in __autowrappers:
if condition(x):
return wrap_fn
return None
def wrap(x):
"""
Wraps x in a Component. Wrappers can be registered using
register_wrapper to allow wrapping more types.
"""
if isinstance(x, Component):
w = wrapper(x)
if w is not None:
return w(x)
else:
return x
for condition, wrapper in __autowrappers:
if condition(x):
return wrapper(x)
return x
def dict_wrap(d):
d_copy = {}
for k,v in d.iteritems():
d[k]=wrap(v)
return d
d_copy[k]=wrap(v)
return d_copy
# Component -> itself
register_wrapper(lambda x: isinstance(x, Component),
lambda x: x)
# Result -> Member
register_wrapper(lambda x: isinstance(x, gof.Result) and not x.owner,
......@@ -831,13 +1035,12 @@ register_wrapper(lambda x: isinstance(x, gof.Result) and x.owner,
# [[Result1], {Result2}, Result3...] -> ComponentList(Member(Result1), Member(Result2), ...)
register_wrapper(lambda x: isinstance(x, (list, tuple)) \
and all(isinstance(r, (gof.Result,Component,list,
tuple, dict)) for r in x),
and all(wrapper(r) is not None for r in x),
lambda x: ComponentList(*map(wrap, x)))
#{ "name1":{Component,Result,list,tuple,dict},...} -> ComponentDict({Component,Result,list,tuple,dict},...)
register_wrapper(lambda x: isinstance(x, dict) \
and all(isinstance(r,(Component,gof.Result,list,tuple,dict)) for r in x.itervalues()),
and all(wrapper(r) is not None for r in x.itervalues()),
lambda x: ComponentDict(dict_wrap(x)))
class Curry:
......@@ -855,7 +1058,7 @@ class Curry:
self.meth = getattr(self.obj, self.name)
class ModuleInstance(ComponentDictInstance):
class ModuleInstance(ComponentDictInstanceNoInit):
"""
WRITEME
......@@ -913,42 +1116,62 @@ class Module(ComponentDict):
self.__set_name__(value)
return
def remove_member(v):
def unpack_member_and_external(v):
if isinstance(v, (Member, External)):
print >> sys.stderr, ("WARNING: assignment of Member or External "
"objects (either directly or indirectly) to Module "
"is deprecated. Just use Result.")
return v.r
elif isinstance(v, (gof.Result,Method,Module)):
return v
elif isinstance(v,(int,bool)):
return v
elif isinstance(v, (list)):
return map(remove_member,v)
return map(unpack_member_and_external,v)
elif isinstance(v, (tuple)):
return tuple(map(remove_member,v))
return tuple(map(unpack_member_and_external,v))
elif isinstance(v,dict):
v_copy = dict()
for k,vv in v.iteritems():
v[k]=remove_member(vv)
v_copy[k]=unpack_member_and_external(vv)
return v
else:
# raise NotImplementedError
# print "WARNING: unknow:",v
return v
value=remove_member(value)
value=unpack_member_and_external(value)
if not hasattr(self,"local_attr"):
self.__dict__["local_attr"]={}
self.__dict__["local_attr_order"]=[]
self.__dict__["local_attr"][attr]=value
self.__dict__["local_attr_order"].append((attr, value))
def build(self, mode, memo):
for k,v in self.local_attr.iteritems():
for k,v in list(self.local_attr_order): #.iteritems():
self.__setattr__(k,v)
inst = super(Module, self).build(mode, memo)
for method in dir(self):
if not isinstance(inst, ModuleInstance):
raise TypeError('The InstanceType of a Module should inherit from ModuleInstance',
(self, type(inst)))
for methodname in dir(self):
# Any method with a name like '_instance_XXX' is added to
# the object built under the name obj.XXX
if method.startswith('_instance_'):
setattr(inst, method[10:], Curry(self, method, inst))
if methodname.startswith('_instance_'):
new_methodname = methodname[len('_instance_'):]
if hasattr(inst, new_methodname):
print >> sys.stderr, "WARNING: not overriding already-defined method",
print >> sys.stderr, getattr(inst, new_methodname),
print >> sys.stderr, "with",
print >> sys.stderr, getattr(self, methodname)
else:
curried = Curry(self, methodname, inst)
# setattr doesn't work here because we overrode __setattr__
# setattr(inst, new_methodname, curried)
inst.__dict__[new_methodname] = curried
assert getattr(inst, new_methodname) == curried
#print 'ADDING METHOD', method, 'to', id(inst), new_methodname, getattr(inst, new_methodname)
return inst
def _instance_initialize(self, inst, init = {}, **kwinit):
......@@ -959,12 +1182,21 @@ class Module(ComponentDict):
inst[name] = value
def make_mi(self, *args, **kwargs):
mods=[]
meth=[]#we put the method after the member to be sure of the ordering.
for k,v in self.local_attr.iteritems():
if isinstance(v,Module):
v=v.make_mi(args,kwargs)
if isinstance(v,Method):
mods.append((k, v))
elif isinstance(v,Method):
meth.append((k,v))
elif isinstance(v, list) and isinstance(v[0],Module):
temp = []
for m in v:
m=m.make_mi(args,kwargs)
m = self.__wrapper__(m)
temp.append(m)
self[k] = self.__wrapper__(temp)
else:
v = self.__wrapper__(v)
try:
......@@ -976,6 +1208,11 @@ class Module(ComponentDict):
self.__dict__[k] = v
# self.__setitem__(k,v)
for k,v in mods:
v=v.make_mi(args,kwargs)
v = self.__wrapper__(v)
self[k] = v
for k,v in meth:
self.__setitem__(k,v)
......
theano/compile/tests/test_inplace_opt_for_value.py
浏览文件 @ 09bce224
#!/usr/bin/env python
import numpy as N
from theano import Op, Apply, tensor as T, Module, Member, Method, Mode, compile
from theano import Op, Apply, tensor as T, Module, Method, Mode, compile
from theano.gof import OpSub, TopoOptimizer
from pylearn.algorithms.minimizer import make_minimizer # minimizer
from theano.printing import Print
from theano.tests import unittest_tools
#import sgd #until Olivier's module-import thing works better
####################
# Library-type stuff
......@@ -15,8 +13,6 @@ from theano.tests import unittest_tools
from theano.compile import module
from theano import tensor as T
from pylearn.algorithms.minimizer import minimizer_factory
class StochasticGradientDescent(module.FancyModule):
"""Fixed stepsize gradient descent"""
def __init__(self, args, cost, params, gradients=None, stepsize=None, WEIRD_STUFF=True):
......@@ -29,18 +25,18 @@ class StochasticGradientDescent(module.FancyModule):
self.stepsize_init = None
if stepsize is None:
self.stepsize = module.Member(T.dscalar())
self.stepsize = (T.dscalar())
elif isinstance(stepsize, T.TensorResult):
self.stepsize = stepsize
else:
if self.WEIRD_STUFF:
#TODO: why is this necessary? why does the else clause not work?
# self.stepsize = module.Member(T.dscalar(), init = stepsize)
self.stepsize = module.Member(T.dscalar())
self.stepsize = (T.dscalar())
self.stepsize_init = stepsize
else:
# self.stepsize = module.Member(T.value(stepsize))
self.stepsize = module.Member(T.constant(stepsize))#work!
self.stepsize = (T.constant(stepsize))#work!
if self.stepsize.ndim != 0:
raise ValueError('stepsize must be a scalar', stepsize)
......@@ -63,7 +59,6 @@ class StochasticGradientDescent(module.FancyModule):
pass
@minimizer_factory('sgd')
def sgd_minimizer(stepsize=None, **args):
def m(i,c,p,g=None):
return StochasticGradientDescent(i, c, p, stepsize=stepsize, **args)
......@@ -101,6 +96,9 @@ class TanhRnn(Op):
return Apply(self, [x, z0, A], [z])
def perform(self, node, (x,z0,A), out):
assert x is not None
assert z0 is not None
assert A is not None
T,M = x.shape
z = N.zeros((T+1, M))
z[0] = z0
......@@ -161,10 +159,10 @@ class ExampleRNN(Module):
self.n_vis = n_vis
#recurrent weight matrix in latent space
self.z0 = Member(T.dvector())
self.w = Member(T.dmatrix())
self.z0 = (T.dvector())
self.w = (T.dmatrix())
self.params = [self.w]
self.params = [self.z0, self.w]
#input and target
x, y = T.dmatrix(), T.dmatrix()
......@@ -176,6 +174,7 @@ class ExampleRNN(Module):
self.minimizer = minimizer([x, y], self.cost, self.params)
def _instance_initialize(self, obj):
print 'INITIALIZE EXAMPLE RNN'
n_vis = self.n_vis
rng = N.random.RandomState(unittest_tools.fetch_seed(2342))
......@@ -185,14 +184,14 @@ class ExampleRNN(Module):
obj.minimizer.initialize()
def test_example_rnn():
minimizer_fn = make_minimizer('sgd', stepsize = 0.001)
minimizer_fn = sgd_minimizer(stepsize = 0.001)
n_vis = 5
n_out = 3
n_hid = 4
rnn_module = ExampleRNN(n_vis, minimizer_fn)
rnn = rnn_module.make(mode='FAST_RUN')
rnn = rnn_module.make()
rng = N.random.RandomState(unittest_tools.fetch_seed(7722342))
x = rng.randn(10,n_vis)
......@@ -212,6 +211,7 @@ def test_example_rnn():
print i, rnn.minimizer.step_cost(x, y), rnn.minimizer.stepsize
else:
rnn.minimizer.step_cost(x, y)
assert rnn.minimizer.step_cost(x,y) < -20 #it starts around -.28
def test_WEIRD_STUFF():
n_vis = 3
......@@ -224,8 +224,8 @@ def test_WEIRD_STUFF():
LAG = 4
y[LAG:] = x[:-LAG, 0:n_vis]
minimizer_fn1 = make_minimizer('sgd', stepsize = 0.001, WEIRD_STUFF = False)
minimizer_fn2 = make_minimizer('sgd', stepsize = 0.001, WEIRD_STUFF = True)
minimizer_fn1 = sgd_minimizer(stepsize = 0.001, WEIRD_STUFF = False)
minimizer_fn2 = sgd_minimizer(stepsize = 0.001, WEIRD_STUFF = True)
rnn_module1 = ExampleRNN(n_vis, minimizer_fn1)
rnn_module2 = ExampleRNN(n_vis, minimizer_fn2)
rnn1 = rnn_module1.make(mode='FAST_RUN')
......
theano/compile/tests/test_module.py
浏览文件 @ 09bce224
#!/usr/bin/env python
"""Test compile.module"""
__docformat__ = "restructuredtext en"
import cPickle, numpy, unittest
from theano.compile.module import *
import theano.tensor as T
import sys
import theano
#TODO: add test for module.make(member=init_value)
class T_test_module(unittest.TestCase):
class T_module(unittest.TestCase):
def test_whats_up_with_submembers(self):
class Blah(FancyModule):
class Blah(Module):
def __init__(self, stepsize):
super(Blah, self).__init__()
self.stepsize = Member(T.value(stepsize))
self.stepsize = T.value(stepsize)
x = T.dscalar()
self.step = Method([x], x - self.stepsize)
B = Blah(0.0)
b = B.make(mode='FAST_RUN')
assert b.stepsize == 0.0
b.step(1.0)
assert b.stepsize == 0.0
......@@ -30,6 +36,7 @@ class T_test_module(unittest.TestCase):
m1=Module()
m1.x=x()
m1.y=y()
m1.emtpylist = []
m1.lx=[x()]#cast Result]
m1.ly=[y()]
m1.llx=[[x()]]#cast Result]
......@@ -57,8 +64,23 @@ class T_test_module(unittest.TestCase):
assert isinstance(m1.x,(gof.Result))
assert isinstance(m1.y,(gof.Result))
for i in [m1.lx[0], m1.ly[0], m1.llx[0][0], m1.lly[0][0], m1.ltx[0][0], m1.lty[0][0], m1.ldx[0]['x'], m1.ldy[0]['y'], m1.tx[0], m1.ty[0], m1.tlx[0][0], m1.tly[0][0], m1.ttx[0][0], m1.tty[0][0], m1.tdx[0]['x'], m1.tdy[0]['y'], m1.dx['x'], m1.dy['y'], m1.dlx['x'][0], m1.dly['y'][0], m1.dtx['x'][0], m1.dty['y'][0], m1.ddx['x']['x'], m1.ddy['y']['y']]:
assert isinstance(i,(gof.Result))
for i, obj in enumerate([
m1.lx[0], #0
m1.llx[0][0],
m1.ltx[0][0],
m1.ldx[0]['x'],
m1.lty[0][0],#5
m1.ldy[0]['y'],
m1.ly[0],
m1.lly[0][0],
m1.tx[0], #8
m1.ty[0], m1.tlx[0][0],
m1.tly[0][0], m1.ttx[0][0], m1.tty[0][0], m1.tdx[0]['x'],
m1.tdy[0]['y'], m1.dx['x'],
m1.dy['y'], m1.dlx['x'][0], m1.dly['y'][0],
m1.dtx['x'][0], m1.dty['y'][0], m1.ddx['x']['x'],
m1.ddy['y']['y']]):
assert isinstance(obj,(gof.Result))
inst=m1.make()
......@@ -77,44 +99,94 @@ class T_test_module(unittest.TestCase):
assert i
#test that we can set a value to the data the get this value
inst.x=-1
inst.y=-2
inst.ldx[0]['x']=-3
inst.ldy[0]['y']=-4
inst.tdx[0]['x']=-5
inst.tdy[0]['y']=-6
inst.ddx['x']['x']=-7
inst.ddy['y']['y']=-8
for i,j in zip(get_l2(),range(len(get_l2()))):
i[0]=j
assert inst.x==-1
assert inst.y==-2
assert inst.ldx[0]['x']==-3
assert inst.ldy[0]['y']==-4
assert inst.tdx[0]['x']==-5
assert inst.tdy[0]['y']==-6
assert inst.ddx['x']['x']==-7
assert inst.ddy['y']['y']==-8
for i,j in zip(get_l2(),range(len(get_l2()))):
assert i[0]==j
local_test(lambda:T.dscalar(),lambda:Member(T.dscalar()))
local_test(lambda:T.value(1),lambda:Member(T.value(2)))
local_test(lambda:T.constant(1),lambda:Member(T.constant(2)))
def test_compound_structure_assignment(self):
if not isinstance(m1.x, gof.Constant):
inst.x=-1
inst.y=-2
inst.ldx[0]['x']=-3
inst.ldy[0]['y']=-4
inst.tdx[0]['x']=-5
inst.tdy[0]['y']=-6
inst.ddx['x']['x']=-7
inst.ddy['y']['y']=-8
for i,j in zip(get_l2(),range(len(get_l2()))):
i[0]=j
assert inst.x==-1
assert inst.y==-2
assert inst.ldx[0]['x']==-3
assert inst.ldy[0]['y']==-4
assert inst.tdx[0]['x']==-5
assert inst.tdy[0]['y']==-6
assert inst.ddx['x']['x']==-7
assert inst.ddy['y']['y']==-8
for i,j in zip(get_l2(),range(len(get_l2()))):
assert i[0]==j
local_test(lambda:T.dscalar(),lambda:T.dscalar())
local_test(lambda:T.value(1),lambda:T.value(2))
local_test(lambda:T.constant(1),lambda:T.constant(2))
def test_list_assign(self):
"""Test that list members can be assigned list-wise"""
def local_test(x,y):
m1=Module()
m1.l=[x(), y()]#cast Result]
#create a list with some results in it
m1.l=[x(), y()]
# create a Method that makes the second list element a shared Member
m1.f=Method([], m1.l[1])
m1.g=Method([], m1.l[0])
m = m1.make()
#assign 4 and 5 to the two results' containers in m
m.l = [4, 5]
print 'm.f', m.f()
assert numpy.all(5 == m.f())
assert numpy.all(4 == m.g())
local_test(lambda:T.dscalar(),lambda:T.dscalar())
local_test(lambda:T.value(1),lambda:T.value(2))
def test_tuple_assign(self):
"""Test that list members can be assigned tuple-wise"""
def local_test(x,y):
m1=Module()
m1.l=(x(), y())
# create a Method that makes the second list element a shared Member
m1.g=Method([], m1.l[0])
m1.f=Method([], m1.l[1])
m = m1.make()
#assign 4 and 5 to the two results' containers in m
m.l = (4, 5)
assert 5 == m.f()
assert 4 == m.g()
local_test(lambda:T.dscalar(),lambda:Member(T.dscalar()))
local_test(lambda:T.value(1),lambda:Member(T.value(2)))
local_test(lambda:T.constant(1),lambda:Member(T.constant(2)))
local_test(lambda:T.dscalar(),lambda:T.dscalar())
local_test(lambda:T.value(1),lambda:T.value(2))
def test_dict_assign(self):
"""Test that list members can be assigned dict-wise"""
def local_test(x,y):
m1=Module()
##DICT
m1.l={'x':x(), 'y':y()}
# create a Method that makes the second list element a shared Member
m1.f=Method([], m1.l['y'])
m1.g=Method([], m1.l['x'])
m = m1.make()
#assign 4 and 5 to the two results' containers in m
m.l = dict(x=4, y=5)
assert 5 == m.f()
assert 4 == m.g()
print 'dscalar test'
local_test(lambda:T.dscalar(),lambda:T.dscalar())
print 'value test'
local_test(lambda:T.value(1),lambda:T.value(2))
def test_method_in_list_or_dict(self):
......@@ -197,11 +269,12 @@ class T_test_module(unittest.TestCase):
def get_element(i):
return [i.x,i.lx[0],i.tx[0],i.dx['x'],i.llx[0][0], i.llx[1][0], i.ltx[0][0], i.ldx[0]['x'], i.tlx[0][0], i.tlx[0][0], i.tdx[0]['x'], i.dlx['x'][0], i.dtx['x'][0], i.ddx['x']['x']]
m1=Module()
m2=Module()
x=T.dscalar()
populate_module(m1,x)
populate_module(m2,Member(x))
populate_module(m2,x)
#m1.x and m2.x should not be shared as their is no hierarchi link between them.
inst1=m1.make()
inst2=m2.make()
......@@ -248,8 +321,8 @@ class T_test_module(unittest.TestCase):
m4=Module()
x=T.dscalar()
populate_module(m1,x)
populate_module(m2,Member(x))
populate_module(m4,Member(x))
populate_module(m2,(x))
populate_module(m4,(x))
#m1.x and m2.x should not be shared as their is no hierarchi link between them.
inst1=m1.make()
inst2=m2.make()
......@@ -323,49 +396,90 @@ class T_test_module(unittest.TestCase):
assert isinstance(inst.dy['y'],theano.compile.function_module.Function)
assert isinstance(inst.tty[0][0],theano.compile.function_module.Function)
print >> sys.stderr, "MODULE TEST IMPLEMENTED BUT WE DON'T KNOW WHAT WE WANT AS A RESULT"
def test_shared_method_N(self):
"""Test that Methods can be shared an arbitrary number of times between many submodules and
internal data structures."""
#put them in subModules, sub-sub-Modules, shared between a list and a dict, shared between
#a list and a submodule with a dictionary, etc...
print >> sys.stderr, "WARNING MODULE TEST NOT IMPLEMENTED"
assert m1.y is m1.ly[0]
assert inst.y is inst.ly[0]
assert inst.y is inst.lly[0][0]
assert inst.y is inst.ty[0]
assert inst.y is inst.tty[0][0]
assert inst.y is inst.dy['y']
def test_member_method_inputs(self):
"""Test that module Members can be named as Method inputs, in which case the function will
*not* use the storage allocated for the Module's version of that Member.
si le module a un membre x et qu''une fct un parametre appele x qui n''est pas le membre cela doit etre bien traiter.
les poids ne change pas
"""
print >> sys.stderr, "WARNING MODULE TEST NOT IMPLEMENTED"
"""
# test that explicit Method inputs don't use shared storage
M = Module()
M.x = T.dscalar()
M.y = T.dscalar()
M.f = Method([M.x], M.x + M.y)
M.g = Method([M.y], M.x - M.y)
m = M.make()
m.y = 77
assert m.f(23) == 100
assert m.x == None
m.x = 1000
assert m.g(23) == 977
assert m.y == 77
assert m.x == 1000
def test_member_input_flags(self):
"""Test that we can manipulate the mutable, strict, etc. flags (see SymbolicInput) of
Method inputs"""
print >> sys.stderr, "WARNING MODULE TEST NOT IMPLEMENTED"
M = Module()
M.x = T.dvector()
M.y = T.dvector()
xval= numpy.asarray([0, 0.5])
M.f = Method([io.In(M.x,
mutable=True,
update=(M.x - M.y),
value=xval)], M.x + M.y)
m = M.make()
m.y = numpy.asarray([1, 2])
assert numpy.all(m.f(xval) == [1, 2.5])
assert numpy.all(xval == [-1, -1.5])
def test_member_output_flags(self):
"""Test that we can manipulate the output flags (just 'borrow' I think, see SymbolicOutput)
of Method outputs"""
print >> sys.stderr, "WARNING MODULE TEST NOT IMPLEMENTED"
M = Module()
M.x = T.dvector()
M.f = Method([M.x], io.Out(M.x*4, borrow=True), mode='FAST_RUN')
m = M.make()
def test_sanity_check_mode(self):
"""Test that Module.make(self) can take the same list of Modes that function can, so we can
debug modules"""
print >> sys.stderr, "WARNING MODULE TEST NOT IMPLEMENTED"
v0 = m.f([5, 8])
v0_copy = v0 * 1
m.f([3, 2])
assert numpy.all(v0 != v0_copy)
def test_member_value(self):
"""Test that module Members of Value work correctly. As Result?"""
print >> sys.stderr, "WARNING MODULE TEST NOT IMPLEMENTED"
M = Module()
x = T.dscalar()
M.y = T.value(40)
M.f = Method([x], x + 2 * M.y)
m = M.make()
m.y = 80
assert m.f(20) == 180
def test_member_constant(self):
"""Test that module Members of Constant work correctly.
As Result with more optimization?"""
print >> sys.stderr, "WARNING MODULE TEST NOT IMPLEMENTED"
M = Module()
x = T.dscalar()
M.y = T.constant(40)
M.f = Method([x], x + 2 * M.y)
m = M.make()
try:
m.y = 77 #fail?
except:
pass
assert m.y == 40
assert m.f(20) == 100
def test_raise_NotImplemented(self):
c=Component()
......@@ -380,24 +494,78 @@ class T_test_module(unittest.TestCase):
self.assertRaises(NotImplementedError, c.get,"n")
self.assertRaises(NotImplementedError, c.set,"n",1)
def test_tuple_members(self):
def test_tuple_members():
M = Module()
M.a = (1,1)
assert isinstance(M.a, tuple)
class Temp(Module):
def __init__(self):
self.a = (1,1)
M = Temp()
assert isinstance(M.a, tuple)
M = Module()
M.a = (1,1)
assert isinstance(M.a, tuple)
class Temp(Module):
def __init__(self):
self.a = (1,1)
M = Temp()
assert isinstance(M.a, tuple)
def test_method_updates():
# updates work
M = Module()
M.x = T.dvector()
x = T.dvector()
xval= numpy.asarray([0, 0.5])
M.f = Method([x], M.x*4, updates={M.x:M.x * 2}, mode='FAST_COMPILE')
m = M.make(mode='FAST_RUN')
m.x = xval
m.f([9,9])
assert numpy.all(m.x == [0, 1])
assert numpy.all(xval == [0, 0.5])
# In(update) works
M = Module()
M.x = T.dvector()
x = T.dvector()
M.f = Method([x, io.In(M.x, value=xval, update=M.x*2)], M.x*4)
m = M.make()
m.f([9,9])
assert m.x is None
assert numpy.all(xval == [0, 1])
# when a result is listed explicitly and in an update, then there's a problem.
M = Module()
M.x = T.dvector()
x = T.dvector()
M.f = Method([x, io.In(M.x, value=xval, update=M.x*2)], M.x*4,
updates={M.x:M.x * 7})
try:
m = M.make()
assert False
except ValueError, e:
if str(e[0]).startswith('Result listed in both inputs and up'):
pass
else:
raise
def test_method_mode():
"""Test that Methods can override the module build mode"""
M = Module()
M.x = T.dvector()
M.f = Method([M.x], M.x*4, mode='FAST_COMPILE')
M.g = Method([M.x], M.x*4)
M.h = Method([M.x], M.x*4)
m = M.make(mode='FAST_RUN')
assert m.f.maker.mode != m.g.maker.mode
assert m.h.maker.mode == m.g.maker.mode
assert numpy.all(m.f([1,2]) == m.g([1,2]))
def test_pickle():
"""Test that a module can be pickled"""
M = Module()
M.x = Member(T.dmatrix())
M.y = Member(T.dmatrix())
M.x = (T.dmatrix())
M.y = (T.dmatrix())
a = T.dmatrix()
M.f = Method([a], a + M.x + M.y)
M.g = Method([a], a * M.x * M.y)
......@@ -418,38 +586,39 @@ def test_pickle():
assert m_dup.x is m_dup.g.input_storage[1].data
assert m_dup.y is m_dup.g.input_storage[2].data
from numpy.testing import *
@dec.knownfailureif(True, "These branch cuts are known to fail")
def test_pickle_aliased_memory():
M = Module()
M.x = Member(T.dmatrix())
M.y = Member(T.dmatrix())
a = T.dmatrix()
M.f = Method([a], a + M.x + M.y)
M.g = Method([a], a * M.x * M.y)
m = M.make(x=numpy.zeros((4,5)), y=numpy.ones((2,3)))
m.y = m.x[:]
m_dup = cPickle.loads(cPickle.dumps(m))
#m's memory is aliased....
m.x[0,0] = 3.14
assert m.y[0,0] == 3.14
#is m_dup's memory aliased?
m_dup.x[0,0] = 3.14
assert m_dup.y[0,0] == 3.14
#m's memory is aliased differently....
m.y = m.x[1:2]
m_dup = cPickle.loads(cPickle.dumps(m))
try:
M = Module()
M.x = (T.dmatrix())
M.y = (T.dmatrix())
a = T.dmatrix()
M.f = Method([a], a + M.x + M.y)
M.g = Method([a], a * M.x * M.y)
m = M.make(x=numpy.zeros((4,5)), y=numpy.ones((2,3)))
m.y = m.x[:]
m_dup = cPickle.loads(cPickle.dumps(m))
#m's memory is aliased....
m.x[0,0] = 3.14
assert m.y[0,0] == 3.14
#is m_dup's memory aliased?
m_dup.x[0,0] = 3.14
assert m_dup.y[0,0] == 3.14
#m's memory is aliased differently....
m.y = m.x[1:2]
m_dup = cPickle.loads(cPickle.dumps(m))
#is m_dup's memory aliased the same way?
m.x[1,0] = 3.142
assert m.y[0,0] == 3.142
m_dup.x[1,0] = 3.142
assert m_dup.y[0,0] == 3.142
except Exception, e:
raise Exception('Known Failure: These branch cuts are known to fail', str(e))
#is m_dup's memory aliased the same way?
m.x[1,0] = 3.142
assert m.y[0,0] == 3.142
m_dup.x[1,0] = 3.142
assert m_dup.y[0,0] == 3.142
if __name__ == '__main__':
......
theano/gof/graph.py
浏览文件 @ 09bce224
......@@ -84,6 +84,9 @@ class Apply(utils.object2):
else:
raise TypeError("The 'outputs' argument to Apply must contain Result instances with no owner, not %s" % output)
self._creation_idx = _creation_idx[0]
_creation_idx[0] += 1
def default_output(self):
"""Returns the default output for this node.
......@@ -123,9 +126,6 @@ class Apply(utils.object2):
return self
def __hash__(self):
if not hasattr(self, '_creation_idx'):
self._creation_idx = _creation_idx[0]
_creation_idx[0] += 1
return self._creation_idx
......
theano/tensor/blas.py
浏览文件 @ 09bce224
......@@ -473,15 +473,6 @@ class GemmLocalOptimizer(LocalOptimizer):
return [T.add(*new_add_inputs)]
return False
@staticmethod
def failure_callback(exc, nav, repl_pairs):
"""WRITEME"""
if not isinstance(exc, InconsistencyError):
traceback.print_exc()
else:
#print 'GEMM caused cycle, it happens.'
pass
@staticmethod
def _as_scalar(res):
"""Return None or a TensorResult whose type is in T.float_scalar_types"""
......@@ -579,11 +570,11 @@ class GemmLocalOptimizer(LocalOptimizer):
# TODO: This could be an equilibriumOptmizer, but I don't know how to combine an OpKeyOptimizer and
# an EquilibriumOptimizer.
compile.optdb.register('inplace_gemm_0', OpKeyOptimizer(GemmLocalOptimizer(),
failure_callback=GemmLocalOptimizer.failure_callback), 70.00, 'fast_run', 'inplace', 'gemm')
failure_callback=OpKeyOptimizer.warn_inplace), 70.00, 'fast_run', 'inplace', 'gemm')
compile.optdb.register('inplace_gemm_1', OpKeyOptimizer(GemmLocalOptimizer(),
failure_callback=GemmLocalOptimizer.failure_callback), 70.01, 'fast_run', 'inplace', 'gemm')
failure_callback=OpKeyOptimizer.warn_inplace), 70.01, 'fast_run', 'inplace', 'gemm')
compile.optdb.register('inplace_gemm_2', OpKeyOptimizer(GemmLocalOptimizer(),
failure_callback=GemmLocalOptimizer.failure_callback), 70.02, 'fast_run', 'inplace', 'gemm')
failure_callback=OpKeyOptimizer.warn_inplace), 70.02, 'fast_run', 'inplace', 'gemm')
class Dot22(GemmRelated):
"""Compute a matrix-matrix product.
......
theano/tensor/tests/test_basic.py
浏览文件 @ 09bce224
......@@ -1305,14 +1305,26 @@ class test_matinv(unittest.TestCase):
ssd, gw = fn(x,w)
#print ssd, x*w, x, w
if i == 0:
str0 = str(ssd)
ssd0 = ssd
w -= 0.4 * gw
return str0, str(ssd)
return ssd0, ssd
def test_reciprocal(self):
"""Matrix reciprocal by gradient descent"""
self.assertEqual(('6.10141615619', '0.00703816291711'), self.mat_reciprocal(3))
ssd0,ssd = self.mat_reciprocal(3)
numpy.random.seed(unittest_tools.fetch_seed(1))
# hand-coded numpy implementation for verification
x = numpy.random.rand(3,3)+0.1
w = numpy.random.rand(3,3)
myssd0 = numpy.sum((x*w - numpy.ones((3,3)))**2.0)
for i in xrange(300):
gw = 2*(x*w - numpy.ones((3,3)))*x # derivative of dMSE/dw
myssd = numpy.sum((x*w - numpy.ones((3,3)))**2)
w -= 0.4 * gw
self.failUnlessAlmostEqual(ssd0, myssd0)
self.failUnlessAlmostEqual(ssd, myssd)
class t_dot(unittest.TestCase):
def setUp(self):
......
theano/tensor/tests/test_naacl09.py
浏览文件 @ 09bce224
......@@ -17,6 +17,8 @@ def cross_entropy(target, output, axis=1):
@warning: OUTPUT and TARGET are reversed in nnet_ops.binary_crossentropy
"""
return -T.mean(target * T.log(output) + (1 - target) * T.log(1 - output), axis=axis)
def quadratic(target, output, axis=1):
return T.mean(T.sqr(target - output), axis=axis)
class QuadraticDenoisingAA(module.Module):
"""Quadratic de-noising Auto-encoder
......@@ -70,27 +72,36 @@ class QuadraticDenoisingAA(module.Module):
# ACQUIRE/MAKE INPUT
if not input:
input = T.matrix('input')
self.input = theano.External(input)
#self.input = theano.External(input)
self.input = (input)
# HYPER-PARAMETERS
self.lr = theano.Member(T.scalar())
#self.lr = theano.Member(T.scalar())
self.lr = (T.scalar())
# PARAMETERS
if _qfilters is None:
self.qfilters = [theano.Member(T.dmatrix('q%i'%i)) for i in xrange(n_quadratic_filters)]
#self.qfilters = [theano.Member(T.dmatrix('q%i'%i)) for i in xrange(n_quadratic_filters)]
self.qfilters = [(T.dmatrix('q%i'%i)) for i in xrange(n_quadratic_filters)]
else:
self.qfilters = [theano.Member(q) for q in _qfilters]
#self.qfilters = [theano.Member(q) for q in _qfilters]
self.qfilters = [(q) for q in _qfilters]
self.w1 = theano.Member(T.matrix('w1')) if _w1 is None else theano.Member(_w1)
#self.w1 = theano.Member(T.matrix('w1')) if _w1 is None else theano.Member(_w1)
self.w1 = (T.matrix('w1')) if _w1 is None else (_w1)
if _w2 is None:
if not tie_weights:
self.w2 = theano.Member(T.matrix())
#self.w2 = theano.Member(T.matrix())
self.w2 = (T.matrix())
else:
self.w2 = self.w1.T
else:
self.w2 = theano.Member(_w2)
self.b1 = theano.Member(T.vector('b1')) if _b1 is None else theano.Member(_b1)
self.b2 = theano.Member(T.vector('b2')) if _b2 is None else theano.Member(_b2)
#self.w2 = theano.Member(_w2)
self.w2 = (_w2)
#self.b1 = theano.Member(T.vector('b1')) if _b1 is None else theano.Member(_b1)
self.b1 = (T.vector('b1')) if _b1 is None else (_b1)
#self.b2 = theano.Member(T.vector('b2')) if _b2 is None else theano.Member(_b2)
self.b2 = (T.vector('b2')) if _b2 is None else (_b2)
# # REGULARIZATION COST
# self.regularization = self.build_regularization()
......@@ -168,6 +179,7 @@ class QuadraticDenoisingAA(module.Module):
#self.validate = theano.Method(self.input, [self.cost, self.output])
def _instance_initialize(self, obj, input_size, hidden_size, seed, lr, qfilter_relscale):
print 'QDAA init'
"""
qfilter_relscale is the initial range for any quadratic filters (relative to the linear
filter's initial range)
......@@ -212,7 +224,8 @@ class SigmoidXEQuadraticDenoisingAA(QuadraticDenoisingAA):
unittest_tools.seed_rng()
def build_corrupted_input(self):
self.noise_level = theano.Member(T.scalar())
#self.noise_level = theano.Member(T.scalar())
self.noise_level = (T.scalar())
return self.random.binomial(T.shape(self.input), 1, 1 - self.noise_level) * self.input
def hid_activation_function(self, activation):
......@@ -262,12 +275,17 @@ class Module_Nclass(module.FancyModule):
def __init__(self, x=None, targ=None, w=None, b=None, lr=None, regularize=False):
super(Module_Nclass, self).__init__() #boilerplate
self.x = module.Member(x) if x is not None else T.matrix('input')
self.targ = module.Member(targ) if targ is not None else T.lvector()
#self.x = module.Member(x) if x is not None else T.matrix('input')
self.x = (x) if x is not None else T.matrix('input')
#self.targ = module.Member(targ) if targ is not None else T.lvector()
self.targ = (targ) if targ is not None else T.lvector()
self.w = module.Member(w) if w is not None else module.Member(T.dmatrix())
self.b = module.Member(b) if b is not None else module.Member(T.dvector())
self.lr = module.Member(lr) if lr is not None else module.Member(T.dscalar())
#self.w = module.Member(w) if w is not None else module.Member(T.dmatrix())
self.w = (w) if w is not None else (T.dmatrix())
#self.b = module.Member(b) if b is not None else module.Member(T.dvector())
self.b = (b) if b is not None else (T.dvector())
#self.lr = module.Member(lr) if lr is not None else module.Member(T.dscalar())
self.lr = (lr) if lr is not None else (T.dscalar())
self.params = [p for p in [self.w, self.b] if p.owner is None]
......@@ -309,8 +327,6 @@ class Module_Nclass(module.FancyModule):
class ConvolutionalMLPInstance(module.FancyModuleInstance, Loss01):
#initialize is called by Module.make
def initialize(self, input_size, input_representation_size, hidden_representation_size, output_size, lr, seed, noise_level, qfilter_relscale):
# ASK JAMES: Is the following necessary?
# super(ConvolutionalMLPInstance, self)._instance_initialize(obj, **kwargs)
R = N.random.RandomState(unittest_tools.fetch_seed(seed))
......@@ -323,19 +339,29 @@ class ConvolutionalMLPInstance(module.FancyModuleInstance, Loss01):
# for layer in obj.layers:
# if layer.lr is None:
# layer.lr = lr
assert self.input_representations[-1] is not self.input_representations[0]
assert self.input_representations[-1].w1 is self.input_representations[0].w1
for i in self.input_representations:
# i.initialize(input_size=self.input_size, hidden_size=self.input_representation_size, seed=R.random_integers(2**30), noise_level=noise_level, qfilter_relscale=qfilter_relscale)
i.initialize(input_size=self.input_size, hidden_size=self.input_representation_size, noise_level=noise_level, seed=R.random_integers(2**30), lr=lr, qfilter_relscale=qfilter_relscale)
i.initialize(input_size=self.input_size,
hidden_size=self.input_representation_size, noise_level=noise_level,
seed=int(R.random_integers(2**30)), lr=lr, qfilter_relscale=qfilter_relscale)
print type(i.w1)
assert isinstance(i.w1, N.ndarray)
for i in self.input_representations[1:]:
print type(i.w1)
assert isinstance(i.w1, N.ndarray)
assert (i.w1 == self.input_representations[0].w1).all()
assert (i.w2 == self.input_representations[0].w2).all()
assert (i.b1 == self.input_representations[0].b1).all()
assert (i.b2 == self.input_representations[0].b2).all()
assert all((a==b).all() for a, b in zip(i.qfilters, self.input_representations[0].qfilters))
self.hidden.initialize(input_size=(len(self.inputs) * self.input_representation_size), hidden_size=self.hidden_representation_size, noise_level=noise_level, seed=R.random_integers(2**30), lr=lr, qfilter_relscale=qfilter_relscale)
self.hidden.initialize(input_size=(len(self.inputs) * self.input_representation_size),
hidden_size=self.hidden_representation_size, noise_level=noise_level,
seed=int(R.random_integers(2**30)), lr=lr, qfilter_relscale=qfilter_relscale)
self.output.initialize(n_in=self.hidden_representation_size, n_out=self.output_size, lr=lr, seed=R.random_integers(2**30))
......@@ -352,7 +378,8 @@ class ConvolutionalMLP(module.FancyModule):
):
super(ConvolutionalMLP, self).__init__()
self.lr = module.Member(T.scalar())
#self.lr = module.Member(T.scalar())
self.lr = (T.scalar())
self.inputs = [T.dmatrix() for i in range(window_size)]
self.targ = T.lvector()
......@@ -382,6 +409,7 @@ class ConvolutionalMLP(module.FancyModule):
_qfilters = self.input_representations[0].qfilters
)
)
assert self.input_representations[-1].w1 is self.input_representations[0].w1
self.input_representation = T.concatenate([i.hidden for i in self.input_representations], axis=1)
self.hidden = QDAA(
......@@ -445,13 +473,11 @@ def create(window_size=3,
""" Create a convolutional model. """
activation_function = T.tanh
import pylearn.algorithms.cost
architecture = ConvolutionalMLP( \
window_size = window_size,
n_quadratic_filters = n_quadratic_filters,
activation_function = activation_function,
reconstruction_cost_function = pylearn.algorithms.cost.quadratic,
reconstruction_cost_function = quadratic,
tie_weights = False
)
model = architecture.make(input_size=input_dimension, input_representation_size=token_representation_size, hidden_representation_size=concatenated_representation_size, output_size=output_vocabsize, lr=lr, seed=seed, noise_level=noise_level, qfilter_relscale=qfilter_relscale, mode=compile_mode)
......@@ -471,13 +497,11 @@ def create_realistic(window_size=3,#7,
""" Create a convolutional model. """
activation_function = T.tanh
import pylearn.algorithms.cost
architecture = ConvolutionalMLP( \
window_size = window_size,
n_quadratic_filters = n_quadratic_filters,
activation_function = activation_function,
reconstruction_cost_function = pylearn.algorithms.cost.quadratic,
reconstruction_cost_function = quadratic,
tie_weights = False
)
model = architecture.make(input_size=input_dimension, input_representation_size=token_representation_size, hidden_representation_size=concatenated_representation_size, output_size=output_vocabsize, lr=lr, seed=seed, noise_level=noise_level, qfilter_relscale=qfilter_relscale, mode=compile_mode)
......@@ -522,8 +546,8 @@ def test_naacl_model(iters_per_unsup=10, iters_per_sup=10,
s0, s1 = [str(j) for j in m.pretraining_update(*inputs)]
print 'huh?', i, iters_per_unsup, iters_per_unsup * (i+1), s0, s1
if iters_per_unsup == 10:
assert s0.startswith('0.40218760858')
assert s1.startswith('0.074450801777')
assert s0.startswith('0.403044')
assert s1.startswith('0.074898')
print 'FINETUNING GRAPH'
print 'SUPERVISED PHASE COSTS (%s)'%optimizer
......@@ -533,9 +557,9 @@ def test_naacl_model(iters_per_unsup=10, iters_per_sup=10,
s0 = str(m.finetuning_update(*(inputs + [targets])))
print iters_per_sup * (i+1), s0
if iters_per_sup == 10:
assert s0.startswith('15.65127763')#should check for the 8 decimal only.
assert s0.startswith('15.6511')#should check for the 8 decimal only.
if __name__ == '__main__':
def jtest_main():
from theano import gof
JTEST = theano.compile.mode.optdb.query(*sys.argv[2:])
print 'JTEST', JTEST
......@@ -543,3 +567,23 @@ if __name__ == '__main__':
optimizer = eval(sys.argv[1])
test_naacl_model(optimizer, 10, 10, realistic=False)
def real_main():
test_naacl_model()
def profile_main():
# This is the main function for profiling
# We've renamed our original main() above to real_main()
import cProfile, pstats, StringIO
prof = cProfile.Profile()
prof = prof.runctx("real_main()", globals(), locals())
stream = StringIO.StringIO()
stats = pstats.Stats(prof)
stats.sort_stats("time") # Or cumulative
stats.print_stats(80) # 80 = how many to print
# The rest is optional.
# stats.print_callees()
# stats.print_callers()
if __name__ == '__main__':
#real_main()
profile_main()
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论