Skip to content

P
pytensor
提交 6d39faaf authored 作者: Brandon T. Willard's avatar Brandon T. Willard 提交者： Brandon T. Willard
浏览文件
操作

Remove the scan_ prefix from modules and relocate them to theano.scan

The one drawback to this change is that `theano.scan`, i.e. the `scan` function exposed by `theano/__init__.py`, shadows the `theano.scan` sub-package. In other words, modules in the `theano.scan` sub-package cannot be accessed via the top-level `theano` module object (e.g. `import theano; theano.scan.op.Scan` won't work). This is a minor inconvenience, and, since internal library code is generally expected to import objects directly from the modules in which they're defined, the appearance of this problem will serve as a welcome warning.
上级 77b87e95
隐藏空白字符变更
内嵌 并排
正在显示 包含 939 行增加 和 1120 行删除
doc/extending/scan.txt
浏览文件 @ 6d39faaf
...@@ -46,24 +46,24 @@ Relevant code files ...@@ -46,24 +46,24 @@ Relevant code files
=================== ===================
The implementation of Scan is spread over several files in The implementation of Scan is spread over several files in
``theano/scan_module``. The different files, and sections of the code they ``theano/scan``. The different files, and sections of the code they
deal with, are : deal with, are :
* ``scan.py`` implements the ``scan`` function. The ``scan`` function * ``basic.py`` implements the ``scan`` function. The ``scan`` function
arranges the arguments of scan correctly, constructs the scan op and arranges the arguments of scan correctly, constructs the scan op and
afterwards calls the constructed scan op on the arguments. This function afterwards calls the constructed scan op on the arguments. This function
takes care of figuring out missing inputs and shared variables. takes care of figuring out missing inputs and shared variables.
* ``scan_op.py`` implements the ``Scan`` op class. The ``Scan`` respects * ``op.py`` implements the ``Scan`` op class. The ``Scan`` respects
the ``Op`` interface, and contains most of the logic of the scan operator. the ``Op`` interface, and contains most of the logic of the scan operator.
* ``scan_utils.py`` contains several helpful functions used throughout out the * ``utils.py`` contains several helpful functions used throughout out the
other files that are specific of the scan operator. other files that are specific of the scan operator.
* ``scan_views.py`` contains different views of the scan op that have * ``views.py`` contains different views of the scan op that have
simpler and easier signatures to be used in specific cases. simpler and easier signatures to be used in specific cases.
* ``scan_opt.py`` contains the list of all Theano graph optimizations for the * ``opt.py`` contains the list of all Theano graph optimizations for the
scan operator. scan operator.
...@@ -269,14 +269,14 @@ Because of the complexity involved in dealing with Scan, a large number of ...@@ -269,14 +269,14 @@ Because of the complexity involved in dealing with Scan, a large number of
helper classes and functions have been developped over time to implement helper classes and functions have been developped over time to implement
operations commonly needed when dealing with the scan op. The scan op operations commonly needed when dealing with the scan op. The scan op
itself defines a large number of them and others can be found in the file itself defines a large number of them and others can be found in the file
``scan_utils.py``. This sections aims to point out the most useful ones sorted ``utils.py``. This sections aims to point out the most useful ones sorted
by usage. by usage.
Accessing/manipulating Scan's inputs and outputs by type Accessing/manipulating Scan's inputs and outputs by type
-------------------------------------------------------- --------------------------------------------------------
Declared in ``scan_utils.py``, the class ``scan_args`` handles the Declared in ``utils.py``, the class ``scan_args`` handles the
parsing of the inputs and outputs (both inner and outer) to a format parsing of the inputs and outputs (both inner and outer) to a format
that is easier to analyse and manipulate. Without this class, that is easier to analyse and manipulate. Without this class,
analysing Scan's inputs and outputs often required convoluted logic analysing Scan's inputs and outputs often required convoluted logic
......
doc/library/compile/opfromgraph.txt
浏览文件 @ 6d39faaf
...@@ -6,7 +6,7 @@ ...@@ -6,7 +6,7 @@
OpFromGraph OpFromGraph
=========== ===========
This page describes :class:`theano.OpFromGraph This page describes :class:`theano.compile.builders.OpFromGraph
<theano.compile.builders.OpFromGraph>`, an Op that allows to <theano.compile.builders.OpFromGraph>`, an Op that allows to
encapsulate a Theano graph in an op. encapsulate a Theano graph in an op.
......
doc/library/d3viz/index.ipynb
浏览文件 @ 6d39faaf
...@@ -391,7 +391,7 @@ ...@@ -391,7 +391,7 @@
"source": [ "source": [
"x, y, z = tt.scalars('xyz')\n", "x, y, z = tt.scalars('xyz')\n",
"e = tt.nnet.sigmoid((x + y + z)**2)\n", "e = tt.nnet.sigmoid((x + y + z)**2)\n",
"op = th.OpFromGraph([x, y, z], [e])\n", "op = th.compile.builders.OpFromGraph([x, y, z], [e])\n",
"\n", "\n",
"e2 = op(x, y, z) + op(z, y, x)\n", "e2 = op(x, y, z) + op(z, y, x)\n",
"f = th.function([x, y, z], e2)" "f = th.function([x, y, z], e2)"
...@@ -436,9 +436,9 @@ ...@@ -436,9 +436,9 @@
"source": [ "source": [
"x, y, z = tt.scalars('xyz')\n", "x, y, z = tt.scalars('xyz')\n",
"e = x * y\n", "e = x * y\n",
"op = th.OpFromGraph([x, y], [e])\n", "op = th.compile.builders.OpFromGraph([x, y], [e])\n",
"e2 = op(x, y) + z\n", "e2 = op(x, y) + z\n",
"op2 = th.OpFromGraph([x, y, z], [e2])\n", "op2 = th.compile.builders.OpFromGraph([x, y, z], [e2])\n",
"e3 = op2(x, y, z) + z\n", "e3 = op2(x, y, z) + z\n",
"f = th.function([x, y, z], [e3])" "f = th.function([x, y, z], [e3])"
] ]
......
doc/library/d3viz/index.txt
浏览文件 @ 6d39faaf
...@@ -220,7 +220,7 @@ defines a nested graph, which will be visualized accordingly by ...@@ -220,7 +220,7 @@ defines a nested graph, which will be visualized accordingly by
x, y, z = tt.scalars('xyz') x, y, z = tt.scalars('xyz')
e = tt.nnet.sigmoid((x + y + z)**2) e = tt.nnet.sigmoid((x + y + z)**2)
op = th.OpFromGraph([x, y, z], [e]) op = th.compile.builders.OpFromGraph([x, y, z], [e])
e2 = op(x, y, z) + op(z, y, x) e2 = op(x, y, z) + op(z, y, x)
f = th.function([x, y, z], e2) f = th.function([x, y, z], e2)
...@@ -249,9 +249,9 @@ the following example. ...@@ -249,9 +249,9 @@ the following example.
x, y, z = tt.scalars('xyz') x, y, z = tt.scalars('xyz')
e = x * y e = x * y
op = th.OpFromGraph([x, y], [e]) op = th.compile.builders.OpFromGraph([x, y], [e])
e2 = op(x, y) + z e2 = op(x, y) + z
op2 = th.OpFromGraph([x, y, z], [e2]) op2 = th.compile.builders.OpFromGraph([x, y, z], [e2])
e3 = op2(x, y, z) + z e3 = op2(x, y, z) + z
f = th.function([x, y, z], [e3]) f = th.function([x, y, z], [e3])
......
doc/library/gradient.txt
浏览文件 @ 6d39faaf
...@@ -52,7 +52,7 @@ list of ops that support R-op: ...@@ -52,7 +52,7 @@ list of ops that support R-op:
* Reshape * Reshape
* Flatten * Flatten
* DimShuffle * DimShuffle
* Scan [In scan_module/tests/test_scan.test_rop] * Scan [In scan/tests/test_scan.test_rop]
* without test * without test
* Split * Split
......
doc/library/scan.txt
浏览文件 @ 6d39faaf
...@@ -539,7 +539,7 @@ value ``max_value``. ...@@ -539,7 +539,7 @@ value ``max_value``.
.. testcode:: .. testcode::
def power_of_2(previous_power, max_value): def power_of_2(previous_power, max_value):
return previous_power*2, theano.scan_module.until(previous_power*2 > max_value) return previous_power*2, theano.scan.utils.until(previous_power*2 > max_value)
max_value = tt.scalar() max_value = tt.scalar()
values, _ = theano.scan(power_of_2, values, _ = theano.scan(power_of_2,
...@@ -557,7 +557,7 @@ value ``max_value``. ...@@ -557,7 +557,7 @@ value ``max_value``.
As you can see, in order to terminate on condition, the only thing required As you can see, in order to terminate on condition, the only thing required
is that the inner function ``power_of_2`` to return also the condition is that the inner function ``power_of_2`` to return also the condition
wrapped in the class ``theano.scan_module.until``. The condition has to be wrapped in the class ``theano.scan.utils.until``. The condition has to be
expressed in terms of the arguments of the inner function (in this case expressed in terms of the arguments of the inner function (in this case
``previous_power`` and ``max_value``). ``previous_power`` and ``max_value``).
...@@ -675,7 +675,7 @@ higher memory usage. ...@@ -675,7 +675,7 @@ higher memory usage.
reference reference
========= =========
.. automodule:: theano.scan_module .. automodule:: theano.scan
.. autofunction:: theano.map .. autofunction:: theano.map
.. autofunction:: theano.reduce .. autofunction:: theano.reduce
......
tests/d3viz/models.py
浏览文件 @ 6d39faaf
...@@ -32,9 +32,9 @@ class OfgNested: ...@@ -32,9 +32,9 @@ class OfgNested:
def __init__(self): def __init__(self):
x, y, z = tt.scalars("xyz") x, y, z = tt.scalars("xyz")
e = x * y e = x * y
op = theano.OpFromGraph([x, y], [e]) op = theano.compile.builders.OpFromGraph([x, y], [e])
e2 = op(x, y) + z e2 = op(x, y) + z
op2 = theano.OpFromGraph([x, y, z], [e2]) op2 = theano.compile.builders.OpFromGraph([x, y, z], [e2])
e3 = op2(x, y, z) + z e3 = op2(x, y, z) + z
self.inputs = [x, y, z] self.inputs = [x, y, z]
...@@ -45,7 +45,7 @@ class Ofg: ...@@ -45,7 +45,7 @@ class Ofg:
def __init__(self): def __init__(self):
x, y, z = tt.scalars("xyz") x, y, z = tt.scalars("xyz")
e = tt.nnet.sigmoid((x + y + z) ** 2) e = tt.nnet.sigmoid((x + y + z) ** 2)
op = theano.OpFromGraph([x, y, z], [e]) op = theano.compile.builders.OpFromGraph([x, y, z], [e])
e2 = op(x, y, z) + op(z, y, x) e2 = op(x, y, z) + op(z, y, x)
self.inputs = [x, y, z] self.inputs = [x, y, z]
...@@ -56,7 +56,7 @@ class OfgSimple: ...@@ -56,7 +56,7 @@ class OfgSimple:
def __init__(self): def __init__(self):
x, y, z = tt.scalars("xyz") x, y, z = tt.scalars("xyz")
e = tt.nnet.sigmoid((x + y + z) ** 2) e = tt.nnet.sigmoid((x + y + z) ** 2)
op = theano.OpFromGraph([x, y, z], [e]) op = theano.compile.builders.OpFromGraph([x, y, z], [e])
e2 = op(x, y, z) e2 = op(x, y, z)
self.inputs = [x, y, z] self.inputs = [x, y, z]
......
tests/gpuarray/test_scan.py
浏览文件 @ 6d39faaf
import numpy as np import numpy as np
import pytest
import theano import theano
import theano.sandbox.rng_mrg import theano.sandbox.rng_mrg
from tests import unittest_tools as utt from tests import unittest_tools as utt
from tests.gpuarray.config import mode_with_gpu, test_ctx_name from tests.gpuarray.config import mode_with_gpu, test_ctx_name
from theano import gpuarray, tensor
from theano.gpuarray.basic_ops import GpuFromHost, HostFromGpu from theano.gpuarray.basic_ops import GpuFromHost, HostFromGpu
from theano.gpuarray.elemwise import GpuElemwise from theano.gpuarray.elemwise import GpuElemwise
from theano.scan.basic import scan
from theano.scan.checkpoints import scan_checkpoints
from theano.scan.op import Scan
pygpu_gpuarray = pytest.importorskip("pygpy.gpuarray")
GpuArrayException = pygpu_gpuarray.GpuArrayException
if theano.config.mode == "FAST_COMPILE":
mode_with_opt = theano.compile.mode.get_mode("FAST_RUN")
else:
mode_with_opt = theano.compile.mode.get_default_mode()
if theano.config.mode in ("DEBUG_MODE", "DebugMode"):
mode_nodebug = theano.compile.mode.get_mode("FAST_RUN")
else:
mode_nodebug = mode_with_opt
class TestScan: class TestScan:
...@@ -22,7 +41,7 @@ class TestScan: ...@@ -22,7 +41,7 @@ class TestScan:
W = theano.tensor.fscalar("w") W = theano.tensor.fscalar("w")
mode = mode_with_gpu.excluding("InputToGpuOptimizer") mode = mode_with_gpu.excluding("InputToGpuOptimizer")
output, updates = theano.scan( output, updates = scan(
f_rnn, f_rnn,
u, u,
x0, x0,
...@@ -64,11 +83,7 @@ class TestScan: ...@@ -64,11 +83,7 @@ class TestScan:
# TO DEL # TO DEL
topo = f2.maker.fgraph.toposort() topo = f2.maker.fgraph.toposort()
scan_node = [ scan_node = [node for node in topo if isinstance(node.op, scan.op.Scan)]
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1 assert len(scan_node) == 1
scan_node = scan_node[0] scan_node = scan_node[0]
...@@ -76,11 +91,7 @@ class TestScan: ...@@ -76,11 +91,7 @@ class TestScan:
assert sum([isinstance(node.op, HostFromGpu) for node in topo]) == 0 assert sum([isinstance(node.op, HostFromGpu) for node in topo]) == 0
assert sum([isinstance(node.op, GpuFromHost) for node in topo]) == 4 assert sum([isinstance(node.op, GpuFromHost) for node in topo]) == 4
scan_node = [ scan_node = [node for node in topo if isinstance(node.op, scan.op.Scan)]
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1 assert len(scan_node) == 1
scan_node = scan_node[0] scan_node = scan_node[0]
scan_node_topo = scan_node.op.fn.maker.fgraph.toposort() scan_node_topo = scan_node.op.fn.maker.fgraph.toposort()
...@@ -99,7 +110,7 @@ class TestScan: ...@@ -99,7 +110,7 @@ class TestScan:
x0 = theano.tensor.fscalar("x0") x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win") W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w") W = theano.tensor.fscalar("w")
output, updates = theano.scan( output, updates = scan(
f_rnn, f_rnn,
u, u,
x0, x0,
...@@ -138,11 +149,7 @@ class TestScan: ...@@ -138,11 +149,7 @@ class TestScan:
assert sum([isinstance(node.op, HostFromGpu) for node in topo]) == 1 assert sum([isinstance(node.op, HostFromGpu) for node in topo]) == 1
assert sum([isinstance(node.op, GpuFromHost) for node in topo]) == 4 assert sum([isinstance(node.op, GpuFromHost) for node in topo]) == 4
scan_node = [ scan_node = [node for node in topo if isinstance(node.op, scan.op.Scan)]
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1 assert len(scan_node) == 1
scan_node = scan_node[0] scan_node = scan_node[0]
scan_node_topo = scan_node.op.fn.maker.fgraph.toposort() scan_node_topo = scan_node.op.fn.maker.fgraph.toposort()
...@@ -162,7 +169,7 @@ class TestScan: ...@@ -162,7 +169,7 @@ class TestScan:
x0 = theano.tensor.fscalar("x0") x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win") W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w") W = theano.tensor.fscalar("w")
output, updates = theano.scan( output, updates = scan(
f_rnn, f_rnn,
u, u,
[x0, None], [x0, None],
...@@ -202,11 +209,7 @@ class TestScan: ...@@ -202,11 +209,7 @@ class TestScan:
utt.assert_allclose(theano_out2, v_out2) utt.assert_allclose(theano_out2, v_out2)
topo = f2.maker.fgraph.toposort() topo = f2.maker.fgraph.toposort()
scan_node = [ scan_node = [node for node in topo if isinstance(node.op, scan.op.Scan)]
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1 assert len(scan_node) == 1
scan_node = scan_node[0] scan_node = scan_node[0]
assert scan_node.op.gpua assert scan_node.op.gpua
...@@ -236,7 +239,7 @@ class TestScan: ...@@ -236,7 +239,7 @@ class TestScan:
* vsample_tm1 * vsample_tm1
) )
theano_vsamples, updates = theano.scan( theano_vsamples, updates = scan(
f, f,
[], [],
vsample, vsample,
...@@ -257,3 +260,469 @@ class TestScan: ...@@ -257,3 +260,469 @@ class TestScan:
# I leave this to tested by debugmode, this test was anyway # I leave this to tested by debugmode, this test was anyway
# more of does the graph compile kind of test # more of does the graph compile kind of test
my_f() my_f()
class ScanGpuTests:
"""
This class defines a number of tests for Scan on GPU as well as a few
helper functions for these tests. The GPU tests defined in this class are
independent of the GPU backend used. Because of this, a class inheriting
from ScanGpuTests should define the following attributes and methods to
make the tests run on a specific backend :
- self.gpu_backend : Reference to the backend module
- self.mode_with_opt : Compilation mode to force usage of the gpu backend
- self.is_scan_on_gpu(node) : Method to determine is a scan node has been
moved to run on a gpu under the specific
backend. Returns a boolean.
"""
def test_one_sequence_one_output_weights_gpu1(self):
def f_rnn(u_t, x_tm1, W_in, W):
return u_t * W_in + x_tm1 * W
u = theano.tensor.fvector("u")
x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w")
# The following line is needed to have the first case being used
# Otherwise, it is the second that is tested.
mode = self.mode_with_gpu.excluding("InputToGpuOptimizer")
output, updates = scan(
f_rnn,
u,
x0,
[W_in, W],
n_steps=None,
truncate_gradient=-1,
go_backwards=False,
mode=mode,
)
output = self.gpu_backend.gpu_from_host(output)
f2 = theano.function(
[u, x0, W_in, W],
output,
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# get random initial values
rng = np.random.RandomState(utt.fetch_seed())
v_u = rng.uniform(size=(4,), low=-5.0, high=5.0)
v_x0 = rng.uniform()
W = rng.uniform()
W_in = rng.uniform()
v_u = np.asarray(v_u, dtype="float32")
v_x0 = np.asarray(v_x0, dtype="float32")
W = np.asarray(W, dtype="float32")
W_in = np.asarray(W_in, dtype="float32")
# compute the output in numpy
v_out = np.zeros((4,))
v_out[0] = v_u[0] * W_in + v_x0 * W
for step in range(1, 4):
v_out[step] = v_u[step] * W_in + v_out[step - 1] * W
theano_values = f2(v_u, v_x0, W_in, W)
utt.assert_allclose(theano_values, v_out)
# TO DEL
topo = f2.maker.fgraph.toposort()
scan_node = [node for node in topo if isinstance(node.op, Scan)]
assert len(scan_node) == 1
scan_node = scan_node[0]
topo = f2.maker.fgraph.toposort()
assert (
sum([isinstance(node.op, self.gpu_backend.HostFromGpu) for node in topo])
== 0
)
assert (
sum([isinstance(node.op, self.gpu_backend.GpuFromHost) for node in topo])
== 4
)
scan_node = [node for node in topo if isinstance(node.op, Scan)]
assert len(scan_node) == 1
scan_node = scan_node[0]
scan_node_topo = scan_node.op.fn.maker.fgraph.toposort()
# check that there is no gpu transfer in the inner loop.
assert any(
[
isinstance(node.op, self.gpu_backend.GpuElemwise)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.HostFromGpu)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.GpuFromHost)
for node in scan_node_topo
]
)
# This second version test the second case in the optimizer to the gpu.
def test_one_sequence_one_output_weights_gpu2(self):
def f_rnn(u_t, x_tm1, W_in, W):
return u_t * W_in + x_tm1 * W
u = theano.tensor.fvector("u")
x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w")
output, updates = scan(
f_rnn,
u,
x0,
[W_in, W],
n_steps=None,
truncate_gradient=-1,
go_backwards=False,
mode=self.mode_with_gpu,
)
f2 = theano.function(
[u, x0, W_in, W],
output,
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# get random initial values
rng = np.random.RandomState(utt.fetch_seed())
v_u = rng.uniform(size=(4,), low=-5.0, high=5.0)
v_x0 = rng.uniform()
W = rng.uniform()
W_in = rng.uniform()
# compute the output in numpy
v_out = np.zeros((4,))
v_out[0] = v_u[0] * W_in + v_x0 * W
for step in range(1, 4):
v_out[step] = v_u[step] * W_in + v_out[step - 1] * W
theano_values = f2(v_u, v_x0, W_in, W)
utt.assert_allclose(theano_values, v_out)
topo = f2.maker.fgraph.toposort()
assert (
sum([isinstance(node.op, self.gpu_backend.HostFromGpu) for node in topo])
== 1
)
assert (
sum([isinstance(node.op, self.gpu_backend.GpuFromHost) for node in topo])
== 4
)
scan_node = [node for node in topo if isinstance(node.op, Scan)]
assert len(scan_node) == 1
scan_node = scan_node[0]
scan_node_topo = scan_node.op.fn.maker.fgraph.toposort()
# check that there is no gpu transfer in the inner loop.
assert any(
[
isinstance(node.op, self.gpu_backend.GpuElemwise)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.HostFromGpu)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.GpuFromHost)
for node in scan_node_topo
]
)
# This third test checks that scan can deal with a mixture of dtypes as
# outputs when is running on GPU
def test_gpu3_mixture_dtype_outputs(self):
def f_rnn(u_t, x_tm1, W_in, W):
return (u_t * W_in + x_tm1 * W, tensor.cast(u_t + x_tm1, "int64"))
u = theano.tensor.fvector("u")
x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w")
output, updates = scan(
f_rnn,
u,
[x0, None],
[W_in, W],
n_steps=None,
truncate_gradient=-1,
go_backwards=False,
mode=self.mode_with_gpu,
)
f2 = theano.function(
[u, x0, W_in, W],
output,
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# get random initial values
rng = np.random.RandomState(utt.fetch_seed())
v_u = rng.uniform(size=(4,), low=-5.0, high=5.0)
v_x0 = rng.uniform()
W = rng.uniform()
W_in = rng.uniform()
# compute the output in numpy
v_out1 = np.zeros((4,))
v_out2 = np.zeros((4,), dtype="int64")
v_out1[0] = v_u[0] * W_in + v_x0 * W
v_out2[0] = v_u[0] + v_x0
for step in range(1, 4):
v_out1[step] = v_u[step] * W_in + v_out1[step - 1] * W
v_out2[step] = np.int64(v_u[step] + v_out1[step - 1])
theano_out1, theano_out2 = f2(v_u, v_x0, W_in, W)
utt.assert_allclose(theano_out1, v_out1)
utt.assert_allclose(theano_out2, v_out2)
topo = f2.maker.fgraph.toposort()
scan_node = [node for node in topo if isinstance(node.op, Scan)]
assert len(scan_node) == 1
scan_node = scan_node[0]
assert self.is_scan_on_gpu(scan_node)
def test_gibbs_chain(self):
rng = np.random.RandomState(utt.fetch_seed())
v_vsample = np.array(
rng.binomial(
1,
0.5,
size=(3, 20),
),
dtype="float32",
)
vsample = theano.shared(v_vsample)
trng = theano.sandbox.rng_mrg.MRG_RandomStreams(utt.fetch_seed())
def f(vsample_tm1):
return (
trng.binomial(vsample_tm1.shape, n=1, p=0.3, dtype="float32")
* vsample_tm1
)
theano_vsamples, updates = scan(
f,
[],
vsample,
[],
n_steps=10,
truncate_gradient=-1,
go_backwards=False,
mode=self.mode_with_gpu,
)
my_f = theano.function(
[],
theano_vsamples[-1],
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# I leave this to tested by debugmode, this test was anyway more of
# doest the graph compile kind of test
my_f()
def test_gpu_memory_usage(self):
# This test validates that the memory usage of the defined theano
# function is reasonnable when executed on the GPU. It checks for
# a bug in which one of scan's optimization was not applied which
# made the scan node compute large and unnecessary outputs which
# brought memory usage on the GPU to ~12G.
# Dimensionality of input and output data (not one-hot coded)
n_in = 100
n_out = 100
# Number of neurons in hidden layer
n_hid = 4000
# Number of minibatches
mb_size = 2
# Time steps in minibatch
mb_length = 200
# Define input variables
xin = tensor.ftensor3(name="xin")
yout = tensor.ftensor3(name="yout")
# Initialize the network parameters
U = theano.shared(np.zeros((n_in, n_hid), dtype="float32"), name="W_xin_to_l1")
V = theano.shared(np.zeros((n_hid, n_hid), dtype="float32"), name="W_l1_to_l1")
W = theano.shared(np.zeros((n_hid, n_out), dtype="float32"), name="W_l1_to_l2")
nparams = [U, V, W]
# Build the forward pass
l1_base = tensor.dot(xin, U)
def scan_l(baseline, last_step):
return baseline + tensor.dot(last_step, V)
zero_output = tensor.alloc(np.asarray(0.0, dtype="float32"), mb_size, n_hid)
l1_out, _ = scan(
scan_l,
sequences=[l1_base],
outputs_info=[zero_output],
mode=self.mode_with_gpu_nodebug,
)
l2_out = tensor.dot(l1_out, W)
# Compute the cost and take the gradient wrt params
cost = tensor.sum((l2_out - yout) ** 2)
grads = tensor.grad(cost, nparams)
updates = list(zip(nparams, (n - g for n, g in zip(nparams, grads))))
# Compile the theano function
feval_backprop = theano.function(
[xin, yout], cost, updates=updates, mode=self.mode_with_gpu_nodebug
)
# Validate that the PushOutScanOutput optimization has been applied
# by checking the number of outputs of the grad Scan node in the
# compiled function.
nodes = feval_backprop.maker.fgraph.toposort()
scan_nodes = [n for n in nodes if isinstance(n.op, Scan)]
# The grad scan is always the 2nd one according to toposort. If the
# optimization has been applied, it has 2 outputs, otherwise 3.
grad_scan_node = scan_nodes[1]
assert len(grad_scan_node.outputs) == 2, len(grad_scan_node.outputs)
# Call the theano function to ensure the absence of a memory error
feval_backprop(
np.zeros((mb_length, mb_size, n_in), dtype="float32"),
np.zeros((mb_length, mb_size, n_out), dtype="float32"),
)
def test_memory_reuse_gpudimshuffle(self):
# Test the memory pre-allocation feature in scan when one output is
# the result of a GpuDimshuffle (because an optimization in
# GpuDimshuffle can cause issues with the memory pre-allocation
# where it falsely thinks that a pre-allocated memory region has
# been used when it hasn't).
def inner_fn(seq1, recurrent_out):
temp = seq1 + recurrent_out.sum()
output1 = temp.dimshuffle(1, 0)
output2 = temp.sum() + recurrent_out
return output1, output2
input1 = theano.tensor.ftensor3()
init = theano.tensor.ftensor3()
outputs_info = [None, init]
out, _ = scan(
inner_fn,
sequences=[input1],
outputs_info=outputs_info,
mode=self.mode_with_gpu,
)
out1 = out[0].flatten()
out2 = out[1].flatten()
fct = theano.function([input1, init], [out1, out2], mode=self.mode_with_gpu)
output = fct(
np.ones((2, 1, 1), dtype="float32"), np.ones((1, 1, 1), dtype="float32")
)
expected_output = (
np.array([2, 4], dtype="float32"),
np.array([3, 7], dtype="float32"),
)
utt.assert_allclose(output, expected_output)
class TestScanGpuarray(ScanGpuTests):
"""
This class takes the gpu tests for scan that are defined in
class ScanGpuTests and runs them using the gpuarray backend.
"""
def setup_method(self):
self.gpu_backend = gpuarray
# This is unfortunate, but required
def gpu_from_host(v):
return gpuarray.GpuFromHost(None)(v)
self.gpu_backend.gpu_from_host = gpu_from_host
self.mode_with_gpu = mode_with_opt.including("gpuarray", "scan")
self.mode_with_gpu_nodebug = mode_nodebug.including("gpuarray", "scan")
# Skip the test if pygpu is not available
if not self.gpu_backend.pygpu_activated:
pytest.skip("Optional package pygpu disabled")
utt.seed_rng()
def is_scan_on_gpu(self, node):
return node.op.info.get("gpua", False)
class TestScanCheckpoint:
def setup_method(self):
self.k = tensor.iscalar("k")
self.A = tensor.vector("A")
result, _ = scan(
fn=lambda prior_result, A: prior_result * A,
outputs_info=tensor.ones_like(self.A),
non_sequences=self.A,
n_steps=self.k,
)
result_check, _ = scan_checkpoints(
fn=lambda prior_result, A: prior_result * A,
outputs_info=tensor.ones_like(self.A),
non_sequences=self.A,
n_steps=self.k,
save_every_N=100,
)
self.result = result[-1]
self.result_check = result_check[-1]
self.grad_A = tensor.grad(self.result.sum(), self.A)
self.grad_A_check = tensor.grad(self.result_check.sum(), self.A)
def test_memory(self):
from tests.gpuarray.config import mode_with_gpu # noqa
f = theano.function(
inputs=[self.A, self.k], outputs=self.grad_A, mode=mode_with_gpu
)
f_check = theano.function(
inputs=[self.A, self.k], outputs=self.grad_A_check, mode=mode_with_gpu
)
free_gmem = theano.gpuarray.type._context_reg[None].free_gmem
data = np.ones(free_gmem // 3000, dtype=np.float32)
# Check that it works with the checkpoints
size = 1000
if isinstance(mode_with_gpu, theano.compile.DebugMode):
size = 100
f_check(data, size)
# Check that the basic scan fails in that case
# Skip that check in DebugMode, as it can fail in different ways
if not isinstance(mode_with_gpu, theano.compile.DebugMode):
with pytest.raises(GpuArrayException):
f(data, 1000)
tests/scan_module/test_scan.py → tests/scan/test_basic.py
浏览文件 @ 6d39faaf
"""
Questions and notes about scan that should be answered :
* Scan seems to do copies of every input variable. Is that needed?
answer : probably not, but it doesn't hurt also ( what we copy is
theano variables, which just cary information about the type / dimension
of the data)
* There is some of scan functionality that is not well documented
"""
import os import os
import pickle import pickle
import shutil import shutil
...@@ -15,20 +26,11 @@ import theano.scalar.sharedvar ...@@ -15,20 +26,11 @@ import theano.scalar.sharedvar
from tests import unittest_tools as utt from tests import unittest_tools as utt
from theano import tensor from theano import tensor
from theano.compile.function.pfunc import rebuild_collect_shared from theano.compile.function.pfunc import rebuild_collect_shared
from theano.scan_module.scan_op import Scan from theano.scan.basic import scan
from theano.scan.op import Scan
from theano.scan.opt import ScanMerge
""" from theano.scan.utils import until
Questions and notes about scan that should be answered :
* Scan seems to do copies of every input variable. Is that needed?
answer : probably not, but it doesn't hurt also ( what we copy is
theano variables, which just cary information about the type / dimension
of the data)
* There is some of scan functionality that is not well documented
"""
if theano.config.mode == "FAST_COMPILE": if theano.config.mode == "FAST_COMPILE":
mode_with_opt = theano.compile.mode.get_mode("FAST_RUN") mode_with_opt = theano.compile.mode.get_mode("FAST_RUN")
...@@ -154,7 +156,7 @@ class multiple_outputs_numeric_grad: ...@@ -154,7 +156,7 @@ class multiple_outputs_numeric_grad:
# DONE - rp # DONE - rp
def scan_project_sum(*args, **kwargs): def scan_project_sum(*args, **kwargs):
rng = theano.tensor.shared_randomstreams.RandomStreams(123) rng = theano.tensor.shared_randomstreams.RandomStreams(123)
scan_outputs, updates = theano.scan(*args, **kwargs) scan_outputs, updates = scan(*args, **kwargs)
if type(scan_outputs) not in [list, tuple]: if type(scan_outputs) not in [list, tuple]:
scan_outputs = [scan_outputs] scan_outputs = [scan_outputs]
# we should ignore the random-state updates so that # we should ignore the random-state updates so that
...@@ -222,7 +224,7 @@ class TestScan: ...@@ -222,7 +224,7 @@ class TestScan:
state = theano.tensor.scalar("state") state = theano.tensor.scalar("state")
n_steps = theano.tensor.iscalar("nsteps") n_steps = theano.tensor.iscalar("nsteps")
output, updates = theano.scan( output, updates = scan(
f_pow2, f_pow2,
[], [],
state, state,
...@@ -268,7 +270,7 @@ class TestScan: ...@@ -268,7 +270,7 @@ class TestScan:
state = theano.tensor.scalar("state") state = theano.tensor.scalar("state")
n_steps = theano.tensor.iscalar("nsteps") n_steps = theano.tensor.iscalar("nsteps")
output, updates = theano.scan(f_pow2, [], state, [], n_steps=n_steps) output, updates = scan(f_pow2, [], state, [], n_steps=n_steps)
f = theano.function( f = theano.function(
[state, n_steps], output, updates=updates, allow_input_downcast=True [state, n_steps], output, updates=updates, allow_input_downcast=True
...@@ -304,7 +306,7 @@ class TestScan: ...@@ -304,7 +306,7 @@ class TestScan:
state = theano.tensor.scalar("state") state = theano.tensor.scalar("state")
n_steps = theano.tensor.iscalar("nsteps") n_steps = theano.tensor.iscalar("nsteps")
# Test return_list at the same time. # Test return_list at the same time.
output, updates = theano.scan( output, updates = scan(
f_pow2, f_pow2,
[], [],
state, state,
...@@ -336,7 +338,7 @@ class TestScan: ...@@ -336,7 +338,7 @@ class TestScan:
state = theano.tensor.vector("state") state = theano.tensor.vector("state")
n_steps = theano.tensor.iscalar("nsteps") n_steps = theano.tensor.iscalar("nsteps")
output, updates = theano.scan( output, updates = scan(
f_pow2, f_pow2,
[], [],
state, state,
...@@ -353,11 +355,7 @@ class TestScan: ...@@ -353,11 +355,7 @@ class TestScan:
updates=updates, updates=updates,
allow_input_downcast=True, allow_input_downcast=True,
) )
nodes = [ nodes = [x for x in my_f.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in my_f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
# This assertion fails if savemem optimization failed on scan # This assertion fails if savemem optimization failed on scan
if theano.config.mode != "FAST_COMPILE": if theano.config.mode != "FAST_COMPILE":
assert nodes[0].op._scan_savemem_visited assert nodes[0].op._scan_savemem_visited
...@@ -369,9 +367,7 @@ class TestScan: ...@@ -369,9 +367,7 @@ class TestScan:
# Compile the Theano function # Compile the Theano function
n_steps = 2 n_steps = 2
inp = tensor.matrix() inp = tensor.matrix()
broadcasted_inp, _ = theano.scan( broadcasted_inp, _ = scan(lambda x: x, non_sequences=[inp], n_steps=n_steps)
lambda x: x, non_sequences=[inp], n_steps=n_steps
)
out = broadcasted_inp.sum() out = broadcasted_inp.sum()
gr = tensor.grad(out, inp) gr = tensor.grad(out, inp)
fun = theano.function([inp], [broadcasted_inp, gr]) fun = theano.function([inp], [broadcasted_inp, gr])
...@@ -396,7 +392,7 @@ class TestScan: ...@@ -396,7 +392,7 @@ class TestScan:
W_in = theano.tensor.scalar("win") W_in = theano.tensor.scalar("win")
W = theano.tensor.scalar("w") W = theano.tensor.scalar("w")
output, updates = theano.scan( output, updates = scan(
f_rnn, f_rnn,
u, u,
x0, x0,
...@@ -436,7 +432,7 @@ class TestScan: ...@@ -436,7 +432,7 @@ class TestScan:
def f_rnn_shared(u_t, x_tm1, tmp_W_in, tmp_W): def f_rnn_shared(u_t, x_tm1, tmp_W_in, tmp_W):
return u_t * tmp_W_in + x_tm1 * tmp_W return u_t * tmp_W_in + x_tm1 * tmp_W
output, updates = theano.scan( output, updates = scan(
f_rnn_shared, f_rnn_shared,
u, u,
x0, x0,
...@@ -491,7 +487,7 @@ class TestScan: ...@@ -491,7 +487,7 @@ class TestScan:
theano.tensor.dot(x_tm1, W_out), theano.tensor.dot(x_tm1, W_out),
] ]
outputs, updates = theano.scan( outputs, updates = scan(
f_rnn_cmpl, f_rnn_cmpl,
[u1, u2], [u1, u2],
[x0, y0], [x0, y0],
...@@ -548,7 +544,7 @@ class TestScan: ...@@ -548,7 +544,7 @@ class TestScan:
theano.tensor.dot(u1_t, W_in1), theano.tensor.dot(u1_t, W_in1),
] ]
outputs, updates = theano.scan( outputs, updates = scan(
f_rnn_cmpl, f_rnn_cmpl,
[u1, dict(input=u2, taps=[-1, 0, 1])], [u1, dict(input=u2, taps=[-1, 0, 1])],
[x0, dict(initial=y0, taps=[-1, -3]), None], [x0, dict(initial=y0, taps=[-1, -3]), None],
...@@ -615,7 +611,7 @@ class TestScan: ...@@ -615,7 +611,7 @@ class TestScan:
# this test refers to a bug reported by Nicolas # this test refers to a bug reported by Nicolas
# Boulanger-Lewandowski June 6th # Boulanger-Lewandowski June 6th
x = theano.tensor.dvector() x = theano.tensor.dvector()
y, updates = theano.scan( y, updates = scan(
lambda x: [x], sequences=dict(input=x, taps=[-1]), outputs_info=[None] lambda x: [x], sequences=dict(input=x, taps=[-1]), outputs_info=[None]
) )
inp = np.arange(5).astype("float64") inp = np.arange(5).astype("float64")
...@@ -629,7 +625,7 @@ class TestScan: ...@@ -629,7 +625,7 @@ class TestScan:
return x return x
x = tensor.fvector("x") x = tensor.fvector("x")
res, upd = theano.scan(lp, sequences=dict(input=x, taps=[-2, -1])) res, upd = scan(lp, sequences=dict(input=x, taps=[-2, -1]))
f = theano.function([x], res, updates=upd) f = theano.function([x], res, updates=upd)
output = f([1, 2, 3, 4, 5]) output = f([1, 2, 3, 4, 5])
...@@ -649,7 +645,7 @@ class TestScan: ...@@ -649,7 +645,7 @@ class TestScan:
a0 = theano.shared(np.arange(2)) a0 = theano.shared(np.arange(2))
b0 = theano.shared(np.arange(2)) b0 = theano.shared(np.arange(2))
(a, b), _ = theano.scan( (a, b), _ = scan(
fn, fn,
outputs_info=[ outputs_info=[
{"initial": a0, "taps": [-2, -1]}, {"initial": a0, "taps": [-2, -1]},
...@@ -684,7 +680,7 @@ class TestScan: ...@@ -684,7 +680,7 @@ class TestScan:
state_next = state_old * 2 + state_current + seq state_next = state_old * 2 + state_current + seq
return state_next return state_next
out, _ = theano.scan( out, _ = scan(
inner_fct, sequences=seq, outputs_info={"initial": x, "taps": [-2, -1]} inner_fct, sequences=seq, outputs_info={"initial": x, "taps": [-2, -1]}
) )
...@@ -717,9 +713,7 @@ class TestScan: ...@@ -717,9 +713,7 @@ class TestScan:
inputs = [tensor.matrix(), tensor.vector()] inputs = [tensor.matrix(), tensor.vector()]
outputs_info = [dict(initial=inputs[0], taps=[-2, -1]), inputs[1]] outputs_info = [dict(initial=inputs[0], taps=[-2, -1]), inputs[1]]
scan_outputs, updates = theano.scan( scan_outputs, updates = scan(fn=inner_fct, outputs_info=outputs_info, n_steps=5)
fn=inner_fct, outputs_info=outputs_info, n_steps=5
)
# Take the gradient of each output wrt its corresponding initial state # Take the gradient of each output wrt its corresponding initial state
gradients = [ gradients = [
...@@ -734,7 +728,7 @@ class TestScan: ...@@ -734,7 +728,7 @@ class TestScan:
def test_verify_second_grad_sitsot(self): def test_verify_second_grad_sitsot(self):
def get_sum_of_grad(inp): def get_sum_of_grad(inp):
scan_outputs, updates = theano.scan( scan_outputs, updates = scan(
fn=lambda x: x * 2, outputs_info=[inp], n_steps=5 fn=lambda x: x * 2, outputs_info=[inp], n_steps=5
) )
...@@ -756,7 +750,7 @@ class TestScan: ...@@ -756,7 +750,7 @@ class TestScan:
def get_sum_of_grad(input0, input1): def get_sum_of_grad(input0, input1):
outputs_info = [dict(initial=input0, taps=[-2]), input1] outputs_info = [dict(initial=input0, taps=[-2]), input1]
scan_outputs, updates = theano.scan( scan_outputs, updates = scan(
fn=inner_fct, outputs_info=outputs_info, n_steps=3 fn=inner_fct, outputs_info=outputs_info, n_steps=3
) )
...@@ -793,13 +787,13 @@ class TestScan: ...@@ -793,13 +787,13 @@ class TestScan:
y_t = tensor.nnet.softmax_graph(a2_t) y_t = tensor.nnet.softmax_graph(a2_t)
return y_t return y_t
y, _ = theano.scan(fn=forward_scanner, sequences=x, outputs_info=[None]) y, _ = scan(fn=forward_scanner, sequences=x, outputs_info=[None])
# loss function # loss function
def error_scanner(y_t, t_t): def error_scanner(y_t, t_t):
return tensor.mean(tensor.nnet.categorical_crossentropy(y_t, t_t)) return tensor.mean(tensor.nnet.categorical_crossentropy(y_t, t_t))
L, _ = theano.scan(fn=error_scanner, sequences=[y, t], outputs_info=[None]) L, _ = scan(fn=error_scanner, sequences=[y, t], outputs_info=[None])
L = tensor.mean(L) L = tensor.mean(L)
# backward pass # backward pass
...@@ -823,7 +817,7 @@ class TestScan: ...@@ -823,7 +817,7 @@ class TestScan:
def f_rnn_shared(u_tm2, x_tm1, x_tm2): def f_rnn_shared(u_tm2, x_tm1, x_tm2):
return u_tm2 * W_in + x_tm1 * W + x_tm2 return u_tm2 * W_in + x_tm1 * W + x_tm2
outputs, updates = theano.scan( outputs, updates = scan(
f_rnn_shared, f_rnn_shared,
dict(input=u, taps=-2), dict(input=u, taps=-2),
dict(initial=x0, taps=[-1, -2]), dict(initial=x0, taps=[-1, -2]),
...@@ -869,7 +863,7 @@ class TestScan: ...@@ -869,7 +863,7 @@ class TestScan:
def f_rnn_shared(u_tm2, u_tp2, x_tm1, x_tm2): def f_rnn_shared(u_tm2, u_tp2, x_tm1, x_tm2):
return (u_tm2 + u_tp2) * W_in + x_tm1 * W + x_tm2 return (u_tm2 + u_tp2) * W_in + x_tm1 * W + x_tm2
output, updates = theano.scan( output, updates = scan(
f_rnn_shared, f_rnn_shared,
dict(input=u, taps=[-2, 2]), dict(input=u, taps=[-2, 2]),
dict(initial=x0, taps=[-1, -2]), dict(initial=x0, taps=[-1, -2]),
...@@ -921,7 +915,7 @@ class TestScan: ...@@ -921,7 +915,7 @@ class TestScan:
u0_t * W_in + x1_tm1 * W + u1_t + u2_t, u0_t * W_in + x1_tm1 * W + u1_t + u2_t,
] ]
outputs, updates = theano.scan( outputs, updates = scan(
f_rnn_shared, f_rnn_shared,
[u0, u1, u2], [u0, u1, u2],
[dict(initial=x0, inplace=u2), dict(initial=x1, inplace=u1)], [dict(initial=x0, inplace=u2), dict(initial=x1, inplace=u1)],
...@@ -939,11 +933,7 @@ class TestScan: ...@@ -939,11 +933,7 @@ class TestScan:
mode=mode, mode=mode,
allow_input_downcast=True, allow_input_downcast=True,
) )
scan_node = [ scan_node = [x for x in f9.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f9.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert 0 in scan_node[0].op.destroy_map.keys() assert 0 in scan_node[0].op.destroy_map.keys()
assert 1 in scan_node[0].op.destroy_map.keys() assert 1 in scan_node[0].op.destroy_map.keys()
# compute output in numpy # compute output in numpy
...@@ -992,7 +982,7 @@ class TestScan: ...@@ -992,7 +982,7 @@ class TestScan:
u0_t * W_in + x1_tm1 * W + u2_tm1 + u2_t + u2_tp1, u0_t * W_in + x1_tm1 * W + u2_tm1 + u2_t + u2_tp1,
] ]
outputs, updates = theano.scan( outputs, updates = scan(
f_rnn_shared, f_rnn_shared,
[u0, dict(input=u1, taps=[0, 1]), dict(input=u2, taps=[-1, 0, +1])], [u0, dict(input=u1, taps=[0, 1]), dict(input=u2, taps=[-1, 0, +1])],
[dict(initial=x0), dict(initial=x1)], [dict(initial=x0), dict(initial=x1)],
...@@ -1010,11 +1000,7 @@ class TestScan: ...@@ -1010,11 +1000,7 @@ class TestScan:
allow_input_downcast=True, allow_input_downcast=True,
) )
scan_node = [ scan_node = [x for x in f9.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f9.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert 0 in scan_node[0].op.destroy_map.keys() assert 0 in scan_node[0].op.destroy_map.keys()
assert 1 in scan_node[0].op.destroy_map.keys() assert 1 in scan_node[0].op.destroy_map.keys()
# compute output in numpy # compute output in numpy
...@@ -1043,7 +1029,7 @@ class TestScan: ...@@ -1043,7 +1029,7 @@ class TestScan:
vx1 = asarrayX(rng.uniform()) vx1 = asarrayX(rng.uniform())
x0 = theano.shared(vx0) x0 = theano.shared(vx0)
x1 = theano.shared(vx1) x1 = theano.shared(vx1)
outputs, updates = theano.scan( outputs, updates = scan(
lambda x, y: (x + asarrayX(1), y + asarrayX(1)), [], [x0, x1], n_steps=3 lambda x, y: (x + asarrayX(1), y + asarrayX(1)), [], [x0, x1], n_steps=3
) )
x0 = asarrayX(np.zeros((3,))) x0 = asarrayX(np.zeros((3,)))
...@@ -1053,11 +1039,7 @@ class TestScan: ...@@ -1053,11 +1039,7 @@ class TestScan:
outputs = theano.clone(outputs, replace=[(to_replace, x0)]) outputs = theano.clone(outputs, replace=[(to_replace, x0)])
mode = theano.compile.mode.get_mode(None).including("inplace") mode = theano.compile.mode.get_mode(None).including("inplace")
f9 = theano.function([], outputs, updates=updates, mode=mode) f9 = theano.function([], outputs, updates=updates, mode=mode)
scan_node = [ scan_node = [x for x in f9.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f9.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert 0 not in scan_node[0].op.destroy_map.keys() assert 0 not in scan_node[0].op.destroy_map.keys()
assert 1 in scan_node[0].op.destroy_map.keys() assert 1 in scan_node[0].op.destroy_map.keys()
...@@ -1075,7 +1057,7 @@ class TestScan: ...@@ -1075,7 +1057,7 @@ class TestScan:
# Their is a bug when floatX=float32 when we remove this line. # Their is a bug when floatX=float32 when we remove this line.
# The trace back is: # The trace back is:
# Traceback (most recent call last): # Traceback (most recent call last):
# File "/u/bastienf/repos/Theano/tests/test_scan.py", line 434, in test_shared_arguments_with_updates # File "/u/bastienf/repos/Theano/tests/scan/test_basic.py", line 434, in test_shared_arguments_with_updates
# theano_y0,theano_y1,theano_y2 = f10(vu2, vy0) # theano_y0,theano_y1,theano_y2 = f10(vu2, vy0)
# File "/u/bastienf/repos/theano/compile/function/types.py", line 480, in __call__ # File "/u/bastienf/repos/theano/compile/function/types.py", line 480, in __call__
# self.fn() # self.fn()
...@@ -1085,9 +1067,9 @@ class TestScan: ...@@ -1085,9 +1067,9 @@ class TestScan:
# th() # th()
# File "/u/bastienf/repos/theano/gof/cc.py", line 1141, in <lambda> # File "/u/bastienf/repos/theano/gof/cc.py", line 1141, in <lambda>
# thunk = lambda p = p, i = node_input_storage, o = node_output_storage, n = node: p(n, [x[0] for x in i], o) # thunk = lambda p = p, i = node_input_storage, o = node_output_storage, n = node: p(n, [x[0] for x in i], o)
# File "/u/bastienf/repos/theano/scan.py", line 922, in perform # File "/u/bastienf/repos/theano/scan/op.py", line 922, in perform
# inplace_map) # inplace_map)
# File "/u/bastienf/repos/theano/scan.py", line 1054, in scan # File "/u/bastienf/repos/theano/scan/basic.py", line 1054, in scan
# something = fn(*fn_args) # something = fn(*fn_args)
# File "/u/bastienf/repos/theano/compile/function/types.py", line 458, in __call__ # File "/u/bastienf/repos/theano/compile/function/types.py", line 458, in __call__
# s.storage[0] = s.type.filter(arg, strict=s.strict) # s.storage[0] = s.type.filter(arg, strict=s.strict)
...@@ -1097,7 +1079,7 @@ class TestScan: ...@@ -1097,7 +1079,7 @@ class TestScan:
# rval = numpy.asarray(a, dtype=dtype, order=order) # rval = numpy.asarray(a, dtype=dtype, order=order)
# File "/u/lisa/local/byhost/ceylon.iro.umontreal.ca//lib64/python2.5/site-packages/numpy/core/numeric.py", line 230, in asarray # File "/u/lisa/local/byhost/ceylon.iro.umontreal.ca//lib64/python2.5/site-packages/numpy/core/numeric.py", line 230, in asarray
# return array(a, dtype, copy=False, order=order) # return array(a, dtype, copy=False, order=order)
# TypeError: ('__array__() takes no arguments (1 given)', <theano.scan.Scan object at 0x3dbbf90>(?_steps, u1, u2, y0, y1, 0.0, W1, W2), 'Sequence id of Apply node=0') # TypeError: ('__array__() takes no arguments (1 given)', <Scan object at 0x3dbbf90>(?_steps, u1, u2, y0, y1, 0.0, W1, W2), 'Sequence id of Apply node=0')
# #
# This don't seam to be a theano related bug... # This don't seam to be a theano related bug...
vu1 = asarrayX(rng.rand(3, 2)) vu1 = asarrayX(rng.rand(3, 2))
...@@ -1123,7 +1105,7 @@ class TestScan: ...@@ -1123,7 +1105,7 @@ class TestScan:
u2 = theano.tensor.matrix("u2") u2 = theano.tensor.matrix("u2")
y0 = theano.tensor.matrix("y0") y0 = theano.tensor.matrix("y0")
outputs, updates = theano.scan( outputs, updates = scan(
f, f,
[u1, u2], [u1, u2],
[dict(initial=y0, taps=[-3, -2, -1]), y1, None], [dict(initial=y0, taps=[-3, -2, -1]), y1, None],
...@@ -1176,7 +1158,7 @@ class TestScan: ...@@ -1176,7 +1158,7 @@ class TestScan:
new_y = tensor.switch(cond, y, tensor.nnet.sigmoid(x)) new_y = tensor.switch(cond, y, tensor.nnet.sigmoid(x))
return new_cond, new_x, new_y return new_cond, new_x, new_y
values, _ = theano.scan( values, _ = scan(
inner_fn, inner_fn,
outputs_info=[c, x, y], outputs_info=[c, x, y],
n_steps=10, n_steps=10,
...@@ -1191,7 +1173,7 @@ class TestScan: ...@@ -1191,7 +1173,7 @@ class TestScan:
def test_simple_shared_mrg_random(self): def test_simple_shared_mrg_random(self):
theano_rng = theano.sandbox.rng_mrg.MRG_RandomStreams(utt.fetch_seed()) theano_rng = theano.sandbox.rng_mrg.MRG_RandomStreams(utt.fetch_seed())
values, updates = theano.scan( values, updates = scan(
lambda: theano_rng.uniform((2,), -1, 1), lambda: theano_rng.uniform((2,), -1, 1),
[], [],
[], [],
...@@ -1209,7 +1191,7 @@ class TestScan: ...@@ -1209,7 +1191,7 @@ class TestScan:
def test_simple_shared_random(self): def test_simple_shared_random(self):
theano_rng = theano.tensor.shared_randomstreams.RandomStreams(utt.fetch_seed()) theano_rng = theano.tensor.shared_randomstreams.RandomStreams(utt.fetch_seed())
values, updates = theano.scan( values, updates = scan(
lambda: theano_rng.uniform((2,), -1, 1), lambda: theano_rng.uniform((2,), -1, 1),
[], [],
[], [],
...@@ -1265,7 +1247,7 @@ class TestScan: ...@@ -1265,7 +1247,7 @@ class TestScan:
trng.binomial(vmean_t.shape, 1, vmean_t), dtype="float32" trng.binomial(vmean_t.shape, 1, vmean_t), dtype="float32"
) )
theano_vsamples, updates = theano.scan( theano_vsamples, updates = scan(
f, [], vsample, [], n_steps=10, truncate_gradient=-1, go_backwards=False f, [], vsample, [], n_steps=10, truncate_gradient=-1, go_backwards=False
) )
...@@ -1306,7 +1288,7 @@ class TestScan: ...@@ -1306,7 +1288,7 @@ class TestScan:
return OrderedDict([(state, 2 * state)]) return OrderedDict([(state, 2 * state)])
n_steps = theano.tensor.iscalar("nstep") n_steps = theano.tensor.iscalar("nstep")
output, updates = theano.scan( output, updates = scan(
f_2, [], [], [], n_steps=n_steps, truncate_gradient=-1, go_backwards=False f_2, [], [], [], n_steps=n_steps, truncate_gradient=-1, go_backwards=False
) )
this_f = theano.function( this_f = theano.function(
...@@ -1323,7 +1305,7 @@ class TestScan: ...@@ -1323,7 +1305,7 @@ class TestScan:
u = theano.tensor.vector("u") u = theano.tensor.vector("u")
outputs, updates = theano.scan( outputs, updates = scan(
f_rnn, u, [], [], n_steps=None, truncate_gradient=-1, go_backwards=False f_rnn, u, [], [], n_steps=None, truncate_gradient=-1, go_backwards=False
) )
...@@ -1360,7 +1342,7 @@ class TestScan: ...@@ -1360,7 +1342,7 @@ class TestScan:
W_in = theano.tensor.scalar("win") W_in = theano.tensor.scalar("win")
W = theano.tensor.scalar("w") W = theano.tensor.scalar("w")
output, updates = theano.scan( output, updates = scan(
f_rnn, f_rnn,
u, u,
x0, x0,
...@@ -1551,11 +1533,9 @@ class TestScan: ...@@ -1551,11 +1533,9 @@ class TestScan:
# assert in Scan.grad() of the new scan input sequence related # assert in Scan.grad() of the new scan input sequence related
# to outer_mitsot_outs, outer_sitsot_outs and # to outer_mitsot_outs, outer_sitsot_outs and
# outer_nitsot_outs. This allow to test an old Scan bug. # outer_nitsot_outs. This allow to test an old Scan bug.
old1 = theano.config.compute_test_value with theano.change_flags(
old2 = theano.config.compute_test_value_opt compute_test_value="raise", compute_test_value_opt="raise"
theano.config.compute_test_value = "raise" ):
theano.config.compute_test_value_opt = "raise"
try:
cost, updates = scan_project_sum( cost, updates = scan_project_sum(
f_rnn_cmpl, f_rnn_cmpl,
[u1, dict(input=u2, taps=[-1, 0, 1])], [u1, dict(input=u2, taps=[-1, 0, 1])],
...@@ -1567,7 +1547,6 @@ class TestScan: ...@@ -1567,7 +1547,6 @@ class TestScan:
) )
params = [u1, u2, x0, y0, W_in1] params = [u1, u2, x0, y0, W_in1]
gparams = theano.tensor.grad(cost, params) gparams = theano.tensor.grad(cost, params)
print(".", file=sys.stderr)
# Test the output including names # Test the output including names
output_str = theano.printing.debugprint(cost, file="str") output_str = theano.printing.debugprint(cost, file="str")
lines = output_str.split("\n") lines = output_str.split("\n")
...@@ -1758,7 +1737,6 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -1758,7 +1737,6 @@ for{cpu,scan_fn}.2 [id H] ''
no_default_updates=True, no_default_updates=True,
allow_input_downcast=True, allow_input_downcast=True,
) )
print(".", file=sys.stderr)
grad_fn = theano.function( grad_fn = theano.function(
[u1, u2, x0, y0, W_in1], [u1, u2, x0, y0, W_in1],
gparams, gparams,
...@@ -1766,10 +1744,6 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -1766,10 +1744,6 @@ for{cpu,scan_fn}.2 [id H] ''
no_default_updates=True, no_default_updates=True,
allow_input_downcast=True, allow_input_downcast=True,
) )
print(".", file=sys.stderr)
finally:
theano.config.compute_test_value = old1
theano.config.compute_test_value_opt = old2
num_grad = multiple_outputs_numeric_grad( num_grad = multiple_outputs_numeric_grad(
cost_fn, [v_u1, v_u2, v_x0, v_y0, vW_in1] cost_fn, [v_u1, v_u2, v_x0, v_y0, vW_in1]
...@@ -2041,7 +2015,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2041,7 +2015,7 @@ for{cpu,scan_fn}.2 [id H] ''
return [h_t, y_t] return [h_t, y_t]
# hidden and outputs of the entire sequence # hidden and outputs of the entire sequence
[h, y], _ = theano.scan( [h, y], _ = scan(
fn=one_step, fn=one_step,
sequences=dict(input=x), sequences=dict(input=x),
# corresponds to the return type of one_step # corresponds to the return type of one_step
...@@ -2085,7 +2059,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2085,7 +2059,7 @@ for{cpu,scan_fn}.2 [id H] ''
x = theano.tensor.matrix("x") x = theano.tensor.matrix("x")
y = trng.binomial(size=x.shape, p=x) y = trng.binomial(size=x.shape, p=x)
z, updates = theano.scan(lambda a: a, non_sequences=y, n_steps=2) z, updates = scan(lambda a: a, non_sequences=y, n_steps=2)
f = theano.function([x], [y, z], updates=updates, allow_input_downcast=True) f = theano.function([x], [y, z], updates=updates, allow_input_downcast=True)
...@@ -2102,7 +2076,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2102,7 +2076,7 @@ for{cpu,scan_fn}.2 [id H] ''
x1 = theano.shared(3.0) x1 = theano.shared(3.0)
x1.name = "x1" x1.name = "x1"
x2 = theano.tensor.vector("x2") x2 = theano.tensor.vector("x2")
y, updates = theano.scan( y, updates = scan(
lambda v: theano.tensor.cast(v * x1, theano.config.floatX), sequences=x2 lambda v: theano.tensor.cast(v * x1, theano.config.floatX), sequences=x2
) )
m = theano.tensor.grad(y.sum(), x1) m = theano.tensor.grad(y.sum(), x1)
...@@ -2115,7 +2089,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2115,7 +2089,7 @@ for{cpu,scan_fn}.2 [id H] ''
x2 = theano.shared(np.array([1, 2, 3, 4, 5]), name="x2") x2 = theano.shared(np.array([1, 2, 3, 4, 5]), name="x2")
K = x2 * x1 K = x2 * x1
out, updates = theano.scan( out, updates = scan(
lambda i, v: theano.tensor.grad(K[i], v), lambda i, v: theano.tensor.grad(K[i], v),
sequences=theano.tensor.arange(K.shape[0]), sequences=theano.tensor.arange(K.shape[0]),
non_sequences=x1, non_sequences=x1,
...@@ -2127,7 +2101,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2127,7 +2101,7 @@ for{cpu,scan_fn}.2 [id H] ''
def test_shared_updates(self): def test_shared_updates(self):
X = theano.shared(np.array(1)) X = theano.shared(np.array(1))
out, updates = theano.scan( out, updates = scan(
lambda: OrderedDict([(X, (X + 1))]), lambda: OrderedDict([(X, (X + 1))]),
outputs_info=[], outputs_info=[],
non_sequences=[], non_sequences=[],
...@@ -2143,7 +2117,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2143,7 +2117,7 @@ for{cpu,scan_fn}.2 [id H] ''
x = theano.shared(np.array(1)) x = theano.shared(np.array(1))
y = theano.shared(np.array(1)) y = theano.shared(np.array(1))
out, updates = theano.scan( out, updates = scan(
lambda: OrderedDict([(x, x + 1), (y, x)]), lambda: OrderedDict([(x, x + 1), (y, x)]),
outputs_info=[], outputs_info=[],
non_sequences=[], non_sequences=[],
...@@ -2175,14 +2149,14 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2175,14 +2149,14 @@ for{cpu,scan_fn}.2 [id H] ''
def inner_func(a): def inner_func(a):
return a + 1, OrderedDict([(b, 2 * b)]) return a + 1, OrderedDict([(b, 2 * b)])
out, updates = theano.scan( out, updates = scan(
inner_func, outputs_info=[OrderedDict([("initial", init_a)])], n_steps=1 inner_func, outputs_info=[OrderedDict([("initial", init_a)])], n_steps=1
) )
out = out[-1] out = out[-1]
assert out.type.ndim == a.type.ndim assert out.type.ndim == a.type.ndim
assert updates[b].type.ndim == b.type.ndim assert updates[b].type.ndim == b.type.ndim
out, updates = theano.scan(inner_func, outputs_info=[init_a], n_steps=1) out, updates = scan(inner_func, outputs_info=[init_a], n_steps=1)
assert out.type.ndim == a.type.ndim + 1 assert out.type.ndim == a.type.ndim + 1
assert updates[b].type.ndim == b.type.ndim assert updates[b].type.ndim == b.type.ndim
...@@ -2195,7 +2169,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2195,7 +2169,7 @@ for{cpu,scan_fn}.2 [id H] ''
y.name = "y" y.name = "y"
gy = theano.tensor.grad(y, x) gy = theano.tensor.grad(y, x)
gy.name = "gy" gy.name = "gy"
hy, updates = theano.scan( hy, updates = scan(
lambda i, gy, x: theano.tensor.grad(gy[i] * fc2, x), lambda i, gy, x: theano.tensor.grad(gy[i] * fc2, x),
sequences=theano.tensor.arange(gy.shape[0]), sequences=theano.tensor.arange(gy.shape[0]),
non_sequences=[gy, x], non_sequences=[gy, x],
...@@ -2342,7 +2316,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2342,7 +2316,7 @@ for{cpu,scan_fn}.2 [id H] ''
y_tm1 + theano.tensor.dot(x_tm1, W_out), y_tm1 + theano.tensor.dot(x_tm1, W_out),
] ]
outputs, updates = theano.scan( outputs, updates = scan(
f_rnn_cmpl, f_rnn_cmpl,
[u1, u2], [u1, u2],
[None, None, x0, dict(initial=y0, taps=[-1, -3])], [None, None, x0, dict(initial=y0, taps=[-1, -3])],
...@@ -2382,7 +2356,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2382,7 +2356,7 @@ for{cpu,scan_fn}.2 [id H] ''
def scanStep(prev, seq, f1): def scanStep(prev, seq, f1):
return prev + f1 * seq return prev + f1 * seq
scanned, _ = theano.scan( scanned, _ = scan(
fn=scanStep, sequences=[seq], outputs_info=[to_scan], non_sequences=[f1] fn=scanStep, sequences=[seq], outputs_info=[to_scan], non_sequences=[f1]
) )
theano.function( theano.function(
...@@ -2425,7 +2399,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2425,7 +2399,7 @@ for{cpu,scan_fn}.2 [id H] ''
y_tm1 + theano.tensor.dot(x_tm1, W_out), y_tm1 + theano.tensor.dot(x_tm1, W_out),
] ]
_outputs, updates = theano.scan( _outputs, updates = scan(
f_rnn_cmpl, f_rnn_cmpl,
[u1, u2], [u1, u2],
[None, dict(initial=x0), dict(initial=y0, taps=[-1, -3])], [None, dict(initial=x0), dict(initial=y0, taps=[-1, -3])],
...@@ -2463,7 +2437,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2463,7 +2437,7 @@ for{cpu,scan_fn}.2 [id H] ''
expr = tensor.dot(h_tm1, W) + x_t expr = tensor.dot(h_tm1, W) + x_t
return expr return expr
expr, _ = theano.scan( expr, _ = scan(
fn=one_step, sequences=[inpt], outputs_info=[initial], non_sequences=[W] fn=one_step, sequences=[inpt], outputs_info=[initial], non_sequences=[W]
) )
...@@ -2503,7 +2477,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2503,7 +2477,7 @@ for{cpu,scan_fn}.2 [id H] ''
u = theano.tensor.vector("u") u = theano.tensor.vector("u")
idx = theano.tensor.iscalar("idx") idx = theano.tensor.iscalar("idx")
jdx = theano.tensor.iscalar("jdx") jdx = theano.tensor.iscalar("jdx")
[x1, x2, x3, x4, x5, x6, x7], updates = theano.scan( [x1, x2, x3, x4, x5, x6, x7], updates = scan(
f_rnn, u, n_steps=None, truncate_gradient=-1, go_backwards=False f_rnn, u, n_steps=None, truncate_gradient=-1, go_backwards=False
) )
...@@ -2548,7 +2522,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2548,7 +2522,7 @@ for{cpu,scan_fn}.2 [id H] ''
x20 = theano.tensor.scalar("x20") x20 = theano.tensor.scalar("x20")
x30 = theano.tensor.vector("x30") x30 = theano.tensor.vector("x30")
x40 = theano.tensor.scalar("x40") x40 = theano.tensor.scalar("x40")
[x1, x2, x3, x4, x5, x6, x7], updates = theano.scan( [x1, x2, x3, x4, x5, x6, x7], updates = scan(
f_rnn, f_rnn,
u, u,
[ [
...@@ -2600,7 +2574,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2600,7 +2574,7 @@ for{cpu,scan_fn}.2 [id H] ''
x = tensor.scalar() x = tensor.scalar()
seq = tensor.vector() seq = tensor.vector()
outputs_info = [x, tensor.zeros_like(x)] outputs_info = [x, tensor.zeros_like(x)]
(out1, out2), updates = theano.scan( (out1, out2), updates = scan(
lambda a, b, c: (a + b, b + c), lambda a, b, c: (a + b, b + c),
sequences=seq, sequences=seq,
outputs_info=outputs_info, outputs_info=outputs_info,
...@@ -2639,7 +2613,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2639,7 +2613,7 @@ for{cpu,scan_fn}.2 [id H] ''
x = tensor.dcol() x = tensor.dcol()
seq = tensor.dcol() seq = tensor.dcol()
outputs_info = [x, tensor.zeros_like(x)] outputs_info = [x, tensor.zeros_like(x)]
(out1, out2), updates = theano.scan( (out1, out2), updates = scan(
lambda a, b, c: (a + b, a + c), sequences=seq, outputs_info=outputs_info lambda a, b, c: (a + b, a + c), sequences=seq, outputs_info=outputs_info
) )
...@@ -2676,10 +2650,10 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2676,10 +2650,10 @@ for{cpu,scan_fn}.2 [id H] ''
def inner_fct(previous_val): def inner_fct(previous_val):
new_val = previous_val + srng.uniform() new_val = previous_val + srng.uniform()
condition = theano.scan_module.until(previous_val > 5) condition = until(previous_val > 5)
return new_val, condition return new_val, condition
out, updates = theano.scan(inner_fct, outputs_info=x, n_steps=10) out, updates = scan(inner_fct, outputs_info=x, n_steps=10)
g_out = tensor.grad(out.sum(), x) g_out = tensor.grad(out.sum(), x)
fct = theano.function([x], [out, g_out]) fct = theano.function([x], [out, g_out])
...@@ -2711,7 +2685,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2711,7 +2685,7 @@ for{cpu,scan_fn}.2 [id H] ''
) )
return next_sitsot_val, next_mitsot_val, nitsot_out return next_sitsot_val, next_mitsot_val, nitsot_out
out, updates = theano.scan( out, updates = scan(
fn=step, fn=step,
sequences=seq, sequences=seq,
outputs_info=[ outputs_info=[
...@@ -2744,7 +2718,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2744,7 +2718,7 @@ for{cpu,scan_fn}.2 [id H] ''
m + trng.uniform(size=[3]), m + trng.uniform(size=[3]),
] ]
[o1, o2], updates = theano.scan( [o1, o2], updates = scan(
lm, lm,
sequences=x, sequences=x,
n_steps=None, n_steps=None,
...@@ -2760,9 +2734,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2760,9 +2734,7 @@ for{cpu,scan_fn}.2 [id H] ''
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
# this new assert is here to test if scan_merging works .. # this new assert is here to test if scan_merging works ..
nb_scan = len( nb_scan = len([n for n in topo if isinstance(n.op, Scan)])
[n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)]
)
assert nb_scan == 1 assert nb_scan == 1
nb_shape_i = len( nb_shape_i = len(
[n for n in topo if isinstance(n.op, theano.tensor.opt.Shape_i)] [n for n in topo if isinstance(n.op, theano.tensor.opt.Shape_i)]
...@@ -2777,64 +2749,64 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2777,64 +2749,64 @@ for{cpu,scan_fn}.2 [id H] ''
def sum(s): def sum(s):
return s + 1 return s + 1
sx, upx = theano.scan(sum, sequences=[x]) sx, upx = scan(sum, sequences=[x])
sy, upy = theano.scan(sum, sequences=[y]) sy, upy = scan(sum, sequences=[y])
f = theano.function( f = theano.function(
[x, y], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops") [x, y], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops")
) )
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
scans = [n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)] scans = [n for n in topo if isinstance(n.op, Scan)]
assert len(scans) == 2 assert len(scans) == 2
sx, upx = theano.scan(sum, sequences=[x], n_steps=2) sx, upx = scan(sum, sequences=[x], n_steps=2)
sy, upy = theano.scan(sum, sequences=[y], n_steps=3) sy, upy = scan(sum, sequences=[y], n_steps=3)
f = theano.function( f = theano.function(
[x, y], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops") [x, y], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops")
) )
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
scans = [n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)] scans = [n for n in topo if isinstance(n.op, Scan)]
assert len(scans) == 2 assert len(scans) == 2
sx, upx = theano.scan(sum, sequences=[x], n_steps=4) sx, upx = scan(sum, sequences=[x], n_steps=4)
sy, upy = theano.scan(sum, sequences=[y], n_steps=4) sy, upy = scan(sum, sequences=[y], n_steps=4)
f = theano.function( f = theano.function(
[x, y], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops") [x, y], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops")
) )
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
scans = [n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)] scans = [n for n in topo if isinstance(n.op, Scan)]
assert len(scans) == 1 assert len(scans) == 1
sx, upx = theano.scan(sum, sequences=[x]) sx, upx = scan(sum, sequences=[x])
sy, upy = theano.scan(sum, sequences=[x]) sy, upy = scan(sum, sequences=[x])
f = theano.function( f = theano.function(
[x], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops") [x], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops")
) )
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
scans = [n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)] scans = [n for n in topo if isinstance(n.op, Scan)]
assert len(scans) == 1 assert len(scans) == 1
sx, upx = theano.scan(sum, sequences=[x]) sx, upx = scan(sum, sequences=[x])
sy, upy = theano.scan(sum, sequences=[x], mode="FAST_COMPILE") sy, upy = scan(sum, sequences=[x], mode="FAST_COMPILE")
f = theano.function( f = theano.function(
[x], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops") [x], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops")
) )
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
scans = [n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)] scans = [n for n in topo if isinstance(n.op, Scan)]
assert len(scans) == 1 assert len(scans) == 1
sx, upx = theano.scan(sum, sequences=[x]) sx, upx = scan(sum, sequences=[x])
sy, upy = theano.scan(sum, sequences=[x], truncate_gradient=1) sy, upy = scan(sum, sequences=[x], truncate_gradient=1)
f = theano.function( f = theano.function(
[x], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops") [x], [sx, sy], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops")
) )
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
scans = [n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)] scans = [n for n in topo if isinstance(n.op, Scan)]
assert len(scans) == 2 assert len(scans) == 2
def test_merge_3scans(self): def test_merge_3scans(self):
...@@ -2847,17 +2819,17 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2847,17 +2819,17 @@ for{cpu,scan_fn}.2 [id H] ''
def sum(s): def sum(s):
return s + 1 return s + 1
sx, upx = theano.scan(sum, sequences=[x], n_steps=4, name="X") sx, upx = scan(sum, sequences=[x], n_steps=4, name="X")
# We need to use an expression of y rather than y so the toposort # We need to use an expression of y rather than y so the toposort
# comes up with the 'Y' scan last. # comes up with the 'Y' scan last.
sy, upy = theano.scan(sum, sequences=[2 * y + 2], n_steps=4, name="Y") sy, upy = scan(sum, sequences=[2 * y + 2], n_steps=4, name="Y")
sz, upz = theano.scan(sum, sequences=[sx], n_steps=4, name="Z") sz, upz = scan(sum, sequences=[sx], n_steps=4, name="Z")
f = theano.function( f = theano.function(
[x, y], [sy, sz], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops") [x, y], [sy, sz], mode=mode_with_opt.excluding("scanOp_pushout_seqs_ops")
) )
topo = f.maker.fgraph.toposort() topo = f.maker.fgraph.toposort()
scans = [n for n in topo if isinstance(n.op, theano.scan_module.scan_op.Scan)] scans = [n for n in topo if isinstance(n.op, Scan)]
assert len(scans) == 2 assert len(scans) == 2
rng = np.random.RandomState(utt.fetch_seed()) rng = np.random.RandomState(utt.fetch_seed())
...@@ -2875,7 +2847,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2875,7 +2847,7 @@ for{cpu,scan_fn}.2 [id H] ''
K = XX + XX.T K = XX + XX.T
return K.sum() return K.sum()
beta, K_updts = theano.scan( beta, K_updts = scan(
init_K, sequences=tensor.arange(E), non_sequences=[inputs, targets] init_K, sequences=tensor.arange(E), non_sequences=[inputs, targets]
) )
...@@ -2895,7 +2867,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2895,7 +2867,7 @@ for{cpu,scan_fn}.2 [id H] ''
Mi = tensor.sum(lb) * h[i, D] Mi = tensor.sum(lb) * h[i, D]
return Mi return Mi
(M), M_updts = theano.scan( (M), M_updts = scan(
predict_mean_i, predict_mean_i,
sequences=tensor.arange(E), sequences=tensor.arange(E),
non_sequences=[x_star, s_star, inputs, beta, hyp], non_sequences=[x_star, s_star, inputs, beta, hyp],
...@@ -2931,7 +2903,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2931,7 +2903,7 @@ for{cpu,scan_fn}.2 [id H] ''
expected_output = dfdm(X, Y, test_m, test_s) expected_output = dfdm(X, Y, test_m, test_s)
# equivalent code for the jacobian using scan # equivalent code for the jacobian using scan
dMdm, dMdm_updts = theano.scan( dMdm, dMdm_updts = scan(
lambda i, M, x: tensor.grad(M[i], x), lambda i, M, x: tensor.grad(M[i], x),
sequences=tensor.arange(M.shape[0]), sequences=tensor.arange(M.shape[0]),
non_sequences=[M, x_star], non_sequences=[M, x_star],
...@@ -2957,7 +2929,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2957,7 +2929,7 @@ for{cpu,scan_fn}.2 [id H] ''
# theano-user mailing list where the optimization raised an exception # theano-user mailing list where the optimization raised an exception
# when applied on this graph. # when applied on this graph.
x = tensor.matrix() x = tensor.matrix()
outputs, updates = theano.scan( outputs, updates = scan(
lambda x: [x * x, tensor.constant(0).copy().copy()], lambda x: [x * x, tensor.constant(0).copy().copy()],
n_steps=2, n_steps=2,
sequences=[], sequences=[],
...@@ -2977,7 +2949,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2977,7 +2949,7 @@ for{cpu,scan_fn}.2 [id H] ''
# one of those outputs. This led the optimization to raise an # one of those outputs. This led the optimization to raise an
# exception. # exception.
outputs, _ = theano.scan(lambda x: (x * x, x), non_sequences=[2], n_steps=2) outputs, _ = scan(lambda x: (x * x, x), non_sequences=[2], n_steps=2)
f = theano.function(inputs=[], outputs=outputs) f = theano.function(inputs=[], outputs=outputs)
outs = f() outs = f()
...@@ -2991,14 +2963,14 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -2991,14 +2963,14 @@ for{cpu,scan_fn}.2 [id H] ''
return s + 1 return s + 1
x = theano.tensor.vector() x = theano.tensor.vector()
sx, upx = theano.scan(fn=incr, sequences=[{"input": x}]) sx, upx = scan(fn=incr, sequences=[{"input": x}])
theano.function([x], sx) theano.function([x], sx)
def test_hash(self): def test_hash(self):
x = theano.tensor.vector() x = theano.tensor.vector()
y = theano.tensor.vector() y = theano.tensor.vector()
scan1, updates = theano.scan(lambda _x: _x + 1, x) scan1, updates = scan(lambda _x: _x + 1, x)
scan2, updates = theano.scan(lambda _x: _x + 1, y) scan2, updates = scan(lambda _x: _x + 1, y)
assert scan1.owner.op == scan2.owner.op assert scan1.owner.op == scan2.owner.op
assert hash(scan1.owner.op) == hash(scan2.owner.op) assert hash(scan1.owner.op) == hash(scan2.owner.op)
...@@ -3017,7 +2989,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3017,7 +2989,7 @@ for{cpu,scan_fn}.2 [id H] ''
d = theano.tensor.dot(i, W) d = theano.tensor.dot(i, W)
return d, d return d, d
outs, updts = theano.scan( outs, updts = scan(
f, sequences=[x], non_sequences=[], outputs_info=[None, memory] f, sequences=[x], non_sequences=[], outputs_info=[None, memory]
) )
...@@ -3167,7 +3139,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3167,7 +3139,7 @@ for{cpu,scan_fn}.2 [id H] ''
_W = theano.tensor.specify_shape(W, v_W.shape) _W = theano.tensor.specify_shape(W, v_W.shape)
_W.name = "_W" _W.name = "_W"
[o, _], _ = theano.scan( [o, _], _ = scan(
rnn_fn, rnn_fn,
sequences=_u, sequences=_u,
outputs_info=[_h0, None], outputs_info=[_h0, None],
...@@ -3187,21 +3159,21 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3187,21 +3159,21 @@ for{cpu,scan_fn}.2 [id H] ''
) )
vnu, vnh0, vnW, vno = fn_rop(v_u, v_h0, v_W, v_eu, v_eh0, v_eW) vnu, vnh0, vnW, vno = fn_rop(v_u, v_h0, v_W, v_eu, v_eh0, v_eW)
n2o_u, _ = theano.scan( n2o_u, _ = scan(
lambda i, o, u, h0, W, eu: (theano.tensor.grad(o[i], u) * eu).sum(), lambda i, o, u, h0, W, eu: (theano.tensor.grad(o[i], u) * eu).sum(),
sequences=tensor.arange(o.shape[0]), sequences=tensor.arange(o.shape[0]),
non_sequences=[o, u, h0, W, eu], non_sequences=[o, u, h0, W, eu],
name="jacobU", name="jacobU",
) )
n2o_h0, _ = theano.scan( n2o_h0, _ = scan(
lambda i, o, u, h0, W, eh0: (theano.tensor.grad(o[i], h0) * eh0).sum(), lambda i, o, u, h0, W, eh0: (theano.tensor.grad(o[i], h0) * eh0).sum(),
sequences=tensor.arange(o.shape[0]), sequences=tensor.arange(o.shape[0]),
non_sequences=[o, u, h0, W, eh0], non_sequences=[o, u, h0, W, eh0],
name="jacobh", name="jacobh",
) )
n2o_W, _ = theano.scan( n2o_W, _ = scan(
lambda i, o, u, h0, W, eW: (theano.tensor.grad(o[i], W) * eW).sum(), lambda i, o, u, h0, W, eW: (theano.tensor.grad(o[i], W) * eW).sum(),
sequences=tensor.arange(o.shape[0]), sequences=tensor.arange(o.shape[0]),
non_sequences=[o, u, h0, W, eW], non_sequences=[o, u, h0, W, eW],
...@@ -3244,7 +3216,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3244,7 +3216,7 @@ for{cpu,scan_fn}.2 [id H] ''
_W = theano.tensor.specify_shape(W, v_W.shape) _W = theano.tensor.specify_shape(W, v_W.shape)
_W.name = "_W" _W.name = "_W"
o, _ = theano.scan( o, _ = scan(
rnn_fn, sequences=_u, outputs_info=_h0, non_sequences=_W, name="rnn_fn" rnn_fn, sequences=_u, outputs_info=_h0, non_sequences=_W, name="rnn_fn"
) )
o = o[-1] o = o[-1]
...@@ -3259,21 +3231,21 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3259,21 +3231,21 @@ for{cpu,scan_fn}.2 [id H] ''
[u, h0, W, eu, eh0, eW], [nwo_u, nwo_h0, nwo_W], on_unused_input="ignore" [u, h0, W, eu, eh0, eW], [nwo_u, nwo_h0, nwo_W], on_unused_input="ignore"
) )
n2o_u, _ = theano.scan( n2o_u, _ = scan(
lambda i, o, u, h0, W, eu: (theano.tensor.grad(o[i], u) * eu).sum(), lambda i, o, u, h0, W, eu: (theano.tensor.grad(o[i], u) * eu).sum(),
sequences=tensor.arange(o.shape[0]), sequences=tensor.arange(o.shape[0]),
non_sequences=[o, u, h0, W, eu], non_sequences=[o, u, h0, W, eu],
name="jacobU", name="jacobU",
) )
n2o_h0, _ = theano.scan( n2o_h0, _ = scan(
lambda i, o, u, h0, W, eh0: (theano.tensor.grad(o[i], h0) * eh0).sum(), lambda i, o, u, h0, W, eh0: (theano.tensor.grad(o[i], h0) * eh0).sum(),
sequences=tensor.arange(o.shape[0]), sequences=tensor.arange(o.shape[0]),
non_sequences=[o, u, h0, W, eh0], non_sequences=[o, u, h0, W, eh0],
name="jacobh", name="jacobh",
) )
n2o_W, _ = theano.scan( n2o_W, _ = scan(
lambda i, o, u, h0, W, eW: (theano.tensor.grad(o[i], W) * eW).sum(), lambda i, o, u, h0, W, eW: (theano.tensor.grad(o[i], W) * eW).sum(),
sequences=tensor.arange(o.shape[0]), sequences=tensor.arange(o.shape[0]),
non_sequences=[o, u, h0, W, eW], non_sequences=[o, u, h0, W, eW],
...@@ -3295,7 +3267,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3295,7 +3267,7 @@ for{cpu,scan_fn}.2 [id H] ''
W = tensor.matrix("W") W = tensor.matrix("W")
h = tensor.matrix("h") h = tensor.matrix("h")
o, _ = theano.scan( o, _ = scan(
lambda hi, him1, W: (hi, tensor.dot(hi + him1, W)), lambda hi, him1, W: (hi, tensor.dot(hi + him1, W)),
outputs_info=[tensor.zeros([h.shape[1]]), None], outputs_info=[tensor.zeros([h.shape[1]]), None],
sequences=[h], sequences=[h],
...@@ -3304,11 +3276,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3304,11 +3276,7 @@ for{cpu,scan_fn}.2 [id H] ''
f = theano.function([W, h], o, mode=mode_with_opt) f = theano.function([W, h], o, mode=mode_with_opt)
scan_nodes = [ scan_nodes = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_nodes) == 1 assert len(scan_nodes) == 1
scan_op = scan_nodes[0].op scan_op = scan_nodes[0].op
assert not any( assert not any(
...@@ -3323,15 +3291,11 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3323,15 +3291,11 @@ for{cpu,scan_fn}.2 [id H] ''
def lambda_fn(h, W1, W2): def lambda_fn(h, W1, W2):
return tensor.dot(h, W1 + W2) return tensor.dot(h, W1 + W2)
o, _ = theano.scan(lambda_fn, non_sequences=[h0, W1, W2], n_steps=5) o, _ = scan(lambda_fn, non_sequences=[h0, W1, W2], n_steps=5)
f = theano.function([h0, W1, W2], o, mode=mode_with_opt) f = theano.function([h0, W1, W2], o, mode=mode_with_opt)
scan_nodes = [ scan_nodes = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_nodes) == 0 assert len(scan_nodes) == 0
seed = utt.fetch_seed() seed = utt.fetch_seed()
...@@ -3358,18 +3322,18 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3358,18 +3322,18 @@ for{cpu,scan_fn}.2 [id H] ''
step_indices = tensor.vector("step_indices") step_indices = tensor.vector("step_indices")
def lambda_fn(step_idx, W1, W2): def lambda_fn(step_idx, W1, W2):
until_condition = theano.scan_module.until(step_idx > 2) until_condition = until(step_idx > 2)
return tensor.dot(W1, W2), until_condition return tensor.dot(W1, W2), until_condition
# Compile a function with the optimization # Compile a function with the optimization
o, _ = theano.scan( o, _ = scan(
lambda_fn, sequences=[step_indices, W1], non_sequences=[W2], n_steps=5 lambda_fn, sequences=[step_indices, W1], non_sequences=[W2], n_steps=5
) )
f = theano.function([W1, W2, step_indices], o, mode=mode_with_opt) f = theano.function([W1, W2, step_indices], o, mode=mode_with_opt)
# Compule an theano function without the optimization # Compule an theano function without the optimization
o, _ = theano.scan( o, _ = scan(
lambda_fn, lambda_fn,
sequences=[step_indices, W1], sequences=[step_indices, W1],
non_sequences=[W2], non_sequences=[W2],
...@@ -3398,17 +3362,11 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3398,17 +3362,11 @@ for{cpu,scan_fn}.2 [id H] ''
def lambda_fn(h, W1, W2): def lambda_fn(h, W1, W2):
return tensor.dot(h, W1 + W2) return tensor.dot(h, W1 + W2)
o, _ = theano.scan( o, _ = scan(lambda_fn, outputs_info=h0, non_sequences=[W1, W2], n_steps=5)
lambda_fn, outputs_info=h0, non_sequences=[W1, W2], n_steps=5
)
f = theano.function([h0, W1, W2], o, mode=mode_with_opt) f = theano.function([h0, W1, W2], o, mode=mode_with_opt)
scan_node = [ scan_node = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)][0]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
][0]
assert ( assert (
len( len(
[ [
...@@ -3426,7 +3384,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3426,7 +3384,7 @@ for{cpu,scan_fn}.2 [id H] ''
def fn(i, i_tm1): def fn(i, i_tm1):
return i + 10, i_tm1 return i + 10, i_tm1
([i_t, i_tm1], _) = theano.scan( ([i_t, i_tm1], _) = scan(
fn, fn,
sequences=[inp], sequences=[inp],
outputs_info=[np.asarray([0.0, 0.0], theano.config.floatX), None], outputs_info=[np.asarray([0.0, 0.0], theano.config.floatX), None],
...@@ -3447,7 +3405,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3447,7 +3405,7 @@ for{cpu,scan_fn}.2 [id H] ''
def lambda_fn(h, W1, W2): def lambda_fn(h, W1, W2):
return tensor.dot(h, W1 * W2) return tensor.dot(h, W1 * W2)
o, _ = theano.scan( o, _ = scan(
lambda_fn, lambda_fn,
outputs_info=h0, outputs_info=h0,
non_sequences=[W1, tensor.zeros_like(W2)], non_sequences=[W1, tensor.zeros_like(W2)],
...@@ -3455,11 +3413,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3455,11 +3413,7 @@ for{cpu,scan_fn}.2 [id H] ''
) )
f = theano.function([h0, W1, W2], o, mode=mode_with_opt) f = theano.function([h0, W1, W2], o, mode=mode_with_opt)
scan_node = [ scan_node = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)][0]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
][0]
assert ( assert (
len( len(
[ [
...@@ -3483,7 +3437,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3483,7 +3437,7 @@ for{cpu,scan_fn}.2 [id H] ''
def lambda_fn(W1, h, W2): def lambda_fn(W1, h, W2):
return W1 * tensor.dot(h, W2) return W1 * tensor.dot(h, W2)
o, _ = theano.scan( o, _ = scan(
lambda_fn, lambda_fn,
sequences=tensor.zeros_like(W1), sequences=tensor.zeros_like(W1),
outputs_info=h0, outputs_info=h0,
...@@ -3492,11 +3446,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3492,11 +3446,7 @@ for{cpu,scan_fn}.2 [id H] ''
) )
f = theano.function([h0, W1, W2], o, mode=mode_with_opt) f = theano.function([h0, W1, W2], o, mode=mode_with_opt)
scan_node = [ scan_node = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)][0]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
][0]
assert ( assert (
len( len(
...@@ -3521,7 +3471,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3521,7 +3471,7 @@ for{cpu,scan_fn}.2 [id H] ''
def lambda_fn(W1, h, W2): def lambda_fn(W1, h, W2):
return W1 * tensor.dot(h, W2) return W1 * tensor.dot(h, W2)
o, _ = theano.scan( o, _ = scan(
lambda_fn, lambda_fn,
sequences=tensor.zeros_like(W1), sequences=tensor.zeros_like(W1),
outputs_info=h0, outputs_info=h0,
...@@ -3530,11 +3480,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3530,11 +3480,7 @@ for{cpu,scan_fn}.2 [id H] ''
) )
f = theano.function([_h0, _W1, _W2], o, mode=mode_with_opt) f = theano.function([_h0, _W1, _W2], o, mode=mode_with_opt)
scan_node = [ scan_node = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)][0]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
][0]
assert len(scan_node.op.inputs) == 1 assert len(scan_node.op.inputs) == 1
...@@ -3542,7 +3488,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3542,7 +3488,7 @@ for{cpu,scan_fn}.2 [id H] ''
x = tensor.vector("x") x = tensor.vector("x")
def lambda_fn(x_t): def lambda_fn(x_t):
return x_t + 1, theano.scan_module.until(x_t > 3) return x_t + 1, until(x_t > 3)
o, _ = theano.scan(lambda_fn, x) o, _ = theano.scan(lambda_fn, x)
f = theano.function([x], o) f = theano.function([x], o)
...@@ -3555,7 +3501,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3555,7 +3501,7 @@ for{cpu,scan_fn}.2 [id H] ''
x = tensor.vector("x") x = tensor.vector("x")
def lambda_fn(x_t): def lambda_fn(x_t):
return x_t + 1, theano.scan_module.until(x_t > 3) return x_t + 1, until(x_t > 3)
o, _ = theano.scan(lambda_fn, x) o, _ = theano.scan(lambda_fn, x)
o2, _ = theano.scan(lambda x_t: x_t + 2, x) o2, _ = theano.scan(lambda x_t: x_t + 2, x)
...@@ -3566,11 +3512,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3566,11 +3512,7 @@ for{cpu,scan_fn}.2 [id H] ''
out, out2 = f(vx) out, out2 = f(vx)
assert len(out) == 24 assert len(out) == 24
assert np.all(out2 == vx + 2) assert np.all(out2 == vx + 2)
lssc = [ lssc = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
# One scan node gets optimnized out # One scan node gets optimnized out
assert len(lssc) == 1 assert len(lssc) == 1
...@@ -3693,10 +3635,10 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3693,10 +3635,10 @@ for{cpu,scan_fn}.2 [id H] ''
x = tensor.vector("x") x = tensor.vector("x")
def lambda_fn(x_t): def lambda_fn(x_t):
return x_t + 1, theano.scan_module.until(x_t > 3) return x_t + 1, until(x_t > 3)
o, _ = theano.scan(lambda_fn, x) o, _ = theano.scan(lambda_fn, x)
o2, _ = theano.scan(lambda x_t: (x_t + 2, theano.scan_module.until(x_t > 3)), x) o2, _ = theano.scan(lambda x_t: (x_t + 2, until(x_t > 3)), x)
f = theano.function([x], [o, o2], mode=mode_with_opt) f = theano.function([x], [o, o2], mode=mode_with_opt)
vx = np.zeros((50,), dtype=theano.config.floatX) vx = np.zeros((50,), dtype=theano.config.floatX)
...@@ -3704,18 +3646,14 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3704,18 +3646,14 @@ for{cpu,scan_fn}.2 [id H] ''
out, out2 = f(vx) out, out2 = f(vx)
assert len(out) == 24 assert len(out) == 24
assert len(out2) == 24 assert len(out2) == 24
lssc = [ lssc = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert len(lssc) == 1 assert len(lssc) == 1
def test_while_infershape(self): def test_while_infershape(self):
x = tensor.vector("x") x = tensor.vector("x")
def lambda_fn(x_t): def lambda_fn(x_t):
return x_t + 1, theano.scan_module.until(x_t > 3) return x_t + 1, until(x_t > 3)
o, _ = theano.scan(lambda_fn, x) o, _ = theano.scan(lambda_fn, x)
...@@ -3745,11 +3683,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3745,11 +3683,7 @@ for{cpu,scan_fn}.2 [id H] ''
out1, out2 = f(vx) out1, out2 = f(vx)
assert out1 == 10 assert out1 == 10
assert out2 == 10 assert out2 == 10
lssc = [ lssc = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert len(lssc) == 0 assert len(lssc) == 0
def test_infershape_nsteps_smaller_seq_length(self): def test_infershape_nsteps_smaller_seq_length(self):
...@@ -3767,11 +3701,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -3767,11 +3701,7 @@ for{cpu,scan_fn}.2 [id H] ''
o1, o2 = f(vx) o1, o2 = f(vx)
assert o1 == 20 assert o1 == 20
assert o2 == 20 assert o2 == 20
lssc = [ lssc = [x for x in f.maker.fgraph.toposort() if isinstance(x.op, Scan)]
x
for x in f.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
assert len(lssc) == 0 assert len(lssc) == 0
def test_oinp_iinp_iout_oout_mappings(self): def test_oinp_iinp_iout_oout_mappings(self):
...@@ -4457,16 +4387,14 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -4457,16 +4387,14 @@ for{cpu,scan_fn}.2 [id H] ''
def test_savemem_does_not_duplicate_number_of_scan_nodes(self): def test_savemem_does_not_duplicate_number_of_scan_nodes(self):
var = tensor.ones(()) var = tensor.ones(())
values, _ = theano.scan( values, _ = theano.scan(
lambda x: ([x], (), theano.scan_module.until(x)), lambda x: ([x], (), until(x)),
outputs_info=[var], outputs_info=[var],
n_steps=2, n_steps=2,
) )
tmp_fn = theano.function([var], values) tmp_fn = theano.function([var], values)
scan_nodes = [ scan_nodes = [
x x for x in tmp_fn.maker.fgraph.toposort() if isinstance(x.op, Scan)
for x in tmp_fn.maker.fgraph.toposort()
if isinstance(x.op, theano.scan_module.scan_op.Scan)
] ]
assert len(scan_nodes) == 1 assert len(scan_nodes) == 1
...@@ -4591,16 +4519,16 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -4591,16 +4519,16 @@ for{cpu,scan_fn}.2 [id H] ''
) )
y2, _ = theano.scan( y2, _ = theano.scan(
lambda x, y: (x + y, theano.scan_module.until(x > 0)), lambda x, y: (x + y, until(x > 0)),
sequences=inps, sequences=inps,
outputs_info=state, outputs_info=state,
n_steps=5, n_steps=5,
) )
scan_node1 = y1.owner.inputs[0].owner scan_node1 = y1.owner.inputs[0].owner
assert isinstance(scan_node1.op, theano.scan_module.scan_op.Scan) assert isinstance(scan_node1.op, Scan)
scan_node2 = y2.owner.inputs[0].owner scan_node2 = y2.owner.inputs[0].owner
assert isinstance(scan_node2.op, theano.scan_module.scan_op.Scan) assert isinstance(scan_node2.op, Scan)
opt_obj = theano.scan_module.scan_opt.ScanMerge() opt_obj = ScanMerge()
# Test the method belongs_to of this class. Specifically see if it # Test the method belongs_to of this class. Specifically see if it
# detects the two scan_nodes as not being similar # detects the two scan_nodes as not being similar
assert not opt_obj.belongs_to_set(scan_node1, [scan_node2]) assert not opt_obj.belongs_to_set(scan_node1, [scan_node2])
...@@ -4790,12 +4718,12 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -4790,12 +4718,12 @@ for{cpu,scan_fn}.2 [id H] ''
def _scan_strict(x, w_ns): def _scan_strict(x, w_ns):
return tensor.dot(x, w_ns) return tensor.dot(x, w_ns)
ret_loose = theano.scan( ret_loose = scan(
_scan_loose, sequences=[], outputs_info=[x0_], n_steps=n, strict=False _scan_loose, sequences=[], outputs_info=[x0_], n_steps=n, strict=False
) )
f_loose = theano.function([x0_], ret_loose[0][-1]) f_loose = theano.function([x0_], ret_loose[0][-1])
ret_strict = theano.scan( ret_strict = scan(
_scan_strict, _scan_strict,
sequences=[], sequences=[],
outputs_info=[x0_], outputs_info=[x0_],
...@@ -4823,9 +4751,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -4823,9 +4751,7 @@ for{cpu,scan_fn}.2 [id H] ''
return tensor.dot(x, w_) return tensor.dot(x, w_)
with pytest.raises(theano.gof.fg.MissingInputError): with pytest.raises(theano.gof.fg.MissingInputError):
theano.scan( scan(_scan_loose, sequences=[], outputs_info=[x0_], n_steps=n, strict=True)
_scan_loose, sequences=[], outputs_info=[x0_], n_steps=n, strict=True
)
def test_monitor_mode(self): def test_monitor_mode(self):
# Test that it is possible to pass an instance of MonitorMode # Test that it is possible to pass an instance of MonitorMode
...@@ -4844,7 +4770,7 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -4844,7 +4770,7 @@ for{cpu,scan_fn}.2 [id H] ''
mode = theano.compile.MonitorMode(post_func=detect_large_outputs) mode = theano.compile.MonitorMode(post_func=detect_large_outputs)
# Symbolic description of the result # Symbolic description of the result
result, updates = theano.scan( result, updates = scan(
fn=lambda prior_result, A: prior_result * A, fn=lambda prior_result, A: prior_result * A,
outputs_info=tensor.ones_like(A), outputs_info=tensor.ones_like(A),
non_sequences=A, non_sequences=A,
...@@ -4868,450 +4794,6 @@ for{cpu,scan_fn}.2 [id H] '' ...@@ -4868,450 +4794,6 @@ for{cpu,scan_fn}.2 [id H] ''
assert detect_large_outputs.large_count == 3 assert detect_large_outputs.large_count == 3
class ScanGpuTests:
"""
This class defines a number of tests for Scan on GPU as well as a few
helper functions for these tests. The GPU tests defined in this class are
independent of the GPU backend used. Because of this, a class inheriting
from ScanGpuTests should define the following attributes and methods to
make the tests run on a specific backend :
- self.gpu_backend : Reference to the backend module
- self.mode_with_opt : Compilation mode to force usage of the gpu backend
- self.is_scan_on_gpu(node) : Method to determine is a scan node has been
moved to run on a gpu under the specific
backend. Returns a boolean.
"""
def test_one_sequence_one_output_weights_gpu1(self):
def f_rnn(u_t, x_tm1, W_in, W):
return u_t * W_in + x_tm1 * W
u = theano.tensor.fvector("u")
x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w")
# The following line is needed to have the first case being used
# Otherwise, it is the second that is tested.
mode = self.mode_with_gpu.excluding("InputToGpuOptimizer")
output, updates = theano.scan(
f_rnn,
u,
x0,
[W_in, W],
n_steps=None,
truncate_gradient=-1,
go_backwards=False,
mode=mode,
)
output = self.gpu_backend.gpu_from_host(output)
f2 = theano.function(
[u, x0, W_in, W],
output,
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# get random initial values
rng = np.random.RandomState(utt.fetch_seed())
v_u = rng.uniform(size=(4,), low=-5.0, high=5.0)
v_x0 = rng.uniform()
W = rng.uniform()
W_in = rng.uniform()
v_u = np.asarray(v_u, dtype="float32")
v_x0 = np.asarray(v_x0, dtype="float32")
W = np.asarray(W, dtype="float32")
W_in = np.asarray(W_in, dtype="float32")
# compute the output in numpy
v_out = np.zeros((4,))
v_out[0] = v_u[0] * W_in + v_x0 * W
for step in range(1, 4):
v_out[step] = v_u[step] * W_in + v_out[step - 1] * W
theano_values = f2(v_u, v_x0, W_in, W)
utt.assert_allclose(theano_values, v_out)
# TO DEL
topo = f2.maker.fgraph.toposort()
scan_node = [
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1
scan_node = scan_node[0]
topo = f2.maker.fgraph.toposort()
assert (
sum([isinstance(node.op, self.gpu_backend.HostFromGpu) for node in topo])
== 0
)
assert (
sum([isinstance(node.op, self.gpu_backend.GpuFromHost) for node in topo])
== 4
)
scan_node = [
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1
scan_node = scan_node[0]
scan_node_topo = scan_node.op.fn.maker.fgraph.toposort()
# check that there is no gpu transfer in the inner loop.
assert any(
[
isinstance(node.op, self.gpu_backend.GpuElemwise)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.HostFromGpu)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.GpuFromHost)
for node in scan_node_topo
]
)
# This second version test the second case in the optimizer to the gpu.
def test_one_sequence_one_output_weights_gpu2(self):
def f_rnn(u_t, x_tm1, W_in, W):
return u_t * W_in + x_tm1 * W
u = theano.tensor.fvector("u")
x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w")
output, updates = theano.scan(
f_rnn,
u,
x0,
[W_in, W],
n_steps=None,
truncate_gradient=-1,
go_backwards=False,
mode=self.mode_with_gpu,
)
f2 = theano.function(
[u, x0, W_in, W],
output,
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# get random initial values
rng = np.random.RandomState(utt.fetch_seed())
v_u = rng.uniform(size=(4,), low=-5.0, high=5.0)
v_x0 = rng.uniform()
W = rng.uniform()
W_in = rng.uniform()
# compute the output in numpy
v_out = np.zeros((4,))
v_out[0] = v_u[0] * W_in + v_x0 * W
for step in range(1, 4):
v_out[step] = v_u[step] * W_in + v_out[step - 1] * W
theano_values = f2(v_u, v_x0, W_in, W)
utt.assert_allclose(theano_values, v_out)
topo = f2.maker.fgraph.toposort()
assert (
sum([isinstance(node.op, self.gpu_backend.HostFromGpu) for node in topo])
== 1
)
assert (
sum([isinstance(node.op, self.gpu_backend.GpuFromHost) for node in topo])
== 4
)
scan_node = [
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1
scan_node = scan_node[0]
scan_node_topo = scan_node.op.fn.maker.fgraph.toposort()
# check that there is no gpu transfer in the inner loop.
assert any(
[
isinstance(node.op, self.gpu_backend.GpuElemwise)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.HostFromGpu)
for node in scan_node_topo
]
)
assert not any(
[
isinstance(node.op, self.gpu_backend.GpuFromHost)
for node in scan_node_topo
]
)
# This third test checks that scan can deal with a mixture of dtypes as
# outputs when is running on GPU
def test_gpu3_mixture_dtype_outputs(self):
def f_rnn(u_t, x_tm1, W_in, W):
return (u_t * W_in + x_tm1 * W, tensor.cast(u_t + x_tm1, "int64"))
u = theano.tensor.fvector("u")
x0 = theano.tensor.fscalar("x0")
W_in = theano.tensor.fscalar("win")
W = theano.tensor.fscalar("w")
output, updates = theano.scan(
f_rnn,
u,
[x0, None],
[W_in, W],
n_steps=None,
truncate_gradient=-1,
go_backwards=False,
mode=self.mode_with_gpu,
)
f2 = theano.function(
[u, x0, W_in, W],
output,
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# get random initial values
rng = np.random.RandomState(utt.fetch_seed())
v_u = rng.uniform(size=(4,), low=-5.0, high=5.0)
v_x0 = rng.uniform()
W = rng.uniform()
W_in = rng.uniform()
# compute the output in numpy
v_out1 = np.zeros((4,))
v_out2 = np.zeros((4,), dtype="int64")
v_out1[0] = v_u[0] * W_in + v_x0 * W
v_out2[0] = v_u[0] + v_x0
for step in range(1, 4):
v_out1[step] = v_u[step] * W_in + v_out1[step - 1] * W
v_out2[step] = np.int64(v_u[step] + v_out1[step - 1])
theano_out1, theano_out2 = f2(v_u, v_x0, W_in, W)
utt.assert_allclose(theano_out1, v_out1)
utt.assert_allclose(theano_out2, v_out2)
topo = f2.maker.fgraph.toposort()
scan_node = [
node
for node in topo
if isinstance(node.op, theano.scan_module.scan_op.Scan)
]
assert len(scan_node) == 1
scan_node = scan_node[0]
assert self.is_scan_on_gpu(scan_node)
def test_gibbs_chain(self):
rng = np.random.RandomState(utt.fetch_seed())
v_vsample = np.array(
rng.binomial(
1,
0.5,
size=(3, 20),
),
dtype="float32",
)
vsample = theano.shared(v_vsample)
trng = theano.sandbox.rng_mrg.MRG_RandomStreams(utt.fetch_seed())
def f(vsample_tm1):
return (
trng.binomial(vsample_tm1.shape, n=1, p=0.3, dtype="float32")
* vsample_tm1
)
theano_vsamples, updates = theano.scan(
f,
[],
vsample,
[],
n_steps=10,
truncate_gradient=-1,
go_backwards=False,
mode=self.mode_with_gpu,
)
my_f = theano.function(
[],
theano_vsamples[-1],
updates=updates,
allow_input_downcast=True,
mode=self.mode_with_gpu,
)
# I leave this to tested by debugmode, this test was anyway more of
# doest the graph compile kind of test
my_f()
def test_gpu_memory_usage(self):
# This test validates that the memory usage of the defined theano
# function is reasonnable when executed on the GPU. It checks for
# a bug in which one of scan's optimization was not applied which
# made the scan node compute large and unnecessary outputs which
# brought memory usage on the GPU to ~12G.
# Dimensionality of input and output data (not one-hot coded)
n_in = 100
n_out = 100
# Number of neurons in hidden layer
n_hid = 4000
# Number of minibatches
mb_size = 2
# Time steps in minibatch
mb_length = 200
# Define input variables
xin = tensor.ftensor3(name="xin")
yout = tensor.ftensor3(name="yout")
# Initialize the network parameters
U = theano.shared(np.zeros((n_in, n_hid), dtype="float32"), name="W_xin_to_l1")
V = theano.shared(np.zeros((n_hid, n_hid), dtype="float32"), name="W_l1_to_l1")
W = theano.shared(np.zeros((n_hid, n_out), dtype="float32"), name="W_l1_to_l2")
nparams = [U, V, W]
# Build the forward pass
l1_base = tensor.dot(xin, U)
def scan_l(baseline, last_step):
return baseline + tensor.dot(last_step, V)
zero_output = tensor.alloc(np.asarray(0.0, dtype="float32"), mb_size, n_hid)
l1_out, _ = theano.scan(
scan_l,
sequences=[l1_base],
outputs_info=[zero_output],
mode=self.mode_with_gpu_nodebug,
)
l2_out = tensor.dot(l1_out, W)
# Compute the cost and take the gradient wrt params
cost = tensor.sum((l2_out - yout) ** 2)
grads = tensor.grad(cost, nparams)
updates = list(zip(nparams, (n - g for n, g in zip(nparams, grads))))
# Compile the theano function
feval_backprop = theano.function(
[xin, yout], cost, updates=updates, mode=self.mode_with_gpu_nodebug
)
# Validate that the PushOutScanOutput optimization has been applied
# by checking the number of outputs of the grad Scan node in the
# compiled function.
nodes = feval_backprop.maker.fgraph.toposort()
scan_nodes = [
n for n in nodes if isinstance(n.op, theano.scan_module.scan_op.Scan)
]
# The grad scan is always the 2nd one according to toposort. If the
# optimization has been applied, it has 2 outputs, otherwise 3.
grad_scan_node = scan_nodes[1]
assert len(grad_scan_node.outputs) == 2, len(grad_scan_node.outputs)
# Call the theano function to ensure the absence of a memory error
feval_backprop(
np.zeros((mb_length, mb_size, n_in), dtype="float32"),
np.zeros((mb_length, mb_size, n_out), dtype="float32"),
)
def test_memory_reuse_gpudimshuffle(self):
# Test the memory pre-allocation feature in scan when one output is
# the result of a GpuDimshuffle (because an optimization in
# GpuDimshuffle can cause issues with the memory pre-allocation
# where it falsely thinks that a pre-allocated memory region has
# been used when it hasn't).
def inner_fn(seq1, recurrent_out):
temp = seq1 + recurrent_out.sum()
output1 = temp.dimshuffle(1, 0)
output2 = temp.sum() + recurrent_out
return output1, output2
input1 = theano.tensor.ftensor3()
init = theano.tensor.ftensor3()
outputs_info = [None, init]
out, _ = theano.scan(
inner_fn,
sequences=[input1],
outputs_info=outputs_info,
mode=self.mode_with_gpu,
)
out1 = out[0].flatten()
out2 = out[1].flatten()
fct = theano.function([input1, init], [out1, out2], mode=self.mode_with_gpu)
output = fct(
np.ones((2, 1, 1), dtype="float32"), np.ones((1, 1, 1), dtype="float32")
)
expected_output = (
np.array([2, 4], dtype="float32"),
np.array([3, 7], dtype="float32"),
)
utt.assert_allclose(output, expected_output)
class TestScanGpuarray(ScanGpuTests):
"""
This class takes the gpu tests for scan that are defined in
class ScanGpuTests and runs them using the gpuarray backend.
"""
def setup_method(self):
from theano import gpuarray
self.gpu_backend = gpuarray
# This is unfortunate, but required
def gpu_from_host(v):
return gpuarray.GpuFromHost(None)(v)
self.gpu_backend.gpu_from_host = gpu_from_host
self.mode_with_gpu = mode_with_opt.including("gpuarray", "scan")
self.mode_with_gpu_nodebug = mode_nodebug.including("gpuarray", "scan")
# Make sure to activate the new backend, if possible otherwise
# tesing this class directly will always skip.
import tests.gpuarray.config # noqa: F401
# Skip the test if pygpu is not available
if not self.gpu_backend.pygpu_activated:
pytest.skip("Optional package pygpu disabled")
utt.seed_rng()
def is_scan_on_gpu(self, node):
return node.op.info.get("gpua", False)
@pytest.mark.skipif( @pytest.mark.skipif(
not theano.config.cxx, reason="G++ not available, so we need to skip this test." not theano.config.cxx, reason="G++ not available, so we need to skip this test."
) )
...@@ -5351,7 +4833,7 @@ def test_speed(): ...@@ -5351,7 +4833,7 @@ def test_speed():
r = np.arange(10000).astype(theano.config.floatX).reshape(1000, 10) r = np.arange(10000).astype(theano.config.floatX).reshape(1000, 10)
s_r = tensor.matrix() s_r = tensor.matrix()
s_y, updates = theano.scan( s_y, updates = scan(
fn=lambda ri, rii: ri + rii, fn=lambda ri, rii: ri + rii,
sequences=[s_r[1:]], sequences=[s_r[1:]],
outputs_info=tensor.constant(r[0]), outputs_info=tensor.constant(r[0]),
...@@ -5419,7 +4901,7 @@ def test_speed_rnn(): ...@@ -5419,7 +4901,7 @@ def test_speed_rnn():
r = np.arange(L * N).astype(theano.config.floatX).reshape(L, N) r = np.arange(L * N).astype(theano.config.floatX).reshape(L, N)
s_r = tensor.matrix() s_r = tensor.matrix()
s_y, updates = theano.scan( s_y, updates = scan(
fn=lambda ri, rii: tensor.tanh(tensor.dot(rii, w)), fn=lambda ri, rii: tensor.tanh(tensor.dot(rii, w)),
sequences=[s_r[1:]], sequences=[s_r[1:]],
outputs_info=tensor.constant(r[0]), outputs_info=tensor.constant(r[0]),
...@@ -5515,46 +4997,32 @@ def test_speed_batchrnn(): ...@@ -5515,46 +4997,32 @@ def test_speed_batchrnn():
def test_compute_test_value(): def test_compute_test_value():
# Verify that test values can be used with scan. # Verify that test values can be used with scan.
backup = theano.config.compute_test_value with theano.change_flags(compute_test_value="raise"):
theano.config.compute_test_value = "raise"
try:
x = tensor.vector("x") x = tensor.vector("x")
xv = np.ones(3, dtype=theano.config.floatX) xv = np.ones(3, dtype=theano.config.floatX)
x.tag.test_value = xv x.tag.test_value = xv
y = theano.shared(np.arange(3, dtype=theano.config.floatX), name="y") y = theano.shared(np.arange(3, dtype=theano.config.floatX), name="y")
z, updates = theano.scan(fn=lambda u, v: u + v, sequences=[x, y]) z, updates = scan(fn=lambda u, v: u + v, sequences=[x, y])
assert not updates assert not updates
z.name = "z" z.name = "z"
# The gradient computation used to crash before 6af465e. # The gradient computation used to crash before 6af465e.
tensor.grad(z.sum(), x) tensor.grad(z.sum(), x)
# f = theano.function([x], g)
# print f(xv)
finally:
theano.config.compute_test_value = backup
def test_compute_test_value_nonseq(): def test_compute_test_value_nonseq():
# Verify that test values can be used for non_sequences with scan. # Verify that test values can be used for non_sequences with scan.
backup = theano.config.compute_test_value with theano.change_flags(compute_test_value="raise"):
theano.config.compute_test_value = "raise"
try:
x = tensor.vector("x") x = tensor.vector("x")
xv = np.ones(3, dtype=theano.config.floatX) xv = np.ones(3, dtype=theano.config.floatX)
x.tag.test_value = xv x.tag.test_value = xv
y = theano.shared( y = theano.shared(
np.arange(9, dtype=theano.config.floatX).reshape(3, 3), name="y" np.arange(9, dtype=theano.config.floatX).reshape(3, 3), name="y"
) )
z, updates = theano.scan( z, updates = scan(fn=lambda u, v: u + v, sequences=[x], non_sequences=[y])
fn=lambda u, v: u + v, sequences=[x], non_sequences=[y]
)
assert not updates assert not updates
z.name = "z" z.name = "z"
# The gradient computation used to crash before 6af465e. # The gradient computation used to crash before 6af465e.
tensor.grad(z.sum(), x) tensor.grad(z.sum(), x)
# f = theano.function([x], g)
# print f(xv)
finally:
theano.config.compute_test_value = backup
def test_compute_test_value_grad(): def test_compute_test_value_grad():
...@@ -5562,12 +5030,7 @@ def test_compute_test_value_grad(): ...@@ -5562,12 +5030,7 @@ def test_compute_test_value_grad():
# https://groups.google.com/d/msg/theano-users/fAP3i2CbskQ/3OgBf4yjqiQJ # https://groups.google.com/d/msg/theano-users/fAP3i2CbskQ/3OgBf4yjqiQJ
WEIGHT = np.array([1, 2, 1, 3, 4, 1, 5, 6, 1, 7, 8, 1], dtype="float32") WEIGHT = np.array([1, 2, 1, 3, 4, 1, 5, 6, 1, 7, 8, 1], dtype="float32")
old_compute_test_val = theano.config.compute_test_value with theano.change_flags(compute_test_value="raise", exception_verbosity="high"):
old_exception_verbosity = theano.config.exception_verbosity
try:
theano.config.compute_test_value = "raise"
theano.config.exception_verbosity = "high"
W_flat = tensor.fvector(name="W") W_flat = tensor.fvector(name="W")
W_flat.tag.test_value = WEIGHT W_flat.tag.test_value = WEIGHT
W = W_flat.reshape((2, 2, 3)) W = W_flat.reshape((2, 2, 3))
...@@ -5582,7 +5045,7 @@ def test_compute_test_value_grad(): ...@@ -5582,7 +5045,7 @@ def test_compute_test_value_grad():
def loss_ti(ti, sum_ti, mi, W): def loss_ti(ti, sum_ti, mi, W):
return W.sum().sum().sum() + sum_ti return W.sum().sum().sum() + sum_ti
result_ti, _ = theano.scan( result_ti, _ = scan(
fn=loss_ti, fn=loss_ti,
outputs_info=outputs_ti, outputs_info=outputs_ti,
sequences=tensor.arange(W.shape[1], dtype="int32"), sequences=tensor.arange(W.shape[1], dtype="int32"),
...@@ -5591,7 +5054,7 @@ def test_compute_test_value_grad(): ...@@ -5591,7 +5054,7 @@ def test_compute_test_value_grad():
lossmi = result_ti[-1] lossmi = result_ti[-1]
return sum_mi + lossmi return sum_mi + lossmi
result_mi, _ = theano.scan( result_mi, _ = scan(
fn=loss_mi, fn=loss_mi,
outputs_info=outputs_mi, outputs_info=outputs_mi,
sequences=tensor.arange(W.shape[0], dtype="int32"), sequences=tensor.arange(W.shape[0], dtype="int32"),
...@@ -5600,9 +5063,6 @@ def test_compute_test_value_grad(): ...@@ -5600,9 +5063,6 @@ def test_compute_test_value_grad():
loss = result_mi[-1] loss = result_mi[-1]
tensor.grad(loss, W_flat) tensor.grad(loss, W_flat)
finally:
theano.config.compute_test_value = old_compute_test_val
theano.config.exception_verbosity = old_exception_verbosity
def test_compute_test_value_grad_cast(): def test_compute_test_value_grad_cast():
...@@ -5610,15 +5070,13 @@ def test_compute_test_value_grad_cast(): ...@@ -5610,15 +5070,13 @@ def test_compute_test_value_grad_cast():
# Reported by Daniel Renshaw at # Reported by Daniel Renshaw at
# https://groups.google.com/d/topic/theano-users/o4jK9xDe5WI/discussion # https://groups.google.com/d/topic/theano-users/o4jK9xDe5WI/discussion
floatX = theano.config.floatX floatX = theano.config.floatX
backup = theano.config.compute_test_value with theano.change_flags(compute_test_value="raise"):
theano.config.compute_test_value = "raise"
try:
h = tensor.matrix("h") h = tensor.matrix("h")
h.tag.test_value = np.array([[1, 2, 3, 4], [5, 6, 7, 8]], dtype=floatX) h.tag.test_value = np.array([[1, 2, 3, 4], [5, 6, 7, 8]], dtype=floatX)
w = theano.shared(np.random.randn(4, 3).astype(floatX), name="w") w = theano.shared(np.random.randn(4, 3).astype(floatX), name="w")
outputs, _ = theano.scan( outputs, _ = scan(
lambda i, h, w: (theano.tensor.dot(h[i], w), i), lambda i, h, w: (theano.tensor.dot(h[i], w), i),
outputs_info=[None, 0], outputs_info=[None, 0],
non_sequences=[h, w], non_sequences=[h, w],
...@@ -5626,25 +5084,19 @@ def test_compute_test_value_grad_cast(): ...@@ -5626,25 +5084,19 @@ def test_compute_test_value_grad_cast():
) )
theano.grad(outputs[0].sum(), w) theano.grad(outputs[0].sum(), w)
finally:
theano.config.compute_test_value = backup
def test_constant_folding_n_steps(): def test_constant_folding_n_steps():
# The following code used to crash at revision 2060b8f, in the constant # The following code used to crash at revision 2060b8f, in the constant
# folding optimization step. # folding optimization step.
res, _ = theano.scan( res, _ = scan(
lambda x: x * 2, lambda x: x * 2,
outputs_info=tensor.ones(()), outputs_info=tensor.ones(()),
# The constant `n_steps` was causing the crash. # The constant `n_steps` was causing the crash.
n_steps=10, n_steps=10,
) )
on_opt_error = theano.config.on_opt_error with theano.change_flags(on_opt_error="raise"):
theano.config.on_opt_error = "raise"
try:
theano.function([], res)() theano.function([], res)()
finally:
theano.config.on_opt_error = on_opt_error
def test_outputs_taps_check(): def test_outputs_taps_check():
...@@ -5657,10 +5109,10 @@ def test_outputs_taps_check(): ...@@ -5657,10 +5109,10 @@ def test_outputs_taps_check():
outputs_info = {"initial": y, "taps": [0]} outputs_info = {"initial": y, "taps": [0]}
with pytest.raises(ValueError): with pytest.raises(ValueError):
theano.scan(f, x, outputs_info) scan(f, x, outputs_info)
outputs_info = {"initial": y, "taps": [-1, -1]} outputs_info = {"initial": y, "taps": [-1, -1]}
with pytest.raises(ValueError): with pytest.raises(ValueError):
theano.scan(f, x, outputs_info) scan(f, x, outputs_info)
def test_default_value_broadcasted(): def test_default_value_broadcasted():
...@@ -5680,7 +5132,7 @@ def test_default_value_broadcasted(): ...@@ -5680,7 +5132,7 @@ def test_default_value_broadcasted():
pre_h = theano.tensor.dot(x, W_x) pre_h = theano.tensor.dot(x, W_x)
return pre_h return pre_h
value, scan_updates = theano.scan( value, scan_updates = scan(
_active, _active,
sequences=X, sequences=X,
outputs_info=[theano.tensor.alloc(floatx(0.0), 1, out_size)], outputs_info=[theano.tensor.alloc(floatx(0.0), 1, out_size)],
...@@ -5696,7 +5148,7 @@ class TestInconsistentBroadcast: ...@@ -5696,7 +5148,7 @@ class TestInconsistentBroadcast:
def test_raise_error(self): def test_raise_error(self):
x = tensor.tensor3() x = tensor.tensor3()
initial_x = tensor.constant(np.zeros((1, 10))) initial_x = tensor.constant(np.zeros((1, 10)))
y, updates = theano.scan( y, updates = scan(
fn=lambda x, prev_x: x + prev_x, fn=lambda x, prev_x: x + prev_x,
sequences=x, sequences=x,
outputs_info=[dict(initial=initial_x)], outputs_info=[dict(initial=initial_x)],
...@@ -5707,7 +5159,7 @@ class TestInconsistentBroadcast: ...@@ -5707,7 +5159,7 @@ class TestInconsistentBroadcast:
# No error here, because the broadcast patterns are consistent. # No error here, because the broadcast patterns are consistent.
initial_x = tensor.unbroadcast(initial_x, 0, 1) initial_x = tensor.unbroadcast(initial_x, 0, 1)
y, updates = theano.scan( y, updates = scan(
fn=lambda x, prev_x: x + prev_x, fn=lambda x, prev_x: x + prev_x,
sequences=x, sequences=x,
outputs_info=[dict(initial=initial_x)], outputs_info=[dict(initial=initial_x)],
...@@ -5724,7 +5176,7 @@ class TestMissingInputError: ...@@ -5724,7 +5176,7 @@ class TestMissingInputError:
def count_up(): def count_up():
return tensor.zeros(()), {c: c + inc} return tensor.zeros(()), {c: c + inc}
_, updates = theano.scan(count_up, n_steps=20) _, updates = scan(count_up, n_steps=20)
theano.function(inputs=[inc], outputs=[], updates=updates) theano.function(inputs=[inc], outputs=[], updates=updates)
...@@ -5738,8 +5190,8 @@ class TestGradUntil: ...@@ -5738,8 +5190,8 @@ class TestGradUntil:
self.numpy_gradient = 2 * np.concatenate([self.seq[:7], z], axis=0) self.numpy_gradient = 2 * np.concatenate([self.seq[:7], z], axis=0)
def test_grad_until(self): def test_grad_until(self):
r, _ = theano.scan( r, _ = scan(
lambda x, u: (x * x, theano.scan_module.until(x > u)), lambda x, u: (x * x, until(x > u)),
sequences=self.x, sequences=self.x,
non_sequences=[self.threshold], non_sequences=[self.threshold],
) )
...@@ -5757,8 +5209,8 @@ class TestGradUntil: ...@@ -5757,8 +5209,8 @@ class TestGradUntil:
X = tensor.matrix(name="x") X = tensor.matrix(name="x")
arr = tile_array(self.seq) arr = tile_array(self.seq)
r, _ = theano.scan( r, _ = scan(
lambda x, u: (x * x, theano.scan_module.until(tensor.all(x > u))), lambda x, u: (x * x, until(tensor.all(x > u))),
sequences=X, sequences=X,
non_sequences=[self.threshold], non_sequences=[self.threshold],
) )
...@@ -5771,8 +5223,8 @@ class TestGradUntil: ...@@ -5771,8 +5223,8 @@ class TestGradUntil:
def test_grad_until_and_truncate(self): def test_grad_until_and_truncate(self):
n = 3 n = 3
r, _ = theano.scan( r, _ = scan(
lambda x, u: (x * x, theano.scan_module.until(x > u)), lambda x, u: (x * x, until(x > u)),
sequences=self.x, sequences=self.x,
non_sequences=[self.threshold], non_sequences=[self.threshold],
truncate_gradient=n, truncate_gradient=n,
...@@ -5787,8 +5239,8 @@ class TestGradUntil: ...@@ -5787,8 +5239,8 @@ class TestGradUntil:
def test_grad_until_and_truncate_sequence_taps(self): def test_grad_until_and_truncate_sequence_taps(self):
n = 3 n = 3
r, _ = theano.scan( r, _ = scan(
lambda x, y, u: (x * y, theano.scan_module.until(y > u)), lambda x, y, u: (x * y, until(y > u)),
sequences=dict(input=self.x, taps=[-2, 0]), sequences=dict(input=self.x, taps=[-2, 0]),
non_sequences=[self.threshold], non_sequences=[self.threshold],
truncate_gradient=n, truncate_gradient=n,
...@@ -5810,10 +5262,10 @@ def test_condition_hidden_inp(): ...@@ -5810,10 +5262,10 @@ def test_condition_hidden_inp():
def accum(prev_value, step): def accum(prev_value, step):
new_value = prev_value + step new_value = prev_value + step
new_step = step + 1 new_step = step + 1
condition = theano.scan_module.until(new_value > max_value) condition = until(new_value > max_value)
return [new_value, new_step], condition return [new_value, new_step], condition
rs, updates = theano.scan(fn=accum, outputs_info=[0, 0], n_steps=n_steps) rs, updates = scan(fn=accum, outputs_info=[0, 0], n_steps=n_steps)
f = theano.function(inputs=[max_value, n_steps], outputs=rs) f = theano.function(inputs=[max_value, n_steps], outputs=rs)
...@@ -5828,7 +5280,7 @@ def test_mintap_onestep(): ...@@ -5828,7 +5280,7 @@ def test_mintap_onestep():
new_sum = prev_sum + seq_t new_sum = prev_sum + seq_t
return new_sum return new_sum
rs, updates = theano.scan(fn=accum, sequences=seq_info, outputs_info=0, n_steps=1) rs, updates = scan(fn=accum, sequences=seq_info, outputs_info=0, n_steps=1)
f = theano.function(inputs=[seq], outputs=rs) f = theano.function(inputs=[seq], outputs=rs)
_seq = np.arange(20).astype("int32") _seq = np.arange(20).astype("int32")
......
tests/scan_module/test_scan_checkpoints.py → tests/scan/test_checkpoints.py
浏览文件 @ 6d39faaf
...@@ -2,29 +2,22 @@ import numpy as np ...@@ -2,29 +2,22 @@ import numpy as np
import pytest import pytest
import theano import theano
import theano.gpuarray
import theano.tensor as tt import theano.tensor as tt
from theano.scan.basic import scan
from theano.scan.checkpoints import scan_checkpoints
try:
from pygpu.gpuarray import GpuArrayException
PYGPU_AVAILABLE = True
except ImportError:
PYGPU_AVAILABLE = False
class TestScanCheckpoint: class TestScanCheckpoint:
def setup_method(self): def setup_method(self):
self.k = tt.iscalar("k") self.k = tt.iscalar("k")
self.A = tt.vector("A") self.A = tt.vector("A")
result, _ = theano.scan( result, _ = scan(
fn=lambda prior_result, A: prior_result * A, fn=lambda prior_result, A: prior_result * A,
outputs_info=tt.ones_like(self.A), outputs_info=tt.ones_like(self.A),
non_sequences=self.A, non_sequences=self.A,
n_steps=self.k, n_steps=self.k,
) )
result_check, _ = theano.scan_checkpoints( result_check, _ = scan_checkpoints(
fn=lambda prior_result, A: prior_result * A, fn=lambda prior_result, A: prior_result * A,
outputs_info=tt.ones_like(self.A), outputs_info=tt.ones_like(self.A),
non_sequences=self.A, non_sequences=self.A,
...@@ -52,34 +45,7 @@ class TestScanCheckpoint: ...@@ -52,34 +45,7 @@ class TestScanCheckpoint:
out, out_check = f(range(10), 101) out, out_check = f(range(10), 101)
assert np.allclose(out, out_check) assert np.allclose(out, out_check)
@pytest.mark.skipif(~PYGPU_AVAILABLE, reason="Requires pygpu.")
@pytest.mark.skipif(
None not in theano.gpuarray.type.list_contexts(),
reason="Requires gpuarray backend.",
)
def test_memory(self):
from tests.gpuarray.config import mode_with_gpu # noqa
f = theano.function(
inputs=[self.A, self.k], outputs=self.grad_A, mode=mode_with_gpu
)
f_check = theano.function(
inputs=[self.A, self.k], outputs=self.grad_A_check, mode=mode_with_gpu
)
free_gmem = theano.gpuarray.type._context_reg[None].free_gmem
data = np.ones(free_gmem // 3000, dtype=np.float32)
# Check that it works with the checkpoints
size = 1000
if isinstance(mode_with_gpu, theano.compile.DebugMode):
size = 100
f_check(data, size)
# Check that the basic scan fails in that case
# Skip that check in DebugMode, as it can fail in different ways
if not isinstance(mode_with_gpu, theano.compile.DebugMode):
with pytest.raises(GpuArrayException):
f(data, 1000)
def test_taps_error(self): def test_taps_error(self):
# Test that an error rises if we use taps in outputs_info. # Test that an error rises if we use taps in outputs_info.
with pytest.raises(RuntimeError): with pytest.raises(RuntimeError):
theano.scan_checkpoints(lambda: None, [], {"initial": self.A, "taps": [-2]}) scan_checkpoints(lambda: None, [], {"initial": self.A, "taps": [-2]})
tests/scan_module/test_scan_opt.py → tests/scan/test_opt.py
浏览文件 @ 6d39faaf
...@@ -4,7 +4,7 @@ import theano ...@@ -4,7 +4,7 @@ import theano
import theano.tensor as tt import theano.tensor as tt
from tests import unittest_tools as utt from tests import unittest_tools as utt
from theano import config from theano import config
from theano.scan_module.scan_op import Scan from theano.scan.op import Scan
mode = theano.compile.mode.get_mode(config.mode) mode = theano.compile.mode.get_mode(config.mode)
......
tests/scan_module/test_scan_utils.py → tests/scan/test_utils.py
浏览文件 @ 6d39faaf
...@@ -5,7 +5,7 @@ import pytest ...@@ -5,7 +5,7 @@ import pytest
import theano import theano
from theano import tensor from theano import tensor
from theano.scan_module.scan_utils import map_variables from theano.scan.utils import map_variables
class TestMapVariables: class TestMapVariables:
...@@ -130,7 +130,7 @@ class TestMapVariables: ...@@ -130,7 +130,7 @@ class TestMapVariables:
# construct the outer graph # construct the outer graph
c = tensor.scalar() c = tensor.scalar()
d = tensor.scalar() d = tensor.scalar()
u = theano.OpFromGraph([a, b], [r])(c, d) u = theano.compile.builders.OpFromGraph([a, b], [r])(c, d)
t = z * u t = z * u
(v,) = map_variables(self.replacer, [t]) (v,) = map_variables(self.replacer, [t])
t2 = z * v t2 = z * v
......
theano/__init__.py
浏览文件 @ 6d39faaf
...@@ -79,7 +79,6 @@ from theano import scalar, tensor ...@@ -79,7 +79,6 @@ from theano import scalar, tensor
from theano.compile import ( from theano.compile import (
In, In,
Mode, Mode,
OpFromGraph,
Out, Out,
Param, Param,
ProfileStats, ProfileStats,
...@@ -194,4 +193,4 @@ def sparse_grad(var): ...@@ -194,4 +193,4 @@ def sparse_grad(var):
import theano.tensor.shared_randomstreams import theano.tensor.shared_randomstreams
from theano.scan_module import clone, foldl, foldr, map, reduce, scan, scan_checkpoints from theano.scan import checkpoints, clone, foldl, foldr, map, reduce, scan
theano/compile/__init__.py
浏览文件 @ 6d39faaf
from theano.compile.builders import OpFromGraph, ops_with_inner_function
from theano.compile.debugmode import DebugMode from theano.compile.debugmode import DebugMode
from theano.compile.function.pfunc import Param, pfunc, rebuild_collect_shared from theano.compile.function.pfunc import Param, pfunc, rebuild_collect_shared
from theano.compile.function.types import ( from theano.compile.function.types import (
......
theano/compile/builders.py
浏览文件 @ 6d39faaf
...@@ -10,12 +10,69 @@ from theano.compile.function.types import orig_function ...@@ -10,12 +10,69 @@ from theano.compile.function.types import orig_function
from theano.compile.mode import optdb from theano.compile.mode import optdb
from theano.compile.sharedvalue import SharedVariable from theano.compile.sharedvalue import SharedVariable
from theano.gof import Variable, ops_with_inner_function from theano.gof import Variable, ops_with_inner_function
from theano.gof.fg import FunctionGraph
from theano.gof.graph import io_connection_pattern from theano.gof.graph import io_connection_pattern
from theano.gof.null_type import NullType from theano.gof.null_type import NullType
from theano.gof.op import Op
from theano.gof.opt import in2out, local_optimizer
from theano.gradient import DisconnectedType from theano.gradient import DisconnectedType
from theano.tensor.opt import ShapeFeature
class OpFromGraph(gof.Op): def infer_shape(outs, inputs, input_shapes):
"""
Compute the shape of the outputs given the shape of the inputs of a theano
graph.
We do it this way to avoid compiling the inner function just to get
the shape. Changes to ShapeFeature could require changes in this function.
"""
# We use a ShapeFeature because it has all the necessary logic
# inside. We don't use the full ShapeFeature interface, but we
# let it initialize itself with an empty fgraph, otherwise we will
# need to do it manually
for inp, inp_shp in zip(inputs, input_shapes):
if inp_shp is not None and len(inp_shp) != inp.ndim:
assert len(inp_shp) == inp.ndim
shape_feature = ShapeFeature()
shape_feature.on_attach(FunctionGraph([], []))
# Initialize shape_of with the input shapes
for inp, inp_shp in zip(inputs, input_shapes):
shape_feature.set_shape(inp, inp_shp)
def local_traverse(out):
"""
Go back in the graph, from out, adding computable shapes to shape_of.
"""
if out in shape_feature.shape_of:
# Its shape is already known
return
elif out.owner is None:
# This is an input of the graph
shape_feature.init_r(out)
else:
# Recurse over inputs
for inp in out.owner.inputs:
if inp not in shape_feature.shape_of:
local_traverse(inp)
# shape_feature.on_import does not actually use an fgraph
# It will call infer_shape and set_shape appropriately
dummy_fgraph = None
shape_feature.on_import(dummy_fgraph, out.owner, reason="dummy")
ret = []
for o in outs:
local_traverse(o)
ret.append(shape_feature.shape_of[o])
return ret
class OpFromGraph(Op):
r""" r"""
This creates an ``Op`` from inputs and outputs lists of variables. This creates an ``Op`` from inputs and outputs lists of variables.
The signature is similar to :func:`theano.function <theano.function>` The signature is similar to :func:`theano.function <theano.function>`
...@@ -28,9 +85,9 @@ class OpFromGraph(gof.Op): ...@@ -28,9 +85,9 @@ class OpFromGraph(gof.Op):
Parameters Parameters
---------- ----------
inputs: list of :class:`Variable <theano.gof.Variable>` inputs: list of :class:`Variable <theano.gof.graph.Variable>`
outputs: list of :class:`Variable <theano.gof.Variable>` outputs: list of :class:`Variable <theano.gof.graph.Variable>`
inline: bool, optional inline: bool, optional
Defaults to ``False`` Defaults to ``False``
...@@ -52,15 +109,15 @@ class OpFromGraph(gof.Op): ...@@ -52,15 +109,15 @@ class OpFromGraph(gof.Op):
arguments as one would specify in grad() method. arguments as one would specify in grad() method.
callable : Should take two args: ``inputs`` and ``output_grads``. callable : Should take two args: ``inputs`` and ``output_grads``.
Each argument is expected to be a list of :class:`Variable <theano.gof.Variable>`. Each argument is expected to be a list of :class:`Variable <theano.gof.graph.Variable>`.
Must return list of :class:`Variable <theano.gof.Variable>`. Must return list of :class:`Variable <theano.gof.graph.Variable>`.
Variable : Variable :
``NullType() instance`` : Treat as non-differentiable ``NullType() instance`` : Treat as non-differentiable
``DisconnectedType() instance`` : Treat as disconnected gradient, numerically gives zero ``DisconnectedType() instance`` : Treat as disconnected gradient, numerically gives zero
list: Each OpFromGraph/callable must return a single list: Each OpFromGraph/callable must return a single
:class:`Variable <theano.gof.Variable>`. Each list element corresponds to gradient of :class:`Variable <theano.gof.graph.Variable>`. Each list element corresponds to gradient of
a specific input, length of list must be equal to number of inputs. a specific input, length of list must be equal to number of inputs.
lop_overrides : single or list of {'default', OpFromGraph, callable, Variable with special type}, optional lop_overrides : single or list of {'default', OpFromGraph, callable, Variable with special type}, optional
...@@ -74,15 +131,15 @@ class OpFromGraph(gof.Op): ...@@ -74,15 +131,15 @@ class OpFromGraph(gof.Op):
arguments as one would specify in grad() method. arguments as one would specify in grad() method.
callable : Should take three args: ``inputs``, ``outputs`` and ``output_grads``. callable : Should take three args: ``inputs``, ``outputs`` and ``output_grads``.
Each argument is expected to be a list of :class:`Variable <theano.gof.Variable>`. Each argument is expected to be a list of :class:`Variable <theano.gof.graph.Variable>`.
Must return list of :class:`Variable <theano.gof.Variable>`. Must return list of :class:`Variable <theano.gof.graph.Variable>`.
Variable : Variable :
``NullType() instance`` : Treat as non-differentiable ``NullType() instance`` : Treat as non-differentiable
``DisconnectedType() instance`` : Treat as disconnected gradient, numerically gives zero ``DisconnectedType() instance`` : Treat as disconnected gradient, numerically gives zero
list: Each OpFromGraph/callable must return a single list: Each OpFromGraph/callable must return a single
:class:`Variable <theano.gof.Variable>`. Each list element corresponds to gradient of :class:`Variable <theano.gof.graph.Variable>`. Each list element corresponds to gradient of
a specific input, length of list must be equal to number of inputs. a specific input, length of list must be equal to number of inputs.
rop_overrides : single or list of {'default', OpFromGraph, callable, Variable with special type}, optional rop_overrides : single or list of {'default', OpFromGraph, callable, Variable with special type}, optional
...@@ -95,15 +152,15 @@ class OpFromGraph(gof.Op): ...@@ -95,15 +152,15 @@ class OpFromGraph(gof.Op):
arguments as one would specify in R_op() method. arguments as one would specify in R_op() method.
callable : Should take two args: ``inputs`` and ``eval_points``. callable : Should take two args: ``inputs`` and ``eval_points``.
Each argument is expected to be a list of :class:`Variable <theano.gof.Variable>`. Each argument is expected to be a list of :class:`Variable <theano.gof.graph.Variable>`.
Must return list of :class:`Variable <theano.gof.Variable>`. Must return list of :class:`Variable <theano.gof.graph.Variable>`.
Variable : Variable :
``NullType() instance`` : Treat as non-differentiable ``NullType() instance`` : Treat as non-differentiable
``DisconnectedType() instance`` : Treat as zero since DisconnectedType is not yet supported in R_op ``DisconnectedType() instance`` : Treat as zero since DisconnectedType is not yet supported in R_op
list: Each OpFromGraph/callable must return a single list: Each OpFromGraph/callable must return a single
:class:`Variable <theano.gof.Variable>`. Each list element corresponds :class:`Variable <theano.gof.graph.Variable>`. Each list element corresponds
to a specific output of R_op, length of list must be equal to number of outputs. to a specific output of R_op, length of list must be equal to number of outputs.
connection_pattern : list of list connection_pattern : list of list
...@@ -158,7 +215,8 @@ class OpFromGraph(gof.Op): ...@@ -158,7 +215,8 @@ class OpFromGraph(gof.Op):
.. code-block:: python .. code-block:: python
from theano import function, OpFromGraph, tensor from theano import function, tensor
from theano.compile.builders import OpFromGraph
x, y, z = tensor.scalars('xyz') x, y, z = tensor.scalars('xyz')
e = x + y * z e = x + y * z
op = OpFromGraph([x, y, z], [e]) op = OpFromGraph([x, y, z], [e])
...@@ -172,7 +230,9 @@ class OpFromGraph(gof.Op): ...@@ -172,7 +230,9 @@ class OpFromGraph(gof.Op):
import numpy as np import numpy as np
import theano import theano
from theano import config, function, OpFromGraph, tensor from theano import config, function, tensor
from theano.compile.builders import OpFromGraph
x, y, z = tensor.scalars('xyz') x, y, z = tensor.scalars('xyz')
s = theano.shared(np.random.rand(2, 2).astype(config.floatX)) s = theano.shared(np.random.rand(2, 2).astype(config.floatX))
e = x + y * z + s e = x + y * z + s
...@@ -185,7 +245,9 @@ class OpFromGraph(gof.Op): ...@@ -185,7 +245,9 @@ class OpFromGraph(gof.Op):
.. code-block:: python .. code-block:: python
from theano import function, OpFromGraph, tensor, grad from theano import function, tensor, grad
from theano.compile.builders import OpFromGraph
x, y, z = tensor.scalars('xyz') x, y, z = tensor.scalars('xyz')
e = x + y * z e = x + y * z
def rescale_dy(inps, grads): def rescale_dy(inps, grads):
...@@ -718,9 +780,8 @@ class OpFromGraph(gof.Op): ...@@ -718,9 +780,8 @@ class OpFromGraph(gof.Op):
return list(map(list, cpmat_self)) return list(map(list, cpmat_self))
def infer_shape(self, node, shapes): def infer_shape(self, node, shapes):
out_shp = theano.scan_module.scan_utils.infer_shape(
self.local_outputs, self.local_inputs, shapes out_shp = infer_shape(self.local_outputs, self.local_inputs, shapes)
)
# Clone the output shape so that shape are computed from outer inputs. # Clone the output shape so that shape are computed from outer inputs.
# Note: # Note:
...@@ -756,7 +817,7 @@ class OpFromGraph(gof.Op): ...@@ -756,7 +817,7 @@ class OpFromGraph(gof.Op):
output[0] = variable.copy() output[0] = variable.copy()
@gof.local_optimizer([OpFromGraph]) @local_optimizer([OpFromGraph])
def inline_ofg_expansion(node): def inline_ofg_expansion(node):
""" """
This optimization expands internal graph of OpFromGraph. This optimization expands internal graph of OpFromGraph.
...@@ -777,7 +838,7 @@ def inline_ofg_expansion(node): ...@@ -777,7 +838,7 @@ def inline_ofg_expansion(node):
# and before the first scan optimizer. # and before the first scan optimizer.
optdb.register( optdb.register(
"inline_ofg_expansion", "inline_ofg_expansion",
gof.opt.in2out(inline_ofg_expansion), in2out(inline_ofg_expansion),
-0.01, -0.01,
"fast_compile", "fast_compile",
"fast_run", "fast_run",
......
theano/compile/debugmode.py
浏览文件 @ 6d39faaf
...@@ -682,7 +682,9 @@ def debugprint( ...@@ -682,7 +682,9 @@ def debugprint(
new_prefix_child = prefix_child + " " new_prefix_child = prefix_child + " "
if hasattr(i, "owner") and hasattr(i.owner, "op"): if hasattr(i, "owner") and hasattr(i.owner, "op"):
if isinstance(i.owner.op, theano.scan_module.scan_op.Scan): from theano.scan.op import Scan
if isinstance(i.owner.op, Scan):
scan_ops.append(i) scan_ops.append(i)
debugprint( debugprint(
......
theano/compile/function/types.py
浏览文件 @ 6d39faaf
...@@ -1386,7 +1386,7 @@ class FunctionMaker: ...@@ -1386,7 +1386,7 @@ class FunctionMaker:
try: try:
with open(graph_db_file, "rb") as f: with open(graph_db_file, "rb") as f:
# Temporary hack to allow # Temporary hack to allow
# tests.scan_module.test_scan.T_Scan to # tests.scan.test_scan.T_Scan to
# finish. Should be changed in definitive version. # finish. Should be changed in definitive version.
tmp = theano.config.unpickle_function tmp = theano.config.unpickle_function
theano.config.unpickle_function = False theano.config.unpickle_function = False
......
theano/compile/io.py
浏览文件 @ 6d39faaf
...@@ -8,8 +8,6 @@ import logging ...@@ -8,8 +8,6 @@ import logging
from theano import gof from theano import gof
from .sharedvalue import SharedVariable
_logger = logging.getLogger("theano.compile.io") _logger = logging.getLogger("theano.compile.io")
...@@ -211,6 +209,8 @@ class In(SymbolicInput): ...@@ -211,6 +209,8 @@ class In(SymbolicInput):
) )
if implicit is None: if implicit is None:
from theano.compile.sharedvalue import SharedVariable
implicit = isinstance(value, gof.Container) or isinstance( implicit = isinstance(value, gof.Container) or isinstance(
value, SharedVariable value, SharedVariable
) )
......
theano/compile/sharedvalue.py
浏览文件 @ 6d39faaf
...@@ -2,17 +2,16 @@ ...@@ -2,17 +2,16 @@
Provide a simple user friendly API to Theano-managed memory. Provide a simple user friendly API to Theano-managed memory.
""" """
# Standard imports
import copy import copy
import logging import logging
# Third-party imports
import numpy as np import numpy as np
# Theano imports from theano.gof.graph import Variable
from theano.gof import Container, Variable, generic, utils from theano.gof.link import Container
from theano.gof.type import generic
from theano.gof.utils import add_tag_trace
_logger = logging.getLogger("theano.compile.sharedvalue") _logger = logging.getLogger("theano.compile.sharedvalue")
...@@ -287,7 +286,7 @@ def shared(value, name=None, strict=False, allow_downcast=None, **kwargs): ...@@ -287,7 +286,7 @@ def shared(value, name=None, strict=False, allow_downcast=None, **kwargs):
allow_downcast=allow_downcast, allow_downcast=allow_downcast,
**kwargs, **kwargs,
) )
utils.add_tag_trace(var) add_tag_trace(var)
return var return var
except TypeError: except TypeError:
continue continue
......
theano/gof/toolbox.py
浏览文件 @ 6d39faaf
...@@ -533,11 +533,9 @@ class ReplaceValidate(History, Validator): ...@@ -533,11 +533,9 @@ class ReplaceValidate(History, Validator):
if verbose is None: if verbose is None:
verbose = config.optimizer_verbose verbose = config.optimizer_verbose
if config.scan.debug: if config.scan.debug:
scans = [ from theano.scan.op import Scan
n
for n in fgraph.apply_nodes scans = [n for n in fgraph.apply_nodes if isinstance(n.op, Scan)]
if isinstance(n.op, theano.scan_module.scan_op.Scan)
]
for r, new_r in replacements: for r, new_r in replacements:
try: try:
...@@ -581,11 +579,9 @@ class ReplaceValidate(History, Validator): ...@@ -581,11 +579,9 @@ class ReplaceValidate(History, Validator):
) )
raise raise
if config.scan.debug: if config.scan.debug:
scans2 = [ from theano.scan.op import Scan
n
for n in fgraph.apply_nodes scans2 = [n for n in fgraph.apply_nodes if isinstance(n.op, Scan)]
if isinstance(n.op, theano.scan_module.scan_op.Scan)
]
nb = len(scans) nb = len(scans)
nb2 = len(scans2) nb2 = len(scans2)
if nb2 > nb: if nb2 > nb:
......
theano/gof/utils.py
浏览文件 @ 6d39faaf
...@@ -103,8 +103,8 @@ def add_tag_trace(thing, user_line=None): ...@@ -103,8 +103,8 @@ def add_tag_trace(thing, user_line=None):
"theano\\scalar\\basic.py", "theano\\scalar\\basic.py",
"theano/sandbox/", "theano/sandbox/",
"theano\\sandbox\\", "theano\\sandbox\\",
"theano/scan_module/", "theano/scan/",
"theano\\scan_module\\", "theano\\scan\\",
"theano/sparse/", "theano/sparse/",
"theano\\sparse\\", "theano\\sparse\\",
"theano/typed_list/", "theano/typed_list/",
......
theano/gpuarray/opt.py
浏览文件 @ 6d39faaf
...@@ -153,7 +153,9 @@ from theano.ifelse import IfElse ...@@ -153,7 +153,9 @@ from theano.ifelse import IfElse
from theano.misc.ordered_set import OrderedSet from theano.misc.ordered_set import OrderedSet
from theano.scalar.basic import Cast, Pow, Scalar, log, neg, true_div from theano.scalar.basic import Cast, Pow, Scalar, log, neg, true_div
from theano.scalar.basic_scipy import Erfcinv, Erfinv from theano.scalar.basic_scipy import Erfcinv, Erfinv
from theano.scan_module import scan_op, scan_opt, scan_utils from theano.scan import utils
from theano.scan.op import Scan
from theano.scan.opt import ScanInplaceOptimizer
from theano.tensor.nnet import bn, conv3d2d from theano.tensor.nnet import bn, conv3d2d
from theano.tensor.nnet.abstract_conv import ( from theano.tensor.nnet.abstract_conv import (
AbstractConv2d, AbstractConv2d,
...@@ -2600,13 +2602,13 @@ def gpu_reconstruct_graph(inputs, outputs, tag=None): ...@@ -2600,13 +2602,13 @@ def gpu_reconstruct_graph(inputs, outputs, tag=None):
givens = {} givens = {}
for nw_x, x in zip(nw_inputs, inputs): for nw_x, x in zip(nw_inputs, inputs):
givens[x] = nw_x givens[x] = nw_x
nw_outputs = scan_utils.clone(outputs, replace=givens) nw_outputs = utils.clone(outputs, replace=givens)
return (nw_inputs, nw_outputs) return (nw_inputs, nw_outputs)
@register_opt("scan", "fast_compile") @register_opt("scan", "fast_compile")
@op_lifter([scan_op.Scan]) @op_lifter([Scan])
@register_opt2([scan_op.Scan], "fast_compile") @register_opt2([Scan], "fast_compile")
def local_gpua_scan_to_gpua(op, context_name, inputs, outputs): def local_gpua_scan_to_gpua(op, context_name, inputs, outputs):
info = copy.deepcopy(op.info) info = copy.deepcopy(op.info)
if info.get("gpua", False): if info.get("gpua", False):
...@@ -2628,7 +2630,7 @@ def local_gpua_scan_to_gpua(op, context_name, inputs, outputs): ...@@ -2628,7 +2630,7 @@ def local_gpua_scan_to_gpua(op, context_name, inputs, outputs):
scan_outs += [op.outputs[-1]] scan_outs += [op.outputs[-1]]
else: else:
scan_outs = [safe_to_gpu(x, context_name) for x in op.outputs] scan_outs = [safe_to_gpu(x, context_name) for x in op.outputs]
scan_outs = scan_utils.clone( scan_outs = utils.clone(
scan_outs, replace=list(zip(op.inputs, (safe_to_cpu(x) for x in scan_ins))) scan_outs, replace=list(zip(op.inputs, (safe_to_cpu(x) for x in scan_ins)))
) )
...@@ -2645,9 +2647,9 @@ def local_gpua_scan_to_gpua(op, context_name, inputs, outputs): ...@@ -2645,9 +2647,9 @@ def local_gpua_scan_to_gpua(op, context_name, inputs, outputs):
dtype=dtype, broadcastable=broadcastable, context_name=context_name dtype=dtype, broadcastable=broadcastable, context_name=context_name
) )
nw_op = scan_op.Scan( nw_op = Scan(scan_ins, scan_outs, info, typeConstructor=typebuild).make_node(
scan_ins, scan_outs, info, typeConstructor=typebuild *nw_ins
).make_node(*nw_ins) )
return nw_op.outputs return nw_op.outputs
...@@ -2916,7 +2918,7 @@ def local_gpu_ctc(op, context_name, inputs, outputs): ...@@ -2916,7 +2918,7 @@ def local_gpu_ctc(op, context_name, inputs, outputs):
# It will be added to fast_run if the GPU is enabled. # It will be added to fast_run if the GPU is enabled.
optdb.register( optdb.register(
"gpua_scanOp_make_inplace", "gpua_scanOp_make_inplace",
scan_opt.ScanInplaceOptimizer(typeInfer=_scan_type_infer, gpua_flag=True), ScanInplaceOptimizer(typeInfer=_scan_type_infer, gpua_flag=True),
75, 75,
"gpuarray", "gpuarray",
"inplace", "inplace",
......
theano/gradient.py
浏览文件 @ 6d39faaf
...@@ -1777,8 +1777,8 @@ def verify_grad( ...@@ -1777,8 +1777,8 @@ def verify_grad(
Notes Notes
----- -----
This function does not support multiple outputs. In This function does not support multiple outputs. In
tests/test_scan.py there is an experimental verify_grad that tests/scan/test_basic.py there is an experimental `verify_grad` that covers
covers that case as well by using random projections. that case as well by using random projections.
""" """
......
theano/ifelse.py
浏览文件 @ 6d39faaf
...@@ -20,7 +20,7 @@ import theano.tensor ...@@ -20,7 +20,7 @@ import theano.tensor
from theano import gof from theano import gof
from theano.compile import optdb from theano.compile import optdb
from theano.gof import Apply, Op from theano.gof import Apply, Op
from theano.scan_module.scan_utils import clone from theano.scan.utils import clone
from theano.tensor import TensorType, opt from theano.tensor import TensorType, opt
......
theano/printing.py
浏览文件 @ 6d39faaf
...@@ -120,6 +120,8 @@ def debugprint( ...@@ -120,6 +120,8 @@ def debugprint(
to the Apply's identifier, to indicate which output a line corresponds to. to the Apply's identifier, to indicate which output a line corresponds to.
""" """
from theano.scan.op import Scan
if not isinstance(depth, int): if not isinstance(depth, int):
raise Exception("depth parameter must be an int") raise Exception("depth parameter must be an int")
if file == "str": if file == "str":
...@@ -202,9 +204,7 @@ N.B.: ...@@ -202,9 +204,7 @@ N.B.:
for r, p, s, o in zip(results_to_print, profile_list, smap, order): for r, p, s, o in zip(results_to_print, profile_list, smap, order):
# Add the parent scan op to the list as well # Add the parent scan op to the list as well
if hasattr(r.owner, "op") and isinstance( if hasattr(r.owner, "op") and isinstance(r.owner.op, Scan):
r.owner.op, theano.scan_module.scan_op.Scan
):
scan_ops.append(r) scan_ops.append(r)
debugmode.debugprint( debugmode.debugprint(
...@@ -265,7 +265,7 @@ N.B.: ...@@ -265,7 +265,7 @@ N.B.:
for idx, i in enumerate(outputs): for idx, i in enumerate(outputs):
if hasattr(i, "owner") and hasattr(i.owner, "op"): if hasattr(i, "owner") and hasattr(i.owner, "op"):
if isinstance(i.owner.op, theano.scan_module.scan_op.Scan): if isinstance(i.owner.op, Scan):
scan_ops.append(i) scan_ops.append(i)
debugmode.debugprint( debugmode.debugprint(
...@@ -804,6 +804,8 @@ def pydotprint( ...@@ -804,6 +804,8 @@ def pydotprint(
scan separately after the top level debugprint output. scan separately after the top level debugprint output.
""" """
from theano.scan.op import Scan
if colorCodes is None: if colorCodes is None:
colorCodes = default_colorCodes colorCodes = default_colorCodes
...@@ -1119,11 +1121,7 @@ def pydotprint( ...@@ -1119,11 +1121,7 @@ def pydotprint(
outfile += "." + format outfile += "." + format
if scan_graphs: if scan_graphs:
scan_ops = [ scan_ops = [(idx, x) for idx, x in enumerate(topo) if isinstance(x.op, Scan)]
(idx, x)
for idx, x in enumerate(topo)
if isinstance(x.op, theano.scan_module.scan_op.Scan)
]
path, fn = os.path.split(outfile) path, fn = os.path.split(outfile)
basename = ".".join(fn.split(".")[:-1]) basename = ".".join(fn.split(".")[:-1])
# Safe way of doing things .. a file name may contain multiple . # Safe way of doing things .. a file name may contain multiple .
......
theano/sandbox/jaxify.py
浏览文件 @ 6d39faaf
...@@ -20,8 +20,8 @@ from theano.compile.ops import ( ...@@ -20,8 +20,8 @@ from theano.compile.ops import (
from theano.gof import FunctionGraph from theano.gof import FunctionGraph
from theano.ifelse import IfElse from theano.ifelse import IfElse
from theano.scalar.basic import Cast, Clip, Composite, Identity, ScalarOp from theano.scalar.basic import Cast, Clip, Composite, Identity, ScalarOp
from theano.scan_module.scan_op import Scan from theano.scan.op import Scan
from theano.scan_module.scan_utils import scan_args as ScanArgs from theano.scan.utils import scan_args as ScanArgs
from theano.tensor.basic import ( from theano.tensor.basic import (
Alloc, Alloc,
AllocEmpty, AllocEmpty,
......
theano/scan_module/__init__.py → theano/scan/__init__.py
浏览文件 @ 6d39faaf
...@@ -5,7 +5,7 @@ Scanning is a general form of recurrence, which can be used for looping. ...@@ -5,7 +5,7 @@ Scanning is a general form of recurrence, which can be used for looping.
The idea is that you *scan* a function along some input sequence, producing The idea is that you *scan* a function along some input sequence, producing
an output at each time-step that can be seen (but not modified) by the an output at each time-step that can be seen (but not modified) by the
function at the next time-step. (Technically, the function can see the function at the next time-step. (Technically, the function can see the
previous K time-steps of your outputs and L time steps (from the past and previous K time-steps of your outputs and L time steps (from past and
future) of your inputs. future) of your inputs.
So for example, ``sum()`` could be computed by scanning the ``z+x_i`` So for example, ``sum()`` could be computed by scanning the ``z+x_i``
...@@ -13,15 +13,21 @@ function over a list, given an initial state of ``z=0``. ...@@ -13,15 +13,21 @@ function over a list, given an initial state of ``z=0``.
Special cases: Special cases:
* A *reduce* operation can be performed by returning only the last * A *reduce* operation can be performed by using only the last
output of a ``scan``. output of a ``scan``.
* A *map* operation can be performed by applying a function that * A *map* operation can be performed by applying a function that
ignores previous steps of the outputs. ignores previous steps of the outputs.
Often a for-loop can be expressed as a ``scan()`` operation, and ``scan`` is Often a for-loop or while-loop can be expressed as a ``scan()`` operation,
the closest that theano comes to looping. The advantage of using ``scan`` and ``scan`` is the closest that theano comes to looping. The advantages
over for loops is that it allows the number of iterations to be a part of of using ``scan`` over `for` loops in python (amongs other) are:
the symbolic graph.
* it allows the number of iterations to be part of the symbolic graph
* it allows computing gradients through the for loop
* there exist a bunch of optimizations that help re-write your loop
such that less memory is used and that it runs faster
* it ensures that data is not copied from host to gpu and gpu to
host at each step
The Scan Op should typically be used by calling any of the following The Scan Op should typically be used by calling any of the following
functions: ``scan()``, ``map()``, ``reduce()``, ``foldl()``, functions: ``scan()``, ``map()``, ``reduce()``, ``foldl()``,
...@@ -41,8 +47,8 @@ __authors__ = ( ...@@ -41,8 +47,8 @@ __authors__ = (
__copyright__ = "(c) 2010, Universite de Montreal" __copyright__ = "(c) 2010, Universite de Montreal"
__contact__ = "Razvan Pascanu <r.pascanu@gmail>" __contact__ = "Razvan Pascanu <r.pascanu@gmail>"
from theano.scan_module import scan_opt from theano.scan import opt
from theano.scan_module.scan import scan from theano.scan.basic import scan
from theano.scan_module.scan_checkpoints import scan_checkpoints from theano.scan.checkpoints import scan_checkpoints
from theano.scan_module.scan_utils import clone, until from theano.scan.utils import clone, until
from theano.scan_module.scan_views import foldl, foldr, map, reduce from theano.scan.views import foldl, foldr, map, reduce
theano/scan_module/scan.py → theano/scan/basic.py
浏览文件 @ 6d39faaf
"""
This module provides the Scan Op.
Scanning is a general form of recurrence, which can be used for looping.
The idea is that you *scan* a function along some input sequence, producing
an output at each time-step that can be seen (but not modified) by the
function at the next time-step. (Technically, the function can see the
previous K time-steps of your outputs and L time steps (from past and
future) of your inputs.
So for example, ``sum()`` could be computed by scanning the ``z+x_i``
function over a list, given an initial state of ``z=0``.
Special cases:
* A *reduce* operation can be performed by using only the last
output of a ``scan``.
* A *map* operation can be performed by applying a function that
ignores previous steps of the outputs.
Often a for-loop or while-loop can be expressed as a ``scan()`` operation,
and ``scan`` is the closest that theano comes to looping. The advantages
of using ``scan`` over `for` loops in python (amongs other) are:
* it allows the number of iterations to be part of the symbolic graph
* it allows computing gradients through the for loop
* there exist a bunch of optimizations that help re-write your loop
such that less memory is used and that it runs faster
* it ensures that data is not copied from host to gpu and gpu to
host at each step
The Scan Op should typically be used by calling any of the following
functions: ``scan()``, ``map()``, ``reduce()``, ``foldl()``,
``foldr()``.
"""
__docformat__ = "restructedtext en" __docformat__ = "restructedtext en"
__authors__ = "Razvan Pascanu " "Frederic Bastien " "James Bergstra " "Pascal Lamblin " __authors__ = "Razvan Pascanu " "Frederic Bastien " "James Bergstra " "Pascal Lamblin "
__copyright__ = "(c) 2010, Universite de Montreal" __copyright__ = "(c) 2010, Universite de Montreal"
...@@ -53,13 +15,14 @@ from theano.compile import SharedVariable, ops ...@@ -53,13 +15,14 @@ from theano.compile import SharedVariable, ops
from theano.compile.function import function from theano.compile.function import function
from theano.compile.mode import Mode from theano.compile.mode import Mode
from theano.gof.utils import TestValueError from theano.gof.utils import TestValueError
from theano.scan_module import scan_op, scan_utils from theano.scan import utils
from theano.scan_module.scan_utils import safe_new, traverse from theano.scan.op import Scan
from theano.scan.utils import safe_new, traverse
from theano.tensor import opt from theano.tensor import opt
from theano.updates import OrderedUpdates from theano.updates import OrderedUpdates
_logger = logging.getLogger("theano.scan_module.scan") _logger = logging.getLogger("theano.scan.basic")
def scan( def scan(
...@@ -167,7 +130,7 @@ def scan( ...@@ -167,7 +130,7 @@ def scan(
.. code-block:: python .. code-block:: python
... ...
return [y1_t, y2_t], {x:x+1}, theano.scan_module.until(x < 50) return [y1_t, y2_t], {x:x+1}, until(x < 50)
Note that a number of steps (considered in here as the maximum Note that a number of steps (considered in here as the maximum
number of steps ) is still required even though a condition is number of steps ) is still required even though a condition is
...@@ -576,7 +539,7 @@ def scan( ...@@ -576,7 +539,7 @@ def scan(
for seq in scan_seqs: for seq in scan_seqs:
lengths_vec.append(seq.shape[0]) lengths_vec.append(seq.shape[0])
if not scan_utils.isNaN_or_Inf_or_None(n_steps): if not utils.isNaN_or_Inf_or_None(n_steps):
# ^ N_steps should also be considered # ^ N_steps should also be considered
lengths_vec.append(tt.as_tensor(n_steps)) lengths_vec.append(tt.as_tensor(n_steps))
...@@ -591,7 +554,7 @@ def scan( ...@@ -591,7 +554,7 @@ def scan(
# If the user has provided the number of steps, do that regardless ( and # If the user has provided the number of steps, do that regardless ( and
# raise an error if the sequences are not long enough ) # raise an error if the sequences are not long enough )
if scan_utils.isNaN_or_Inf_or_None(n_steps): if utils.isNaN_or_Inf_or_None(n_steps):
actual_n_steps = lengths_vec[0] actual_n_steps = lengths_vec[0]
for contestant in lengths_vec[1:]: for contestant in lengths_vec[1:]:
actual_n_steps = tt.minimum(actual_n_steps, contestant) actual_n_steps = tt.minimum(actual_n_steps, contestant)
...@@ -671,7 +634,7 @@ def scan( ...@@ -671,7 +634,7 @@ def scan(
# the initial state over. We do this using the expand function # the initial state over. We do this using the expand function
# defined in scan utils # defined in scan utils
sit_sot_scan_inputs.append( sit_sot_scan_inputs.append(
scan_utils.expand_empty( utils.expand_empty(
tt.unbroadcast(tt.shape_padleft(actual_arg), 0), tt.unbroadcast(tt.shape_padleft(actual_arg), 0),
actual_n_steps, actual_n_steps,
) )
...@@ -695,7 +658,7 @@ def scan( ...@@ -695,7 +658,7 @@ def scan(
mit_sot_tap_array.append(init_out["taps"]) mit_sot_tap_array.append(init_out["taps"])
# Sequence # Sequence
mit_sot_scan_inputs.append( mit_sot_scan_inputs.append(
scan_utils.expand_empty(init_out["initial"][:mintap], actual_n_steps) utils.expand_empty(init_out["initial"][:mintap], actual_n_steps)
) )
if i in return_steps: if i in return_steps:
...@@ -783,7 +746,7 @@ def scan( ...@@ -783,7 +746,7 @@ def scan(
# when we apply the lambda expression we get a mixture of update rules # when we apply the lambda expression we get a mixture of update rules
# and outputs that needs to be separated # and outputs that needs to be separated
condition, outputs, updates = scan_utils.get_updates_and_outputs(fn(*args)) condition, outputs, updates = utils.get_updates_and_outputs(fn(*args))
if condition is not None: if condition is not None:
as_while = True as_while = True
else: else:
...@@ -838,7 +801,7 @@ def scan( ...@@ -838,7 +801,7 @@ def scan(
if condition is not None: if condition is not None:
outputs.append(condition) outputs.append(condition)
fake_nonseqs = [x.type() for x in non_seqs] fake_nonseqs = [x.type() for x in non_seqs]
fake_outputs = scan_utils.clone( fake_outputs = utils.clone(
outputs, replace=OrderedDict(zip(non_seqs, fake_nonseqs)) outputs, replace=OrderedDict(zip(non_seqs, fake_nonseqs))
) )
all_inputs = filter( all_inputs = filter(
...@@ -927,7 +890,7 @@ def scan( ...@@ -927,7 +890,7 @@ def scan(
if isinstance(new_var.type, ops.expandable_types): if isinstance(new_var.type, ops.expandable_types):
sit_sot_inner_inputs.append(new_var) sit_sot_inner_inputs.append(new_var)
sit_sot_scan_inputs.append( sit_sot_scan_inputs.append(
scan_utils.expand_empty( utils.expand_empty(
tt.unbroadcast(tt.shape_padleft(input.variable), 0), tt.unbroadcast(tt.shape_padleft(input.variable), 0),
actual_n_steps, actual_n_steps,
) )
...@@ -1065,7 +1028,7 @@ def scan( ...@@ -1065,7 +1028,7 @@ def scan(
else: else:
new_givens = givens new_givens = givens
new_outs = scan_utils.clone(inner_outs, replace=new_givens) new_outs = utils.clone(inner_outs, replace=new_givens)
## ##
# Step 7. Create the Scan Op # Step 7. Create the Scan Op
...@@ -1095,7 +1058,7 @@ def scan( ...@@ -1095,7 +1058,7 @@ def scan(
info["allow_gc"] = allow_gc info["allow_gc"] = allow_gc
info["strict"] = strict info["strict"] = strict
local_op = scan_op.Scan(inner_inputs, new_outs, info) local_op = Scan(inner_inputs, new_outs, info)
## ##
# Step 8. Compute the outputs using the scan op # Step 8. Compute the outputs using the scan op
......
theano/scan_module/c_code/scan_perform.c → theano/scan/c_code/scan_perform.c
浏览文件 @ 6d39faaf
This source diff could not be displayed because it is too large. You can view the blob instead.
theano/scan_module/scan_op.py → theano/scan/op.py
浏览文件 @ 6d39faaf
""" """This module provides the `Scan` `Op`.
This module provides the Scan Op.
See scan.py for details on scan.
Memory reuse in scan Memory reuse in scan
-------------------- --------------------
...@@ -58,6 +54,7 @@ import numpy as np ...@@ -58,6 +54,7 @@ import numpy as np
import theano import theano
from theano import compile, config, gof, gradient, tensor from theano import compile, config, gof, gradient, tensor
from theano.compile.builders import infer_shape
from theano.compile.function import function from theano.compile.function import function
from theano.compile.io import In, Out from theano.compile.io import In, Out
from theano.compile.mode import AddFeatureOptimizer from theano.compile.mode import AddFeatureOptimizer
...@@ -66,8 +63,7 @@ from theano.gof import Apply, PureOp ...@@ -66,8 +63,7 @@ from theano.gof import Apply, PureOp
from theano.gof.graph import equal_computations, io_connection_pattern from theano.gof.graph import equal_computations, io_connection_pattern
from theano.gof.toolbox import NoOutputFromInplace from theano.gof.toolbox import NoOutputFromInplace
from theano.gradient import DisconnectedType, NullType, grad_undefined from theano.gradient import DisconnectedType, NullType, grad_undefined
from theano.scan_module import scan_utils from theano.scan.utils import Validator, forced_replace, hash_listsDictsTuples, safe_new
from theano.scan_module.scan_utils import forced_replace, safe_new
from theano.tensor import TensorType, as_tensor_variable from theano.tensor import TensorType, as_tensor_variable
from theano.tensor.opt import Shape_i from theano.tensor.opt import Shape_i
...@@ -78,7 +74,7 @@ __copyright__ = "(c) 2010, Universite de Montreal" ...@@ -78,7 +74,7 @@ __copyright__ = "(c) 2010, Universite de Montreal"
__contact__ = "Razvan Pascanu <r.pascanu@gmail>" __contact__ = "Razvan Pascanu <r.pascanu@gmail>"
# Logging function for sending warning or info # Logging function for sending warning or info
_logger = logging.getLogger("theano.scan_module.scan_op") _logger = logging.getLogger("theano.scan.op")
class Scan(PureOp): class Scan(PureOp):
...@@ -805,7 +801,7 @@ class Scan(PureOp): ...@@ -805,7 +801,7 @@ class Scan(PureOp):
# and a hash representing the inner graph using the # and a hash representing the inner graph using the
# CLinker.cmodule_key_ # CLinker.cmodule_key_
self._hash_inner_graph, self._hash_inner_graph,
scan_utils.hash_listsDictsTuples(self.info), hash_listsDictsTuples(self.info),
) )
) )
...@@ -1795,13 +1791,13 @@ class Scan(PureOp): ...@@ -1795,13 +1791,13 @@ class Scan(PureOp):
self_outs = self.outputs[:-1] self_outs = self.outputs[:-1]
else: else:
self_outs = self.outputs self_outs = self.outputs
outs_shape = scan_utils.infer_shape( outs_shape = infer_shape(
outs=self_outs, inputs=self.inputs, input_shapes=inner_ins_shapes outs=self_outs, inputs=self.inputs, input_shapes=inner_ins_shapes
) )
# Will be used to check if outs_shape can be expressed without using # Will be used to check if outs_shape can be expressed without using
# variables in self.inputs. # variables in self.inputs.
# The shapes of node.inputs are valid. # The shapes of node.inputs are valid.
validator = scan_utils.Validator( validator = Validator(
valid=input_shapes, invalid=self.inputs, valid_equivalent=out_equivalent valid=input_shapes, invalid=self.inputs, valid_equivalent=out_equivalent
) )
...@@ -2830,9 +2826,7 @@ class Scan(PureOp): ...@@ -2830,9 +2826,7 @@ class Scan(PureOp):
self_outputs = self.outputs self_outputs = self.outputs
# Step 1. Compute the R_op of the inner function # Step 1. Compute the R_op of the inner function
inner_eval_points = [ inner_eval_points = [safe_new(x, "_evalpoint") for x in rop_of_inputs]
scan_utils.safe_new(x, "_evalpoint") for x in rop_of_inputs
]
if self.as_while: if self.as_while:
rop_self_outputs = self_outputs[:-1] rop_self_outputs = self_outputs[:-1]
else: else:
...@@ -3077,7 +3071,7 @@ def profile_printer( ...@@ -3077,7 +3071,7 @@ def profile_printer(
"If you enable Theano profiler with " "If you enable Theano profiler with "
"'theano.function(..., profile=True)', you must manually" "'theano.function(..., profile=True)', you must manually"
" enable the profiling for each scan too: " " enable the profiling for each scan too: "
"'theano.scan_module.scan(...,profile=True)'." "'theano.scan(...,profile=True)'."
" Or use Theano flag 'profile=True'.", " Or use Theano flag 'profile=True'.",
file=file, file=file,
) )
......
theano/scan_module/scan_opt.py → theano/scan/opt.py
浏览文件 @ 6d39faaf
...@@ -64,8 +64,16 @@ from theano.compile.function.types import deep_copy_op ...@@ -64,8 +64,16 @@ from theano.compile.function.types import deep_copy_op
from theano.gof import DestroyHandler, InconsistencyError, toolbox from theano.gof import DestroyHandler, InconsistencyError, toolbox
from theano.gof.graph import equal_computations from theano.gof.graph import equal_computations
from theano.gof.opt import Optimizer, pre_constant_merge, pre_greedy_local_optimizer from theano.gof.opt import Optimizer, pre_constant_merge, pre_greedy_local_optimizer
from theano.scan_module import scan_op, scan_utils from theano.scan.op import Scan
from theano.scan_module.scan_utils import scan_args from theano.scan.utils import (
clone,
compress_outs,
expand_empty,
reconstruct_graph,
safe_new,
scan_args,
scan_can_remove_outs,
)
from theano.tensor import Alloc, AllocEmpty, get_scalar_constant_value, opt from theano.tensor import Alloc, AllocEmpty, get_scalar_constant_value, opt
...@@ -82,7 +90,7 @@ __contact__ = "Razvan Pascanu <r.pascanu@gmail>" ...@@ -82,7 +90,7 @@ __contact__ = "Razvan Pascanu <r.pascanu@gmail>"
# Logging function for sending warning or info # Logging function for sending warning or info
_logger = logging.getLogger("theano.scan_module.scan_opt") _logger = logging.getLogger("theano.scan.opt")
list_opt_slice = [ list_opt_slice = [
tensor.opt.local_abs_merge, tensor.opt.local_abs_merge,
...@@ -101,7 +109,7 @@ def info(*msg): ...@@ -101,7 +109,7 @@ def info(*msg):
_logger.info("INFO theano.scan: " + " ".join(msg)) _logger.info("INFO theano.scan: " + " ".join(msg))
@gof.local_optimizer([scan_op.Scan]) @gof.local_optimizer([Scan])
def remove_constants_and_unused_inputs_scan(node): def remove_constants_and_unused_inputs_scan(node):
""" """
Move constants into the inner graph, and remove unused inputs. Move constants into the inner graph, and remove unused inputs.
...@@ -113,7 +121,7 @@ def remove_constants_and_unused_inputs_scan(node): ...@@ -113,7 +121,7 @@ def remove_constants_and_unused_inputs_scan(node):
not on initial states. not on initial states.
""" """
if not isinstance(node.op, scan_op.Scan): if not isinstance(node.op, Scan):
return False return False
op = node.op op = node.op
# We only need to take care of sequences and other arguments # We only need to take care of sequences and other arguments
...@@ -203,11 +211,11 @@ def remove_constants_and_unused_inputs_scan(node): ...@@ -203,11 +211,11 @@ def remove_constants_and_unused_inputs_scan(node):
nw_outer.extend(nw_outer_nonseq) nw_outer.extend(nw_outer_nonseq)
if len(nw_inner) != len(op_ins): if len(nw_inner) != len(op_ins):
op_outs = scan_utils.clone(op_outs, replace=givens) op_outs = clone(op_outs, replace=givens)
nw_info = copy.deepcopy(op.info) nw_info = copy.deepcopy(op.info)
nw_info["n_seqs"] = nw_n_seqs nw_info["n_seqs"] = nw_n_seqs
# DEBUG CHECK # DEBUG CHECK
nwScan = scan_op.Scan(nw_inner, op_outs, nw_info) nwScan = Scan(nw_inner, op_outs, nw_info)
nw_outs = nwScan(*nw_outer, **dict(return_list=True)) nw_outs = nwScan(*nw_outer, **dict(return_list=True))
return OrderedDict([("remove", [node])] + list(zip(node.outputs, nw_outs))) return OrderedDict([("remove", [node])] + list(zip(node.outputs, nw_outs)))
else: else:
...@@ -229,7 +237,7 @@ class PushOutNonSeqScan(gof.Optimizer): ...@@ -229,7 +237,7 @@ class PushOutNonSeqScan(gof.Optimizer):
fgraph.attach_feature(gof.toolbox.ReplaceValidate()) fgraph.attach_feature(gof.toolbox.ReplaceValidate())
def apply(self, fgraph): def apply(self, fgraph):
nodelist = [x for x in fgraph.toposort() if isinstance(x.op, scan_op.Scan)] nodelist = [x for x in fgraph.toposort() if isinstance(x.op, Scan)]
for node in nodelist: for node in nodelist:
self.process_node(fgraph, node) self.process_node(fgraph, node)
...@@ -242,9 +250,7 @@ class PushOutNonSeqScan(gof.Optimizer): ...@@ -242,9 +250,7 @@ class PushOutNonSeqScan(gof.Optimizer):
""" """
# this flag tells if there was any change during the last iterations # this flag tells if there was any change during the last iterations
clean_inputs, clean_outputs = scan_utils.reconstruct_graph( clean_inputs, clean_outputs = reconstruct_graph(node.op.inputs, node.op.outputs)
node.op.inputs, node.op.outputs
)
local_fgraph_topo = theano.gof.graph.io_toposort(clean_inputs, clean_outputs) local_fgraph_topo = theano.gof.graph.io_toposort(clean_inputs, clean_outputs)
local_fgraph_outs_set = set(clean_outputs) local_fgraph_outs_set = set(clean_outputs)
...@@ -331,7 +337,7 @@ class PushOutNonSeqScan(gof.Optimizer): ...@@ -331,7 +337,7 @@ class PushOutNonSeqScan(gof.Optimizer):
# Step 2. Create variables for replacements # Step 2. Create variables for replacements
for idx, y in enumerate(nd.outputs): for idx, y in enumerate(nd.outputs):
y_place_holder = scan_utils.safe_new(y, "_replace") y_place_holder = safe_new(y, "_replace")
add_to_replace(y) add_to_replace(y)
replace_with_in.append(y_place_holder) replace_with_in.append(y_place_holder)
assert isinstance(y, type(nw_outer_node.outputs[idx])) assert isinstance(y, type(nw_outer_node.outputs[idx]))
...@@ -377,11 +383,11 @@ class PushOutNonSeqScan(gof.Optimizer): ...@@ -377,11 +383,11 @@ class PushOutNonSeqScan(gof.Optimizer):
nw_outer.append(repl_out) nw_outer.append(repl_out)
givens[to_repl] = repl_in givens[to_repl] = repl_in
op_outs = scan_utils.clone(clean_outputs, replace=givens) op_outs = clone(clean_outputs, replace=givens)
op_ins = clean_inputs + nw_inner op_ins = clean_inputs + nw_inner
# Reconstruct node # Reconstruct node
nwScan = scan_op.Scan(op_ins, op_outs, op.info) nwScan = Scan(op_ins, op_outs, op.info)
# Do not call make_node for test_value # Do not call make_node for test_value
nw_node = nwScan(*(node.inputs + nw_outer), **dict(return_list=True))[ nw_node = nwScan(*(node.inputs + nw_outer), **dict(return_list=True))[
...@@ -447,7 +453,7 @@ class PushOutSeqScan(gof.Optimizer): ...@@ -447,7 +453,7 @@ class PushOutSeqScan(gof.Optimizer):
fgraph.attach_feature(gof.toolbox.ReplaceValidate()) fgraph.attach_feature(gof.toolbox.ReplaceValidate())
def apply(self, fgraph): def apply(self, fgraph):
nodelist = [x for x in fgraph.toposort() if isinstance(x.op, scan_op.Scan)] nodelist = [x for x in fgraph.toposort() if isinstance(x.op, Scan)]
for node in nodelist: for node in nodelist:
self.process_node(fgraph, node) self.process_node(fgraph, node)
...@@ -460,9 +466,7 @@ class PushOutSeqScan(gof.Optimizer): ...@@ -460,9 +466,7 @@ class PushOutSeqScan(gof.Optimizer):
""" """
# this flag tells if there was any change during the last iterations # this flag tells if there was any change during the last iterations
clean_inputs, clean_outputs = scan_utils.reconstruct_graph( clean_inputs, clean_outputs = reconstruct_graph(node.op.inputs, node.op.outputs)
node.op.inputs, node.op.outputs
)
local_fgraph_topo = theano.gof.graph.io_toposort(clean_inputs, clean_outputs) local_fgraph_topo = theano.gof.graph.io_toposort(clean_inputs, clean_outputs)
local_fgraph_outs_set = set(clean_outputs) local_fgraph_outs_set = set(clean_outputs)
...@@ -554,7 +558,7 @@ class PushOutSeqScan(gof.Optimizer): ...@@ -554,7 +558,7 @@ class PushOutSeqScan(gof.Optimizer):
# Step 2. Create variables for replacements # Step 2. Create variables for replacements
for idx, y in enumerate(nd.outputs): for idx, y in enumerate(nd.outputs):
y_place_holder = scan_utils.safe_new(y, "_replace") y_place_holder = safe_new(y, "_replace")
add_to_replace(y) add_to_replace(y)
replace_with_in.append(y_place_holder) replace_with_in.append(y_place_holder)
replace_with_out.append(nw_outer_node.outputs[idx]) replace_with_out.append(nw_outer_node.outputs[idx])
...@@ -582,7 +586,7 @@ class PushOutSeqScan(gof.Optimizer): ...@@ -582,7 +586,7 @@ class PushOutSeqScan(gof.Optimizer):
new_ord += (old_ord + 1,) new_ord += (old_ord + 1,)
new_outer = outside_ins.dimshuffle(new_ord) new_outer = outside_ins.dimshuffle(new_ord)
y = nd.outputs[0] y = nd.outputs[0]
y_place_holder = scan_utils.safe_new(y, "_replace") y_place_holder = safe_new(y, "_replace")
add_to_replace(y) add_to_replace(y)
replace_with_in.append(y_place_holder) replace_with_in.append(y_place_holder)
replace_with_out.append(new_outer) replace_with_out.append(new_outer)
...@@ -638,13 +642,13 @@ class PushOutSeqScan(gof.Optimizer): ...@@ -638,13 +642,13 @@ class PushOutSeqScan(gof.Optimizer):
givens[to_repl] = repl_in givens[to_repl] = repl_in
op_outs = scan_utils.clone(clean_outputs, replace=givens) op_outs = clone(clean_outputs, replace=givens)
op_ins = nw_inner + clean_inputs op_ins = nw_inner + clean_inputs
# Reconstruct node # Reconstruct node
nw_info = op.info.copy() nw_info = op.info.copy()
nw_info["n_seqs"] += len(nw_inner) nw_info["n_seqs"] += len(nw_inner)
nwScan = scan_op.Scan(op_ins, op_outs, nw_info) nwScan = Scan(op_ins, op_outs, nw_info)
# Do not call make_node for test_value # Do not call make_node for test_value
nw_node = nwScan( nw_node = nwScan(
*(node.inputs[:1] + nw_outer + node.inputs[1:]), *(node.inputs[:1] + nw_outer + node.inputs[1:]),
...@@ -712,7 +716,7 @@ class PushOutScanOutput(gof.Optimizer): ...@@ -712,7 +716,7 @@ class PushOutScanOutput(gof.Optimizer):
nodelist = [ nodelist = [
x x
for x in fgraph.toposort() for x in fgraph.toposort()
if (isinstance(x.op, scan_op.Scan) and not x.op.as_while) if (isinstance(x.op, Scan) and not x.op.as_while)
] ]
for node in nodelist: for node in nodelist:
# Process the node as long as something gets optimized # Process the node as long as something gets optimized
...@@ -925,7 +929,7 @@ class PushOutScanOutput(gof.Optimizer): ...@@ -925,7 +929,7 @@ class PushOutScanOutput(gof.Optimizer):
new_scan_args.outer_in_nit_sot.extend(new_nitsots_initial_value) new_scan_args.outer_in_nit_sot.extend(new_nitsots_initial_value)
# Create the scan op from the scan_args # Create the scan op from the scan_args
new_scan_op = scan_op.Scan( new_scan_op = Scan(
new_scan_args.inner_inputs, new_scan_args.inner_outputs, new_scan_args.info new_scan_args.inner_inputs, new_scan_args.inner_outputs, new_scan_args.info
) )
...@@ -1027,9 +1031,7 @@ class ScanInplaceOptimizer(Optimizer): ...@@ -1027,9 +1031,7 @@ class ScanInplaceOptimizer(Optimizer):
else: else:
typeConstructor = self.typeInfer(node) typeConstructor = self.typeInfer(node)
new_op = scan_op.Scan( new_op = Scan(op.inputs, op.outputs, info, typeConstructor=typeConstructor)
op.inputs, op.outputs, info, typeConstructor=typeConstructor
)
# Do not call make_node for test_value # Do not call make_node for test_value
new_outs = new_op(*inputs, **dict(return_list=True)) new_outs = new_op(*inputs, **dict(return_list=True))
...@@ -1062,7 +1064,7 @@ class ScanInplaceOptimizer(Optimizer): ...@@ -1062,7 +1064,7 @@ class ScanInplaceOptimizer(Optimizer):
scan_nodes = [ scan_nodes = [
x x
for x in nodes for x in nodes
if (isinstance(x.op, scan_op.Scan) and x.op.info["gpua"] == self.gpua_flag) if (isinstance(x.op, Scan) and x.op.info["gpua"] == self.gpua_flag)
] ]
for scan_idx in range(len(scan_nodes)): for scan_idx in range(len(scan_nodes)):
...@@ -1438,9 +1440,7 @@ class ScanSaveMem(gof.Optimizer): ...@@ -1438,9 +1440,7 @@ class ScanSaveMem(gof.Optimizer):
nw_inputs[0] = nw_steps nw_inputs[0] = nw_steps
# 3.2 check orphane outputs to see if we can eliminate any # 3.2 check orphane outputs to see if we can eliminate any
required, not_required = scan_utils.scan_can_remove_outs( required, not_required = scan_can_remove_outs(node.op, orphane_outs)
node.op, orphane_outs
)
# 3.3. compose replace pairs for those nodes that need not # 3.3. compose replace pairs for those nodes that need not
# to store everything in memory ( or ar orphane and required # to store everything in memory ( or ar orphane and required
# by the inner function .. ) # by the inner function .. )
...@@ -1485,7 +1485,7 @@ class ScanSaveMem(gof.Optimizer): ...@@ -1485,7 +1485,7 @@ class ScanSaveMem(gof.Optimizer):
tmp = pre_greedy_local_optimizer(list_opt_slice, tmp_idx) tmp = pre_greedy_local_optimizer(list_opt_slice, tmp_idx)
tmp = pre_constant_merge([tmp])[0] tmp = pre_constant_merge([tmp])[0]
nw_input = scan_utils.expand_empty(_nw_input, tmp) nw_input = expand_empty(_nw_input, tmp)
else: else:
tmp = tensor.as_tensor_variable(val) tmp = tensor.as_tensor_variable(val)
initl = tensor.as_tensor_variable(init_l[i]) initl = tensor.as_tensor_variable(init_l[i])
...@@ -1539,7 +1539,7 @@ class ScanSaveMem(gof.Optimizer): ...@@ -1539,7 +1539,7 @@ class ScanSaveMem(gof.Optimizer):
) )
): ):
_nw_input = nw_inputs[in_idx].owner.inputs[1] _nw_input = nw_inputs[in_idx].owner.inputs[1]
nw_input = scan_utils.expand_empty(_nw_input, nw_steps) nw_input = expand_empty(_nw_input, nw_steps)
nw_inputs[in_idx] = nw_input nw_inputs[in_idx] = nw_input
else: else:
nw_input = nw_inputs[in_idx][: (initl + nw_steps)] nw_input = nw_inputs[in_idx][: (initl + nw_steps)]
...@@ -1550,7 +1550,7 @@ class ScanSaveMem(gof.Optimizer): ...@@ -1550,7 +1550,7 @@ class ScanSaveMem(gof.Optimizer):
nw_inputs[in_idx] = nw_steps nw_inputs[in_idx] = nw_steps
# 3.5 Remove unwanted orphane outputs # 3.5 Remove unwanted orphane outputs
(inps, outs, info, node_ins, compress_map) = scan_utils.compress_outs( (inps, outs, info, node_ins, compress_map) = compress_outs(
op, not_required, nw_inputs op, not_required, nw_inputs
) )
inv_compress_map = OrderedDict() inv_compress_map = OrderedDict()
...@@ -1568,9 +1568,7 @@ class ScanSaveMem(gof.Optimizer): ...@@ -1568,9 +1568,7 @@ class ScanSaveMem(gof.Optimizer):
return return
# Do not call make_node for test_value # Do not call make_node for test_value
new_outs = scan_op.Scan(inps, outs, info)( new_outs = Scan(inps, outs, info)(*node_ins, **dict(return_list=True))
*node_ins, **dict(return_list=True)
)
old_new = [] old_new = []
# 3.7 Get replace pairs for those outputs that do not change # 3.7 Get replace pairs for those outputs that do not change
...@@ -1677,7 +1675,7 @@ class ScanSaveMem(gof.Optimizer): ...@@ -1677,7 +1675,7 @@ class ScanSaveMem(gof.Optimizer):
def apply(self, fgraph): def apply(self, fgraph):
nodelist = [x for x in fgraph.toposort() if isinstance(x.op, scan_op.Scan)] nodelist = [x for x in fgraph.toposort() if isinstance(x.op, Scan)]
for node in nodelist: for node in nodelist:
self.process_node(fgraph, node) self.process_node(fgraph, node)
...@@ -1796,7 +1794,7 @@ class ScanMerge(gof.Optimizer): ...@@ -1796,7 +1794,7 @@ class ScanMerge(gof.Optimizer):
flat_inner_ins = sum(inner_ins[idx], []) flat_inner_ins = sum(inner_ins[idx], [])
flat_inner_outs = sum(inner_outs[idx], []) flat_inner_outs = sum(inner_outs[idx], [])
# clone # clone
flat_inner_ins, flat_inner_outs = scan_utils.reconstruct_graph( flat_inner_ins, flat_inner_outs = reconstruct_graph(
flat_inner_ins, flat_inner_outs flat_inner_ins, flat_inner_outs
) )
# split the new inner variables again in seq, mitmot, etc. # split the new inner variables again in seq, mitmot, etc.
...@@ -1846,7 +1844,7 @@ class ScanMerge(gof.Optimizer): ...@@ -1846,7 +1844,7 @@ class ScanMerge(gof.Optimizer):
else: else:
new_inner_outs += inner_outs[idx][gr_idx] new_inner_outs += inner_outs[idx][gr_idx]
new_op = scan_op.Scan(new_inner_ins, new_inner_outs, info) new_op = Scan(new_inner_ins, new_inner_outs, info)
new_outs = new_op(*outer_ins) new_outs = new_op(*outer_ins)
if not isinstance(new_outs, (list, tuple)): if not isinstance(new_outs, (list, tuple)):
...@@ -1902,7 +1900,7 @@ class ScanMerge(gof.Optimizer): ...@@ -1902,7 +1900,7 @@ class ScanMerge(gof.Optimizer):
def apply(self, fgraph): def apply(self, fgraph):
# Collect all scan nodes ordered according to toposort # Collect all scan nodes ordered according to toposort
scan_nodes = [nd for nd in fgraph.toposort() if isinstance(nd.op, scan_op.Scan)] scan_nodes = [nd for nd in fgraph.toposort() if isinstance(nd.op, Scan)]
# All sets of possibly mergeable nodes # All sets of possibly mergeable nodes
all_sets = [] all_sets = []
...@@ -1958,9 +1956,9 @@ def make_equiv(lo, li): ...@@ -1958,9 +1956,9 @@ def make_equiv(lo, li):
return left, right return left, right
@gof.local_optimizer([scan_op.Scan]) @gof.local_optimizer([Scan])
def scan_merge_inouts(node): def scan_merge_inouts(node):
if not isinstance(node.op, scan_op.Scan): if not isinstance(node.op, Scan):
return False return False
# Do a first pass to merge identical external inputs. # Do a first pass to merge identical external inputs.
...@@ -2004,9 +2002,9 @@ def scan_merge_inouts(node): ...@@ -2004,9 +2002,9 @@ def scan_merge_inouts(node):
outer_inputs = a.outer_inputs outer_inputs = a.outer_inputs
info = a.info info = a.info
a_inner_outs = a.inner_outputs a_inner_outs = a.inner_outputs
inner_outputs = scan_utils.clone(a_inner_outs, replace=inp_equiv) inner_outputs = clone(a_inner_outs, replace=inp_equiv)
op = scan_op.Scan(inner_inputs, inner_outputs, info) op = Scan(inner_inputs, inner_outputs, info)
outputs = op(*outer_inputs) outputs = op(*outer_inputs)
if not isinstance(outputs, (list, tuple)): if not isinstance(outputs, (list, tuple)):
...@@ -2152,7 +2150,7 @@ class PushOutDot1(gof.Optimizer): ...@@ -2152,7 +2150,7 @@ class PushOutDot1(gof.Optimizer):
def apply(self, fgraph): def apply(self, fgraph):
nodes = fgraph.toposort() nodes = fgraph.toposort()
scan_nodes = [x for x in nodes if (isinstance(x.op, scan_op.Scan))] scan_nodes = [x for x in nodes if (isinstance(x.op, Scan))]
for node in scan_nodes: for node in scan_nodes:
self.apply_opt(fgraph, node) self.apply_opt(fgraph, node)
...@@ -2260,10 +2258,10 @@ class PushOutDot1(gof.Optimizer): ...@@ -2260,10 +2258,10 @@ class PushOutDot1(gof.Optimizer):
+ inner_nitsot_outs + inner_nitsot_outs
+ inner_shared_outs + inner_shared_outs
) )
new_inner_inps, new_inner_outs = scan_utils.reconstruct_graph( new_inner_inps, new_inner_outs = reconstruct_graph(
_new_inner_inps, _new_inner_outs _new_inner_inps, _new_inner_outs
) )
new_op = scan_op.Scan(new_inner_inps, new_inner_outs, new_info) new_op = Scan(new_inner_inps, new_inner_outs, new_info)
_scan_inputs = ( _scan_inputs = (
[node.inputs[0]] [node.inputs[0]]
+ outer_seqs + outer_seqs
......
theano/scan_module/scan_perform_ext.py → theano/scan/scan_perform_ext.py
浏览文件 @ 6d39faaf
""" """
To update the scan Cython code in Theano you must To update the scan Cython code in Theano you must
- update the version in this file and scan_perform.py - update the version in this file and scan_perform.py
- call "cd theano/scan_module/; cython scan_perform.pyx; patch scan_perform.c numpy_api_changes.diff" - call "cd theano/scan/; cython scan_perform.pyx; patch scan_perform.c numpy_api_changes.diff"
""" """
...@@ -21,7 +21,7 @@ from theano.gof import cmodule ...@@ -21,7 +21,7 @@ from theano.gof import cmodule
from theano.gof.compilelock import get_lock, release_lock from theano.gof.compilelock import get_lock, release_lock
_logger = logging.getLogger("theano.scan_module.scan_perform") _logger = logging.getLogger("theano.scan.scan_perform")
version = 0.296 # must match constant returned in function get_version() version = 0.296 # must match constant returned in function get_version()
...@@ -68,9 +68,7 @@ except ImportError: ...@@ -68,9 +68,7 @@ except ImportError:
raise ImportError("no c compiler, can't compile cython code") raise ImportError("no c compiler, can't compile cython code")
_logger.info("Compiling C code for scan") _logger.info("Compiling C code for scan")
dirname = "scan_perform" dirname = "scan_perform"
cfile = os.path.join( cfile = os.path.join(theano.__path__[0], "scan", "c_code", "scan_perform.c")
theano.__path__[0], "scan_module", "c_code", "scan_perform.c"
)
if not os.path.exists(cfile): if not os.path.exists(cfile):
# This can happen in not normal case. We just # This can happen in not normal case. We just
# disable the cython code. If we are here the user # disable the cython code. If we are here the user
......
theano/scan_module/scan_utils.py → theano/scan/utils.py
浏览文件 @ 6d39faaf
""" """This module provides utility functions for the `Scan` `Op`."""
This module provides utility functions for the Scan Op.
See scan.py for details on scan.
"""
__docformat__ = "restructedtext en" __docformat__ = "restructedtext en"
__authors__ = ( __authors__ = (
...@@ -33,7 +27,7 @@ from theano.tensor.basic import get_scalar_constant_value ...@@ -33,7 +27,7 @@ from theano.tensor.basic import get_scalar_constant_value
# Logging function for sending warning or info # Logging function for sending warning or info
_logger = logging.getLogger("theano.scan_utils") _logger = logging.getLogger("theano.scan.utils")
def safe_new(x, tag="", dtype=None): def safe_new(x, tag="", dtype=None):
...@@ -323,8 +317,8 @@ def map_variables(replacer, graphs, additional_inputs=None): ...@@ -323,8 +317,8 @@ def map_variables(replacer, graphs, additional_inputs=None):
return False return False
# importing Scan into module scope would be circular # importing Scan into module scope would be circular
from theano.compile import OpFromGraph from theano.compile.builders import OpFromGraph
from theano.scan_module.scan_op import Scan from theano.scan.op import Scan
if isinstance(node.op, (Scan, OpFromGraph)): if isinstance(node.op, (Scan, OpFromGraph)):
# recurse on the inner graph # recurse on the inner graph
...@@ -388,7 +382,7 @@ def _map_variables_inner( ...@@ -388,7 +382,7 @@ def _map_variables_inner(
from itertools import chain from itertools import chain
from theano import gof from theano import gof
from theano.scan_module import scan_utils from theano.scan import utils
def inner_replacer(graph): def inner_replacer(graph):
new_graph = replacer(graph) new_graph = replacer(graph)
...@@ -439,7 +433,7 @@ def _map_variables_inner( ...@@ -439,7 +433,7 @@ def _map_variables_inner(
# as an inner input, connect it through a new # as an inner input, connect it through a new
# inner input # inner input
if outer_input not in outer_to_inner.keys(): if outer_input not in outer_to_inner.keys():
inner_input = scan_utils.safe_new(outer_input, tag="_copy") inner_input = utils.safe_new(outer_input, tag="_copy")
outer_to_inner[outer_input] = inner_input outer_to_inner[outer_input] = inner_input
extra_inner_inputs.append(inner_input) extra_inner_inputs.append(inner_input)
extra_outer_inputs.append(outer_input) extra_outer_inputs.append(outer_input)
...@@ -502,7 +496,7 @@ def get_updates_and_outputs(ls): ...@@ -502,7 +496,7 @@ def get_updates_and_outputs(ls):
return False return False
def is_condition(elem): def is_condition(elem):
return isinstance(elem, theano.scan_module.until) return isinstance(elem, until)
def _list(x): def _list(x):
if isinstance(x, (list, tuple)): if isinstance(x, (list, tuple)):
...@@ -515,7 +509,7 @@ def get_updates_and_outputs(ls): ...@@ -515,7 +509,7 @@ def get_updates_and_outputs(ls):
Ensure `x` is made only of allowed data types. Ensure `x` is made only of allowed data types.
Return True iff `x` is made only of lists, tuples, dictionaries, Theano Return True iff `x` is made only of lists, tuples, dictionaries, Theano
variables or `theano.scan_module.until` objects. variables or `theano.scan.utils.until` objects.
""" """
# Is `x` a container we can iterate on? # Is `x` a container we can iterate on?
...@@ -527,15 +521,13 @@ def get_updates_and_outputs(ls): ...@@ -527,15 +521,13 @@ def get_updates_and_outputs(ls):
if iter_on is not None: if iter_on is not None:
return all(_filter(y) for y in iter_on) return all(_filter(y) for y in iter_on)
else: else:
return isinstance(x, theano.Variable) or isinstance( return isinstance(x, theano.Variable) or isinstance(x, until)
x, theano.scan_module.until
)
if not _filter(ls): if not _filter(ls):
raise ValueError( raise ValueError(
"The return value of your scan lambda expression may only be " "The return value of your scan lambda expression may only be "
"made of lists, tuples, or dictionaries containing Theano " "made of lists, tuples, or dictionaries containing Theano "
"variables (or `theano.scan_module.until` objects for " "variables (or `theano.scan.utils.until` objects for "
"conditions). In particular if you need to use constant " "conditions). In particular if you need to use constant "
"values, you can use `tensor.constant` to turn them into " "values, you can use `tensor.constant` to turn them into "
"Theano variables." "Theano variables."
...@@ -633,59 +625,6 @@ def expand_empty(tensor_var, size): ...@@ -633,59 +625,6 @@ def expand_empty(tensor_var, size):
return ret return ret
def infer_shape(outs, inputs, input_shapes):
"""
Compute the shape of the outputs given the shape of the inputs of a theano
graph.
We do it this way to avoid compiling the inner function just to get
the shape. Changes to ShapeFeature could require changes in this function.
"""
# We use a ShapeFeature because it has all the necessary logic
# inside. We don't use the full ShapeFeature interface, but we
# let it initialize itself with an empty fgraph, otherwise we will
# need to do it manually
for inp, inp_shp in zip(inputs, input_shapes):
if inp_shp is not None and len(inp_shp) != inp.ndim:
assert len(inp_shp) == inp.ndim
shape_feature = tensor.opt.ShapeFeature()
shape_feature.on_attach(theano.gof.FunctionGraph([], []))
# Initialize shape_of with the input shapes
for inp, inp_shp in zip(inputs, input_shapes):
shape_feature.set_shape(inp, inp_shp)
def local_traverse(out):
"""
Go back in the graph, from out, adding computable shapes to shape_of.
"""
if out in shape_feature.shape_of:
# Its shape is already known
return
elif out.owner is None:
# This is an input of the graph
shape_feature.init_r(out)
else:
# Recurse over inputs
for inp in out.owner.inputs:
if inp not in shape_feature.shape_of:
local_traverse(inp)
# shape_feature.on_import does not actually use an fgraph
# It will call infer_shape and set_shape appropriately
dummy_fgraph = None
shape_feature.on_import(dummy_fgraph, out.owner, reason="dummy")
ret = []
for o in outs:
local_traverse(o)
ret.append(shape_feature.shape_of[o])
return ret
class Validator: class Validator:
""" """
Check if variables can be expressed without using variables in invalid. Check if variables can be expressed without using variables in invalid.
......
theano/scan_module/scan_views.py → theano/scan/views.py
浏览文件 @ 6d39faaf
""" """This module provides a convenient constructor for the `Scan` `Op`."""
This module provides syntax shortcut for the Scan Op.
See scan.py for details on scan.
"""
__docformat__ = "restructedtext en" __docformat__ = "restructedtext en"
__authors__ = "Razvan Pascanu " "Frederic Bastien " "James Bergstra " "Pascal Lamblin " __authors__ = "Razvan Pascanu " "Frederic Bastien " "James Bergstra " "Pascal Lamblin "
...@@ -14,19 +8,10 @@ __contact__ = "Razvan Pascanu <r.pascanu@gmail>" ...@@ -14,19 +8,10 @@ __contact__ = "Razvan Pascanu <r.pascanu@gmail>"
import logging import logging
from theano.scan_module import scan from theano.scan import scan
# Logging function for sending warning or info
_logger = logging.getLogger("theano.scan_module.scan_views")
#
# Declaration of views for scan
#
_logger = logging.getLogger("theano.scan.views")
# The ``map`` view of Scan Op.
def map( def map(
...@@ -38,8 +23,7 @@ def map( ...@@ -38,8 +23,7 @@ def map(
mode=None, mode=None,
name=None, name=None,
): ):
""" """Construct a `Scan` `Op` that functions like `map`.
Similar behaviour as python's map.
Parameters Parameters
---------- ----------
...@@ -75,7 +59,6 @@ def map( ...@@ -75,7 +59,6 @@ def map(
) )
# The ``reduce`` view of Scan Op.
def reduce( def reduce(
fn, fn,
sequences, sequences,
...@@ -85,8 +68,7 @@ def reduce( ...@@ -85,8 +68,7 @@ def reduce(
mode=None, mode=None,
name=None, name=None,
): ):
""" """Construct a `Scan` `Op` that functions like `reduce`.
Similar behaviour as python's reduce.
Parameters Parameters
---------- ----------
...@@ -128,10 +110,8 @@ def reduce( ...@@ -128,10 +110,8 @@ def reduce(
return rval[0][-1], rval[1] return rval[0][-1], rval[1]
# The ``foldl`` view of Scan Op.
def foldl(fn, sequences, outputs_info, non_sequences=None, mode=None, name=None): def foldl(fn, sequences, outputs_info, non_sequences=None, mode=None, name=None):
""" """Construct a `Scan` `Op` that functions like Haskell's `foldl`.
Similar behaviour as haskell's foldl.
Parameters Parameters
---------- ----------
...@@ -164,10 +144,8 @@ def foldl(fn, sequences, outputs_info, non_sequences=None, mode=None, name=None) ...@@ -164,10 +144,8 @@ def foldl(fn, sequences, outputs_info, non_sequences=None, mode=None, name=None)
) )
# The ``foldl`` view of Scan Op.
def foldr(fn, sequences, outputs_info, non_sequences=None, mode=None, name=None): def foldr(fn, sequences, outputs_info, non_sequences=None, mode=None, name=None):
""" """Construct a `Scan` `Op` that functions like Haskell's `foldr`.
Similar behaviour as haskell' foldr.
Parameters Parameters
---------- ----------
......
theano/tensor/nnet/nnet.py
浏览文件 @ 6d39faaf
...@@ -2137,7 +2137,7 @@ def sigmoid_binary_crossentropy(output, target): ...@@ -2137,7 +2137,7 @@ def sigmoid_binary_crossentropy(output, target):
inp = [output, target] inp = [output, target]
outp = softplus(-abs(output)) + output * ((output > 0) - target) outp = softplus(-abs(output)) + output * ((output > 0) - target)
return theano.OpFromGraph( return theano.compile.builders.OpFromGraph(
inp, inp,
[outp], [outp],
grad_overrides=grad, grad_overrides=grad,
......
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论