Merge pull request #910 from goodfeli/int_grad

doc/extending/op.txt

@@ -98,34 +98,56 @@ following methods:
   lifetime of self.  Op instances should be immutable in this
   sense.
 
-.. function:: connection_pattern():
+.. function:: connection_pattern( node ):
 
-  Optional (but in extremely rare cases needed to have it work with
-   {tensor,sparse}.grad).
+  Optional method; sometimes needed for gradient.grad to
+  work correctly.
 
-  Returns a list of bools the same length as the op's inputs list.
+  Returns a list of list of bools.
 
-  True signifies that the elements of an input have an effect on its
-  output.
+  Op.connection_pattern[input_idx][output_idx] is true if the
+  elements of inputs[input_idx] have an effect on the elements of
+  outputs[output_idx].
 
-  False signifies that they do not--in other words, the op acts only
-  one the input's metadata such as its shape.
+  The ``node'' parameter is needed to determine the number of
+  inputs. Some ops such as Subtensor take a variable number of
+  inputs.
 
-  If no connection_pattern is implemented, tensor.grad will assume
-  it is a list containing only True.
+  If no connection_pattern is specified, gradient.grad will
+  assume that all inputs have some elements connected to some
+  elements of all outputs.
+
+  This method conveys two pieces of information that are otherwise
+  not part of the theano graph:
+
+  1) Which of the op's inputs are truly ancestors of each of the
+     op's outputs. Suppose an op has two inputs, x and y, and
+     outputs f(x) and g(y). y is not really an ancestor of f, but
+     it appears to be so in the theano graph.
+  2) Whether the actual elements of each input/output are relevant
+     to a computation.
+     For example, the shape op does not read its input's elements,
+     only its shape metadata. d shape(x) / dx should thus raise
+     a disconnected input exception (if these exceptions are
+     enabled).
+     As another example, the elements of the Alloc op's outputs
+     are not affected by the shape arguments to the Alloc op.
 
   Failing to implement this function for an op that needs it can
-  result in tensor.grad erroneously reporting that a gradient is
-  undefined. Returning 0 for this input in the grad method is not
-  the same as specifying that the elements of this input are not
-  connected to the output. If the gradient with respect to the
-  op's output is NaN but the elements of the input are not connected
-  to it, then the NaN never enters into the expression for the
-  gradient.
+  result in two types of incorrect behavior:
+  
+  1) gradient.grad erroneously raising a TypeError reporting that
+     a gradient is undefined.
+  2) gradient.grad failing to raise a ValueError reporting that
+     an input is disconnected.
+
+  Even if connection_pattern is not implemented correctly,
+  if gradient.grad returns an expression, that expression will
+  be numerically correct.
 
 .. function:: grad(inputs, output_gradients)
 
-  Optional (but needed to have it work with {tensor,sparse}.grad()).
+  Optional (but needed to have it work with gradient.grad()).
 
   If the Op being defined is differentiable, its gradient may be specified 
   symbolically in this method. Both ``inputs`` and ``output_gradients``
@@ -217,6 +239,70 @@ following methods:
   Both the partial differentiation and the multiplication have to be performed by
   :func:`grad`.
 
+
+  Theano currently imposes the following constraints on the values returned by the grad method:
+  
+  1) They must be Variable instances.
+  2) When they are types that have dtypes, they must never have an integer dtype.
+
+  Integers are a tricky subject. Integers are the main reason for having DisconnectedType,
+  NullType or zero gradient. When you have an integer as an argument to your grad method,
+  recall the definition of a derivative to help you decide what value to return:
+
+  :math:`\frac{d f}{d x} = \lim_{\epsilon \rightarrow 0} (f(x+\epsilon)-f(x))/\epsilon`.
+
+  Suppose your function f has an integer-valued output. For most functions you're likely
+  to implement in theano, this means your gradient should be zero, because f(x+epsilon)
+  = f(x) for almost all x. (The only other option is that the gradient could be undefined,
+  if your function is discontinuous everywhere, like the rational indicator function)
+
+  Suppose your function f has an integer-valued input. This is a little trickier, because
+  you need to think about what you mean mathematically when you make a variable integer-valued
+  in theano. Most of the time in machine learning we mean "f is a function of a real-valued
+  x, but we are only going to pass in integer-values of x". In this case, f(x+epsilon) exists,
+  so the gradient through f should be the same whether x is an integer or a floating point
+  variable. Sometimes what we mean is "f is a function of an integer-valued x, and f is only
+  defined where x is an integer." Since f(x+epsilon) doesn't exist, the gradient is undefined.
+  Finally, many times in theano, integer valued inputs don't actually affect the elements of
+  the output, only its shape.
+
+  If your function f has both an integer-valued input and an
+  integer-valued output, then both rules have to be combined:
+
+  - If f is defined at (x+epsilon), then the input gradient is
+    defined. Since f(x+epsilon) would be equal to f(x) almost
+    everywhere, the gradient should be 0 (first rule).
+
+  - If f is only defined where x is an integer, then the gradient
+    is undefined, regardless of what the gradient with respect to the
+    output is.
+
+  Examples:
+
+  1) f(x,y) = dot product between x and y. x and y are integers.
+        Since the output is also an integer, f is a step function.
+        Its gradient is zero almost everywhere, so Op.grad should return
+        zeros in the shape of x and y.
+  2) f(x,y) = dot product between x and y. x is floating point and y is an integer.
+        In this case the output is floating point. It doesn't matter that y is an integer.
+        We consider f to still be defined at f(x,y+epsilon). The gradient is exactly the
+        same as if y were floating point.
+  3) f(x,y) = argmax of x along axis y.
+        The gradient with respect to y is undefined, because f(x,y) is not defined for
+        floating point y. How could you take an argmax along a fraActional axis?
+        The gradient with respect to x is 0, because f(x+epsilon, y) = f(x) almost
+        everywhere.
+  4) f(x,y) = a vector with y elements, each of which taking on the value x
+        The grad method should return DisconnectedType()() for y, because the elements of
+        f don't depend on y. Only the shape of f depends on y. You probably also want to
+        implement a connection_pattern method to encode this.
+  5) f(x) = int(x) converts float x into an int. g(y) = float(y) converts an integer y into a float.
+        If the final cost C = 0.5 * g(y) = 0.5 g(f(x)), then the
+        gradient with respect to y will be 0.5, even if y is an
+        integer. However, the gradient with respect to x will be 0,
+        because the output of f is integer-valued.
+
+
 .. function:: infer_shape(node, shapes)
 
    Optional.

theano/gof/null_type.py

@@ -29,3 +29,9 @@ class NullType(Type):
 
     def values_eq(a, b, force_same_dtype=True):
         raise ValueError("NullType has no values to compare")
+
+    def __eq__(self, other):
+        return type(self) == type(other)
+
+    def __hash__(self, other):
+        return hash(type(self))

theano/gradient.py

@@ -213,51 +213,68 @@ def Rop(f, wrt, eval_points):
 
     def _traverse(node):
         """ TODO: writeme """
+
         if node is None:
-            return None
-        else:
-            op = node.op
-            inputs = node.inputs
+            return
 
-            # Compute the evaluation points corresponding to each of the
-            # inputs of the node
-            local_eval_points = []
-            for inp in inputs:
-                if inp in wrt:
-                    local_eval_points.append(eval_points[wrt.index(inp)])
-                elif inp.owner is None:
-                    try:
-                        local_eval_points.append(inp.zeros_like())
-                    except:
-                        # None should be used for non-differentiable
-                        # arguments, like for example random states
-                        local_eval_points.append(None)
-                elif inp.owner in seen_nodes:
-
-                    local_eval_points.append(
-                        seen_nodes[inp.owner][inp.owner.outputs.index(inp)])
+        op = node.op
+        inputs = node.inputs
+
+        # Compute the evaluation points corresponding to each of the
+        # inputs of the node
+        local_eval_points = []
+        for inp in inputs:
+            if inp in wrt:
+                local_eval_points.append(eval_points[wrt.index(inp)])
+            elif inp.owner is None:
+                try:
+                    local_eval_points.append(inp.zeros_like())
+                except:
+                    # None should be used for non-differentiable
+                    # arguments, like for example random states
+                    local_eval_points.append(None)
+            elif inp.owner in seen_nodes:
+
+                local_eval_points.append(
+                    seen_nodes[inp.owner][inp.owner.outputs.index(inp)])
 
-                else:
-                    # We actually need to compute the R_op for this node
-
-                    _traverse(inp.owner)
-                    local_eval_points.append(
-                        seen_nodes[inp.owner][inp.owner.outputs.index(inp)])
-            same_type_eval_points = []
-            for x, y in zip(inputs, local_eval_points):
-                if y is not None:
-                    if not isinstance(x, gof.Variable):
-                        x = as_tensor_variable(x)
-                    if not isinstance(y, gof.Variable):
-                        y = as_tensor_variable(y)
+            else:
+                # We actually need to compute the R_op for this node
+
+                _traverse(inp.owner)
+                local_eval_points.append(
+                    seen_nodes[inp.owner][inp.owner.outputs.index(inp)])
+        same_type_eval_points = []
+        for x, y in zip(inputs, local_eval_points):
+            if y is not None:
+                if not isinstance(x, gof.Variable):
+                    x = as_tensor_variable(x)
+                if not isinstance(y, gof.Variable):
+                    y = as_tensor_variable(y)
+                try:
                     y = x.type.filter_variable(y)
-                    assert x.type == y.type
-                    same_type_eval_points.append(y)
-                else:
-                    same_type_eval_points.append(y)
+                except TypeError:
+                    # This is a hack
+                    # Originally both grad and Rop were written
+                    # with the assumption that a variable and the
+                    # gradient wrt that variable would have the same
+                    # dtype. This was a bad assumption because the
+                    # gradient wrt an integer can take on non-integer
+                    # values.
+                    # grad is now fixed, but Rop is not, so when grad
+                    # does the right thing and violates this assumption
+                    # we have to make it be wrong for Rop to keep working
+                    # Rop should eventually be upgraded to handle integers
+                    # correctly, the same as grad
+                    y = theano.tensor.cast(y, x.type.dtype)
+                    y = x.type.filter_variable(y)
+                assert x.type == y.type
+                same_type_eval_points.append(y)
+            else:
+                same_type_eval_points.append(y)
 
-            seen_nodes[node] = op.R_op(node.inputs, same_type_eval_points)
-            return None
+        seen_nodes[node] = op.R_op(node.inputs, same_type_eval_points)
+    #end _traverse
 
     # Populate the dictionary
     for out in f:
@@ -276,7 +293,7 @@ def Rop(f, wrt, eval_points):
     return format_as(using_list, using_tuple, rval)
 
 
-def Lop(f, wrt, eval_points, consider_constant=None, warn_type=False,
+def Lop(f, wrt, eval_points, consider_constant=None,
         disconnected_inputs='raise'):
     """
     Computes the L operation on `f` wrt to `wrt` evaluated at points given
@@ -329,8 +346,7 @@ def Lop(f, wrt, eval_points, consider_constant=None, warn_type=False,
 
     gmap = grad_sources_inputs(
         arg1,
-        arg2,
-        warn_type=warn_type)
+        arg2)
 
     # Note : If p is not in gmap there can be several reasons, among which
     # is the fact that p might not be part of the computational graph. A
@@ -369,7 +385,7 @@ def Lop(f, wrt, eval_points, consider_constant=None, warn_type=False,
 # Gradient
 #########################
 
-def grad(cost, wrt, g_cost=None, consider_constant=None, warn_type=False,
+def grad(cost, wrt, g_cost=None, consider_constant=None,
         disconnected_inputs='raise', add_names=True):
     """
     :type cost: Scalar (0-dimensional) Variable.
@@ -380,9 +396,6 @@ def grad(cost, wrt, g_cost=None, consider_constant=None, warn_type=False,
     :param consider_constant: a list of expressions not to backpropagate
         through
 
-    :param warn_type: a value of True will cause warnings to be logged for any
-        Op that emits a gradient that does not match its input type.
-
     :type disconnected_inputs: string
     :param disconnected_inputs: Defines the behaviour if some of the variables
         in ``wrt`` are not part of the computational graph computing ``cost``
@@ -438,13 +451,13 @@ def grad(cost, wrt, g_cost=None, consider_constant=None, warn_type=False,
     if not using_list and not using_tuple:
         wrt = [wrt]
 
-    var_to_node_to_idx = _populate_var_to_node_to_idx([cost])
+    var_to_node_to_idx = _populate_var_to_node_to_idx([cost], wrt)
 
     # build a dict mapping var to the gradient of cost with respect to var
     grad_dict = {}
     # by default, the gradient of the cost is 1
     if g_cost is None:
-        g_cost = tensor.ones_like(cost)
+        g_cost = _float_ones_like(cost)
     grad_dict[cost] = g_cost
 
     # the gradient of the constants is 0
@@ -477,13 +490,18 @@ def grad(cost, wrt, g_cost=None, consider_constant=None, warn_type=False,
     if add_names:
         cost_name = cost.name
 
+    # Make sure we didn't initialize the grad_dict with any ints
+    for var in grad_dict:
+        g = grad_dict[var]
+        if hasattr(g.type, 'dtype'):
+            assert g.type.dtype.find('float') != -1
+
     rval = _populate_grad_dict(var_to_node_to_idx,
-            grad_dict, wrt, warn_type,
-            cost_name)
+            grad_dict, wrt, cost_name)
 
     for i in xrange(len(rval)):
         if isinstance(rval[i].type, DisconnectedType):
-            rval[i] = wrt[i].zeros_like()
+            rval[i] = _float_zeros_like(wrt[i])
 
     if using_tuple:
         rval = tuple(rval)
@@ -492,25 +510,79 @@ def grad(cost, wrt, g_cost=None, consider_constant=None, warn_type=False,
     return rval
 
 
-def _populate_var_to_node_to_idx(outputs):
+def _node_to_pattern(node):
+    """ given an apply node, obtain its connection pattern
+     this is just a wrapper around Op.connection_pattern
+     that does type checking and supplies the default value
+     if the method is not implemented
     """
-        Common code shared between grad and grad_sources_inputs
 
-        outputs: a list of nodes we want to take gradients of
+    if hasattr(node.op, 'connection_pattern'):
+        connection_pattern = node.op.connection_pattern(node)
+
+        if not isinstance(connection_pattern, list):
+            raise TypeError("Op.connection_pattern should return " + \
+                    ("list of list of bool, but for Op=%s" % node.op) +\
+                    "got %s with type %s." % (connection_pattern,
+                        type(connection_pattern)))
+        if len(connection_pattern) != len(node.inputs):
+            raise ValueError('%s.connection_pattern should have %d' %
+                    (node.op, len(node.inputs)) + ' rows but has %d.' %
+                    len(connection_pattern))
+        for ii, output_pattern in enumerate(connection_pattern):
+            if not isinstance(output_pattern, list):
+                raise TypeError('%s.connection_pattern should return' %
+                        node.op + ' a list of lists, but element %d' % ii\
+                        + 'is %s of type %s.' % (output_pattern,
+                            type(output_pattern)))
+    else:
+        connection_pattern = \
+            [[True for output in node.outputs]
+                    for ipt in node.inputs]
+    assert isinstance(connection_pattern, list)
+    assert len(connection_pattern) == len(node.inputs)
+    for ii in xrange(len(node.inputs)):
+        assert isinstance(connection_pattern[ii], list)
+        assert len(connection_pattern[ii]) == \
+                len(node.outputs)
+    return connection_pattern
+
+
+def _populate_var_to_node_to_idx(outputs, wrt):
+    """
+    Common code shared between grad and grad_sources_inputs
 
-        returns:
-            var_to_node_to_idx: a dictionary mapping a variable to
-                a second dictionary.
-                the second dictionary maps apply nodes acting on
-                this variable to the variable's index in the apply
-                node's input list
+    outputs: a list of variables we want to take gradients of
+
+    wrt: a list of variables we want to take the gradient with
+        respect to.
+
+    returns:
+        var_to_node_to_idx: a dictionary mapping a variable to
+            a second dictionary.
+            the second dictionary maps apply nodes acting on
+            this variable to the variable's index in the apply
+            node's input list
+            This dictionary will only contain variables that
+            meet two criteria:
+                1) The elements of at least one output are a
+                   function of the elements of the variable
+                2) The elements of the variable are a function
+                   of the elements of at least one member of
+                   wrt
+            This set is exactly the set of variables that
+            connect the variables in wrt to the cost being
+            differentiated.
 
     """
 
     # var_to_node_to_idx[var][node] = [i,j] means node has
     # var as input at positions i and j
     var_to_node_to_idx = {}
-    # set of variables or nodes that have been added to their parents
+    # set of variables or nodes that have been added to their true parents
+    # ('true' here means that the elements of the variable are a function
+    #  of the elements of the parent, according to the op's
+    #  connection_pattern)
     accounted_for = set([])
 
     def account_for(var):
@@ -521,7 +593,18 @@ def _populate_var_to_node_to_idx(outputs):
             node = var.owner
             if node not in accounted_for:
                 accounted_for.add(node)
+
+                connection_pattern = _node_to_pattern(node)
+
+                var_idx = node.outputs.index(var)
+
                 for i, ipt in enumerate(node.inputs):
+
+                    #don't process ipt if it is not a true
+                    #parent of var
+                    if not connection_pattern[i][var_idx]:
+                        continue
+
                     if ipt not in var_to_node_to_idx:
                         var_to_node_to_idx[ipt] = {}
                     node_to_idx = var_to_node_to_idx[ipt]
@@ -532,14 +615,43 @@ def _populate_var_to_node_to_idx(outputs):
                     idx.append(i)
                     account_for(ipt)
 
+    # add all variables that are true ancestors of the cost
     for output in outputs:
         account_for(output)
 
+    # determine which variables have elements of wrt as a true
+    # ancestor. Do this with an upward pass starting from wrt,
+    # following only true connections
+    visited = set([])
+
+    def visit(var):
+        if var in visited:
+            return
+        if var not in var_to_node_to_idx:
+            return
+        visited.add(var)
+        nodes = var_to_node_to_idx[var]
+        for node in nodes:
+            connection_pattern = _node_to_pattern(node)
+            for idx in nodes[node]:
+                for ii, output in enumerate(node.outputs):
+                    if connection_pattern[idx][ii]:
+                        visit(output)
+
+    for elem in wrt:
+        visit(elem)
+
+    # Remove variables that don't have wrt as a true ancestor
+    orig_vars = list(var_to_node_to_idx.keys())
+    for var in orig_vars:
+        if var not in visited:
+            del var_to_node_to_idx[var]
+
     return var_to_node_to_idx
 
 
 def _populate_grad_dict(var_to_node_to_idx,
-        grad_dict, wrt, warn_type, cost_name=None):
+        grad_dict, wrt, cost_name=None):
     """
         Common code shared between grad_sources_inputs and grad
 
@@ -561,9 +673,6 @@ def _populate_grad_dict(var_to_node_to_idx,
 
         wrt: the minimal set of variables that must be included in grad_dict
 
-        warn_type: if True, log a warning when a gradient term for a variable
-                    has a different type from that variable
-
         cost_name: The name of the cost being differentiated, optional.
                     used to name the grad with respect to x as
                     (d<cost_name>/dx)
@@ -575,36 +684,50 @@ def _populate_grad_dict(var_to_node_to_idx,
     # its inputs' gradients
     term_dict = {}
 
-    # populate term_dict[node] and return it
     def access_term_cache(node):
+        """ Populates term_dict[node] and returns it """
+
         if node not in term_dict:
 
             inputs = node.inputs
 
-            # Each Op's grad function requires inputs and output_grads
-            # If the Op destroys any input, but the grad expression uses it,
-            # then chances are the resulting graph will have a dependency
-            # cycle. We avoid this cycle by passing (symbolic) copies of
-            # each destroyed input.
-            try:
-                dinputs = [node.inputs[x[0]] for x in
-                        node.op.destroy_map.values()]
-            except AttributeError:
-                dinputs = []
-
-            def try_to_copy_if_needed(var):
-                if var in dinputs and hasattr(var, 'copy'):
-                    return var.copy()
-                return var
-
-            inputs = [try_to_copy_if_needed(ipt) for ipt in inputs]
-
             output_grads = [access_grad_cache(var) for var in node.outputs]
 
-            if False in [isinstance(g.type, DisconnectedType)
-                    for g in output_grads]:
-                # Some outputs of this op are connected to the cost so we must
-                # call the ops grad method
+            # list of bools indicating if each output is connected to the cost
+            outputs_connected = [not isinstance(g.type, DisconnectedType)
+                    for g in output_grads]
+
+            connection_pattern = _node_to_pattern(node)
+
+            # list of bools indicating if each input is connected to the cost
+            inputs_connected = [
+                    (True in [input_to_output and output_to_cost for
+                        input_to_output, output_to_cost in
+                        zip(input_to_outputs, outputs_connected)]) for
+                        input_to_outputs in connection_pattern
+                    ]
+
+            if True in inputs_connected:
+                # At least one input of this op is connected to the cost so we must
+                # call the op's grad method
+
+                # Each Op's grad function requires inputs and output_grads
+                # If the Op destroys any input, but the grad expression uses it,
+                # then chances are the resulting graph will have a dependency
+                # cycle. We avoid this cycle by passing (symbolic) copies of
+                # each destroyed input.
+                try:
+                    dinputs = [node.inputs[x[0]] for x in
+                            node.op.destroy_map.values()]
+                except AttributeError:
+                    dinputs = []
+
+                def try_to_copy_if_needed(var):
+                    if var in dinputs and hasattr(var, 'copy'):
+                        return var.copy()
+                    return var
+
+                inputs = [try_to_copy_if_needed(ipt) for ipt in inputs]
 
                 input_grads = node.op.grad(inputs, output_grads)
 
@@ -625,33 +748,141 @@ def _populate_grad_dict(var_to_node_to_idx,
 
             # must convert to list in case the op returns a tuple
             # we won't be able to post-process out the Nones if it does that
-            term_dict[node] = list(input_grads)
-
-            for i in xrange(len(term_dict[node])):
-
-                if term_dict[node][i] is None:
-                    # we don't know what None means. in the past it has been
-                    # used to
-                    # mean undefined, zero, or disconnected. So for now we
-                    # assume it is
-                    # zero. Assuming it is zero prevents
-                    # us from disconnecting NaNs above.
-                    # eventually we should disallow this
-                    # return type and force all ops
-                    # to return the correct thing
-                    # raise AssertionError('%s returned None for' +\
-                    # ' a gradient term, '
-                    #        'this is prohibited' % node.op)
-                    term_dict[node][i] = node.inputs[i].zeros_like()
-
-                if warn_type:
-                    g_r_type = term_dict[node][i].type
-                    r_type = inputs[i].type
-                    if g_r_type != r_type:
-                        _logger.warning(
-                            '%s.grad returned a different type (%s) '
-                            'for input %i of type (%s)',
-                            node.op, g_r_type, i, r_type)
+            input_grads = list(input_grads)
+
+            # Do type checking on the result
+
+            #List of bools indicating if each output is an integer dtype
+            output_is_int = [hasattr(output.type, 'dtype') and
+                    output.type.dtype.find('int') != -1
+                    for output in node.outputs]
+
+            #List of bools indicating if each input only has integer outputs
+            only_connected_to_int = [(True not in
+                [in_to_out and out_to_cost and not out_int
+                    for in_to_out, out_to_cost, out_int in
+                    zip(in_to_outs, outputs_connected, output_is_int)])
+                for in_to_outs in connection_pattern]
+
+            for i, term in enumerate(input_grads):
+
+                # Disallow Nones
+                if term is None:
+                    # We don't know what None means. in the past it has been
+                    # used to mean undefined, zero, or disconnected.
+                    # We therefore don't allow it because its usage has become
+                    # so muddied.
+                    raise TypeError(('%s.grad returned None for' +\
+                             ' a gradient term, '
+                            'this is prohibited. Instead of None,'
+                            'return zeros_like(input), DisconnectedType()(),'
+                            ' or a NullType variable such as those made with '
+                            'the grad_undefined or grad_unimplemented helper '
+                            'functions.') % node.op)
+
+                if not isinstance(term.type,
+                        (NullType, DisconnectedType)):
+                    if term.type.dtype.find('float') == -1:
+                        raise TypeError(str(node.op) + '.grad illegally '
+                                ' returned an integer-valued variable.'
+                                ' (Input index %d, dtype %s)' % (i,
+                                    term.type.dtype))
+                    if only_connected_to_int[i]:
+                        # This term has only integer outputs and we know
+                        # it's not undefined or disconnected
+                        # The only other valid thing it can be is 0
+
+                        no_constant_value = True
+                        try:
+                            constant_value = tensor.get_constant_value(term)
+                            no_constant_value = False
+                        except TypeError:
+                            pass
+
+                        extra_msg = ''
+
+                        # The above won't work if it's a sparse type, handle sparse
+                        # types here
+                        if no_constant_value:
+                            if isinstance(term.type, theano.sparse.SparseType):
+                                if term.owner is not None and isinstance(term.owner.op,
+                                        theano.sparse.CSM):
+                                    data = term.owner.inputs[0]
+                                    try:
+                                        constant_value = tensor.get_constant_value(data)
+                                        no_constant_value = False
+                                    except TypeError:
+                                        print theano.printing.min_informative_str(data)
+                                        extra_msg += " It is a CSM, but its data isn't constant."
+                                        pass
+                                else:
+                                    extra_msg += " It is a SparseType but theano doesn't know how"
+                                    extra_msg += " to turn it into a constant."
+                                #end if CSM
+                            else:
+                                extra_msg += " It is not a SparseType."
+                            #end if SparseType
+                        #end if no_constant_value
+
+                        if no_constant_value:
+                            msg = "%s.grad returned %s of type %s for input"
+                            msg += " %d. This input's only connections to "
+                            msg += "the cost through this op are via "
+                            msg += "integer-valued outputs so it should be "
+                            msg += "NullType, DisconnectedType, or some form "
+                            msg += "of zeros. It is not NullType or "
+                            msg += "DisconnectedType and theano can't "
+                            msg += "simplify it to a constant, so it's not "
+                            msg += "verifiably zeros."
+                            msg += extra_msg
+
+                            msg = msg % (str(node.op), str(term),
+                                    str(type(term)), i)
+
+                            raise ValueError(msg)
+                        if constant_value != 0:
+                            msg = "%s.grad returned %s of type %s for input"
+                            msg += " %d. Since this input is only connected "
+                            msg += "to integer-valued outputs, it should "
+                            msg += "evaluate to zeros, but it evaluates to"
+                            msg += "%s."
+
+                            msg % (str(node.op), str(term), str(type(term)),
+                                    i, str(constant_value))
+
+                            raise ValueError(msg)
+
+            #Check that op.connection_pattern matches the connectivity
+            #logic driving the op.grad method
+            for i, packed in \
+                enumerate(zip(inputs, input_grads, inputs_connected)):
+                ipt, ig, connected = packed
+                actually_connected = \
+                    not isinstance(ig.type, DisconnectedType)
+
+                if actually_connected and not connected:
+                    msg = "%s.grad returned %s of type %s for input %d."
+                    msg += " Expected DisconnectedType instance based on "
+                    msg += " the output of the op's connection_pattern "
+                    msg += "method."
+                    msg = msg % (str(node.op), str(ig), str(ig.type), i)
+                    raise TypeError(msg)
+
+                if connected and not actually_connected:
+                    msg = "%s.grad returned DisconnectedType for input"
+                    msg += " %d."
+                    msg = msg % (str(node.op), i)
+                    if hasattr(node.op, 'connection_pattern'):
+                        msg += ' Its connection_pattern method does not'
+                        msg += ' allow this.'
+                        raise TypeError(msg)
+                    else:
+                        msg += ' You may want to implement a '
+                        msg += 'connection_pattern method for it.'
+                        warnings.warn(msg)
+
+            #cache the result
+            term_dict[node] = input_grads
 
         return term_dict[node]
 
@@ -664,11 +895,6 @@ def _populate_grad_dict(var_to_node_to_idx,
                 for node in node_to_idx:
                     for idx in node_to_idx[node]:
 
-                        if hasattr(node.op, 'connection_pattern'):
-                            pattern = node.op.connection_pattern()
-                            if not pattern[idx]:
-                                continue
-
                         term = access_term_cache(node)[idx]
 
                         if not isinstance(term, gof.Variable):
@@ -681,10 +907,20 @@ def _populate_grad_dict(var_to_node_to_idx,
                                 "encountered a NaN. " +\
                                     term.type.why_null)
 
+                        #Don't try to sum up DisconnectedType placeholders
+                        if isinstance(term.type, DisconnectedType):
+                            continue
+
                         terms.append(term)
-                #the next line is like sum(terms) but doesn't add an
-                #extraneous TensorConstant(0)
-                grad_dict[var] = reduce(lambda x,y: x+y, terms)
+
+                # Add up the terms to get the total gradient on this variable
+                if len(terms) > 0:
+                    # the next line is like sum(terms) but doesn't add an
+                    # extraneous TensorConstant(0)
+                    grad_dict[var] = reduce(lambda x, y: x + y, terms)
+                else:
+                    grad_dict[var] = DisconnectedType()()
+
                 if cost_name is not None and var.name is not None:
                     grad_dict[var].name = '(d%s/d%s)' % (cost_name, var.name)
             else:
@@ -698,7 +934,7 @@ def _populate_grad_dict(var_to_node_to_idx,
     return rval
 
 
-def grad_sources_inputs(sources, graph_inputs, warn_type=True):
+def grad_sources_inputs(sources, graph_inputs):
     """
     Used to compute the gradient of a cost with respect to all the
     variables between graph_input and cost, but in the special
@@ -742,10 +978,6 @@ def grad_sources_inputs(sources, graph_inputs, warn_type=True):
     :type graph_inputs: list of Variable
     :param graph_inputs: variables considered to be constant
         (do not backpropagate through them)
-    :type warn_type: bool
-    :param warn_type: True will trigger warnings via the logging module when
-       the gradient on an expression has a different type than the original
-       expression
 
     :rtype: dictionary whose keys and values are of type Variable
     :return: mapping from each Variable encountered in the backward
@@ -770,7 +1002,7 @@ def grad_sources_inputs(sources, graph_inputs, warn_type=True):
 
     wrt = graph_inputs
 
-    var_to_node_to_idx = _populate_var_to_node_to_idx(outputs)
+    var_to_node_to_idx = _populate_var_to_node_to_idx(outputs, wrt)
 
     # build a dict mapping var to the gradient of cost with respect to var
     grad_dict = {}
@@ -787,17 +1019,41 @@ def grad_sources_inputs(sources, graph_inputs, warn_type=True):
             grad_dict[elem] = DisconnectedType()()
 
     _populate_grad_dict(var_to_node_to_idx,
-            grad_dict, wrt, warn_type)
+            grad_dict, wrt)
 
     # post-process out the DisconnectedTypes
     for key in grad_dict:
         if isinstance(grad_dict[key].type, DisconnectedType):
             if hasattr(key, 'zeros_like'):
-                grad_dict[key] = key.zeros_like()
+                grad_dict[key] = _float_zeros_like(key)
 
     return grad_dict
 
 
+def _float_zeros_like(x):
+    """ Like zeros_like, but forces the object to have a
+    a floating point dtype """
+
+    rval = x.zeros_like()
+
+    if rval.type.dtype.find('float') != -1:
+        return rval
+
+    return rval.astype(theano.config.floatX)
+
+
+def _float_ones_like(x):
+    """ Like ones_like, but forces the object to have a
+    floating point dtype """
+
+    rval = tensor.ones_like(x)
+
+    if rval.type.dtype.find('float') != -1:
+        return rval
+
+    return rval.astype(theano.config.floatX)
+
+
 class numeric_grad(object):
     """
     Compute the numeric derivative of a scalar-valued function at a particular
@@ -1179,7 +1435,7 @@ Exception args: %s""" % (self.err_pos, self.arg,
 verify_grad.E_grad = GradientError
 
 
-def jacobian(expression, wrt, consider_constant=None, warn_type=False,
+def jacobian(expression, wrt, consider_constant=None,
              disconnected_inputs='raise'):
     """
     :type expression: Vector (1-dimensional) Variable
@@ -1188,9 +1444,6 @@ def jacobian(expression, wrt, consider_constant=None, warn_type=False,
     :param consider_constant: a list of expressions not to backpropagate
         through
 
-    :param warn_type: a value of True will cause warnings to be logged for any
-        Op that emits a gradient that does not match its input type.
-
     :type disconnected_inputs: string
     :param disconnected_inputs: Defines the behaviour if some of the variables
         in ``wrt`` are not part of the computational graph computing ``cost``
@@ -1234,7 +1487,6 @@ def jacobian(expression, wrt, consider_constant=None, warn_type=False,
             rval = grad(expr[idx],
                      inp,
                      consider_constant=consider_constant,
-                     warn_type=warn_type,
                      disconnected_inputs=disconnected_inputs)
             rvals.append(rval)
         return rvals
@@ -1252,7 +1504,7 @@ def jacobian(expression, wrt, consider_constant=None, warn_type=False,
     return format_as(using_list, using_tuple, jacobs)
 
 
-def hessian(cost, wrt, consider_constant=None, warn_type=False,
+def hessian(cost, wrt, consider_constant=None,
              disconnected_inputs='raise'):
     """
     :type cost: Scalar (0-dimensional) Variable.
@@ -1262,9 +1514,6 @@ def hessian(cost, wrt, consider_constant=None, warn_type=False,
     :param consider_constant: a list of expressions not to backpropagate
         through
 
-    :param warn_type: a value of True will cause warnings to be logged for any
-        Op that emits a gradient that does not match its input type.
-
     :type disconnected_inputs: string
     :param disconnected_inputs: Defines the behaviour if some of the variables
         in ``wrt`` are not part of the computational graph computing ``cost``
@@ -1307,7 +1556,6 @@ def hessian(cost, wrt, consider_constant=None, warn_type=False,
                             y[i],
                             x,
                             consider_constant=consider_constant,
-                            warn_type=warn_type,
                             disconnected_inputs=disconnected_inputs),
                        sequences=arange(expr.shape[0]),
                        non_sequences=[expr, input])

theano/ifelse.py

@@ -4,8 +4,8 @@ linkers). It resembles the if clause of any programming language, that
 has a `then` and `else` branch, and executes either one or the other
 according to the condition provided.
 
-This op contrast the already existent `switch` op, that will evaluate both
-branches of the clause and afterwards pick (according to the condition)
+This op differs from the already existent `switch` op, that evaluates both
+branches of the clause and afterwards picks (according to the condition)
 which value to report. Note also that `switch` is an elemwise operation (so
 it picks each entry of a matrix according to the condition) while `ifelse`
 is a global operation with a scalar condition.
@@ -60,7 +60,7 @@ class IfElse(PureOp):
 
     :note:
         Other Linkers then CVM and VM are INCOMPATIBLE with this Op, and
-        will ingnore its lazy characteristic, computing both the True and
+        will ignore its lazy characteristic, computing both the True and
         False branch before picking one.
 
     """
@@ -212,7 +212,14 @@ class IfElse(PureOp):
                                        for t in ts])
         if_false = ([ins[0]] + [theano.tensor.zeros_like(f)
                                 for f in fs] + grads)
-        return ([None] +
+
+        condition = ins[0]
+        # condition does affect the elements of the output so it is connected.
+        # For the sake of making the gradient convenient we assume that
+        # condition + epsilon always triggers the same branch as condition
+        condition_grad = condition.zeros_like().astype(theano.config.floatX)
+
+        return ([condition_grad] +
                 if_true_op.make_node(*if_true).outputs +
                 if_false_op.make_node(*if_false).outputs)
 

theano/sandbox/cuda/tests/test_mlp.py

@@ -172,26 +172,27 @@ def run_conv_nnet1(use_gpu):
     if config.mode == 'DEBUG_MODE':
         n_train = 1
 
-    logical_hid_shape = tcn.blas.GpuConv.logical_output_shape_2d(shape_img[2:],shape_kern[2:], 'valid')
+    logical_hid_shape = tcn.blas.GpuConv.logical_output_shape_2d(
+        shape_img[2:], shape_kern[2:], 'valid')
     n_hid = n_kern * logical_hid_shape[0] * logical_hid_shape[1]
     n_out = 10
 
-    w = shared_fn(0.01*(my_rand(*shape_kern)-0.5), 'w')
+    w = shared_fn(0.01 * (my_rand(*shape_kern) - 0.5), 'w')
     b = shared_fn(my_zeros((n_kern,)), 'b')
     v = shared_fn(my_zeros((n_hid, n_out)), 'c')
     c = shared_fn(my_zeros(n_out), 'c')
 
-    x = tensor.Tensor(dtype='float32', broadcastable=(0,1,0,0))('x')
+    x = tensor.Tensor(dtype='float32', broadcastable=(0, 1, 0, 0))('x')
     y = tensor.fmatrix('y')
     lr = tensor.fscalar('lr')
 
     conv_op = conv.ConvOp(shape_img[2:], shape_kern[2:], n_kern, n_batch, 1, 1)
     conv_op.set_flops()
 
-    hid = tensor.tanh(conv_op(x, w)+b.dimshuffle((0,'x','x')))
+    hid = tensor.tanh(conv_op(x, w) + b.dimshuffle((0, 'x', 'x')))
     hid_flat = hid.reshape((n_batch, n_hid))
-    out = tensor.tanh(tensor.dot(hid_flat, v)+c)
-    loss = tensor.sum(0.5 * (out-y)**2 * lr)
+    out = tensor.tanh(tensor.dot(hid_flat, v) + c)
+    loss = tensor.sum(0.5 * (out - y) ** 2 * lr)
     #print 'loss type', loss.type
 
     params = [w, b, v, c]
@@ -200,7 +201,8 @@ def run_conv_nnet1(use_gpu):
     mode = get_mode(use_gpu)
 
     #print 'building pfunc ...'
-    train = pfunc([x,y,lr], [loss], mode=mode, updates=[(p, p-g) for p,g in zip(params, gparams)])
+    train = pfunc([x, y, lr], [loss], mode=mode, updates=[(p, p - g) for p,
+        g in zip(params, gparams)])
 
 #    for i, n in enumerate(train.maker.fgraph.toposort()):
 #        print i, n
@@ -221,10 +223,10 @@ def test_conv_nnet1():
     rval_cpu = run_conv_nnet1(False)
     utt.seed_rng()
     rval_gpu = run_conv_nnet1(True)
-    assert numpy.allclose(rval_cpu, rval_gpu,rtol=1e-4,atol=1e-6)
+    assert numpy.allclose(rval_cpu, rval_gpu, rtol=1e-4, atol=1e-6)
 
 
-def run_conv_nnet2(use_gpu): # pretend we are training LeNet for MNIST
+def run_conv_nnet2(use_gpu):  # pretend we are training LeNet for MNIST
     if use_gpu:
         shared_fn = tcn.shared_constructor
     else:
@@ -239,10 +241,8 @@ def run_conv_nnet2(use_gpu): # pretend we are training LeNet for MNIST
     #n_train=10, n_batch=60, n_kern=10, n_kern1=10, error see of -5.26905e-05
     #n_train=30, n_batch=60, n_kern=10, n_kern1=10, error see of -3.8147e-06
 
-
     #n_train=30, n_batch=60, n_kern=20, n_kern1=10, error see of 6.82771e-05
     #n_train=30, n_batch=60, n_kern=20, n_kern1=30, error see of 0.000231534
-
     n_batch = 60
     shape_img = (n_batch, 1, 32, 32)
 
@@ -252,35 +252,40 @@ def run_conv_nnet2(use_gpu): # pretend we are training LeNet for MNIST
     n_kern1 = 10
     shape_kern1 = (n_kern1, n_kern, 5, 5)
 
-    n_train=30
-    if config.mode=='DEBUG_MODE': n_train=1
+    n_train = 30
+    if config.mode == 'DEBUG_MODE':
+        n_train = 1
 
-    logical_hid_shape = tcn.blas.GpuConv.logical_output_shape_2d(tuple(shape_img[2:]),tuple(shape_kern[2:]), 'valid')
-    logical_hid_shape1 = tcn.blas.GpuConv.logical_output_shape_2d((logical_hid_shape[0]/2, logical_hid_shape[1]/2), tuple(shape_kern1[2:]), 'valid')
+    logical_hid_shape = tcn.blas.GpuConv.logical_output_shape_2d(tuple(
+        shape_img[2:]), tuple(shape_kern[2:]), 'valid')
+    logical_hid_shape1 = tcn.blas.GpuConv.logical_output_shape_2d((
+        logical_hid_shape[0]/2, logical_hid_shape[1]/2), tuple(shape_kern1[2:]), 'valid')
     n_hid = n_kern1 * logical_hid_shape1[0] * logical_hid_shape1[1]
     n_out = 10
 
-    w0 = shared_fn(0.01*(my_rand(*shape_kern)-0.5), 'w0')
+    w0 = shared_fn(0.01 * (my_rand(*shape_kern) - 0.5), 'w0')
     b0 = shared_fn(my_zeros((n_kern,)), 'b0')
-    w1 = shared_fn(0.01*(my_rand(*shape_kern1)-0.5), 'w1')
+    w1 = shared_fn(0.01 * (my_rand(*shape_kern1) - 0.5), 'w1')
     b1 = shared_fn(my_zeros((n_kern1,)), 'b1')
     v = shared_fn(my_zeros((n_hid, n_out)), 'c')
     c = shared_fn(my_zeros(n_out), 'c')
 
-    x = tensor.Tensor(dtype='float32', broadcastable=(0,1,0,0))('x')
+    x = tensor.Tensor(dtype='float32', broadcastable=(0, 1, 0, 0))('x')
     y = tensor.fmatrix('y')
     lr = tensor.fscalar('lr')
 
     conv_op = conv.ConvOp(shape_img[2:], shape_kern[2:], n_kern, n_batch, 1, 1)
-    conv_op1 = conv.ConvOp((n_kern,logical_hid_shape[0]/2, logical_hid_shape[1]/2), shape_kern1[2:], n_kern1, n_batch, 1, 1)
+    conv_op1 = conv.ConvOp((n_kern, logical_hid_shape[0] / 2,
+         logical_hid_shape[1] / 2), shape_kern1[2:], n_kern1, n_batch, 1, 1)
     conv_op.set_flops()
     conv_op1.set_flops()
 
-    hid = tensor.tanh(conv_op(x, w0)+b0.dimshuffle((0,'x','x')))
-    hid1 = tensor.tanh(conv_op1(hid[:,:,::2,::2], w1) + b1.dimshuffle((0,'x','x')))
+    hid = tensor.tanh(conv_op(x, w0) + b0.dimshuffle((0, 'x', 'x')))
+    hid1 = tensor.tanh(conv_op1(hid[:, :, ::2, ::2], w1) + b1.dimshuffle((
+        0, 'x', 'x')))
     hid_flat = hid1.reshape((n_batch, n_hid))
-    out = tensor.tanh(tensor.dot(hid_flat, v)+c)
-    loss = tensor.sum(0.5 * (out-y)**2 * lr)
+    out = tensor.tanh(tensor.dot(hid_flat, v) + c)
+    loss = tensor.sum(0.5 * (out - y) ** 2 * lr)
     #print 'loss type', loss.type
 
     params = [w0, b0, w1, b1, v, c]
@@ -289,13 +294,14 @@ def run_conv_nnet2(use_gpu): # pretend we are training LeNet for MNIST
     mode = get_mode(use_gpu)
 
     #print 'building pfunc ...'
-    train = pfunc([x,y,lr], [loss], mode=mode, updates=[(p, p-g) for p,g in zip(params, gparams)])
+    train = pfunc([x, y, lr], [loss], mode=mode, updates=[(p, p - g) for p,
+        g in zip(params, gparams)])
 
 #    for i, n in enumerate(train.maker.fgraph.toposort()):
 #        print i, n
 
     xval = my_rand(*shape_img)
-    yval = my_rand(n_batch,n_out)#int32 make all 0...
+    yval = my_rand(n_batch, n_out)  # int32 make all 0...
     lr = theano._asarray(0.01, dtype='float32')
     for i in xrange(n_train):
         rval = train(xval, yval, lr)
@@ -311,7 +317,7 @@ def test_conv_nnet2():
         utt.seed_rng()
         rval_cpu = run_conv_nnet2(False)
         #print rval_cpu[0], rval_gpu[0],rval_cpu[0]-rval_gpu[0]
-        assert numpy.allclose(rval_cpu, rval_gpu,rtol=1e-4,atol=1e-4)
+        assert numpy.allclose(rval_cpu, rval_gpu, rtol=1e-4, atol=1e-4)
 
 
 def build_conv_nnet2_classif(use_gpu, isize, ksize, n_batch,
@@ -322,68 +328,71 @@ def build_conv_nnet2_classif(use_gpu, isize, ksize, n_batch,
     else:
         shared_fn = shared
 
-    isize1=isize
-    isize2=isize
-    if isinstance(isize,(tuple,)):
-        isize1=isize[0]
-        isize2=isize[1]
+    isize1 = isize
+    isize2 = isize
+    if isinstance(isize, (tuple, )):
+        isize1 = isize[0]
+        isize2 = isize[1]
 
     shape_img = (n_batch, 1, isize1, isize2)
 
     n_kern = 20  # 6 were used in LeNet5
     shape_kern = (n_kern, 1, ksize, ksize)
 
-    n_kern1 = 30 # 16 were used in LeNet5
+    n_kern1 = 30  # 16 were used in LeNet5
     shape_kern1 = (n_kern1, n_kern, ksize, ksize)
 
-    logical_hid_shape = tcn.blas.GpuConv.logical_output_shape_2d((isize1, isize2), (ksize, ksize), 'valid')
+    logical_hid_shape = tcn.blas.GpuConv.logical_output_shape_2d((
+        isize1, isize2), (ksize, ksize), 'valid')
     logical_hid_shape1 = tcn.blas.GpuConv.logical_output_shape_2d((logical_hid_shape[0]/2,
         logical_hid_shape[1]/2), (ksize, ksize), 'valid')
     n_hid = n_kern1 * logical_hid_shape1[0] * logical_hid_shape1[1]
     n_out = 10
 
-
-    w0 = shared_fn(0.01*(my_rand(*shape_kern)-0.5), 'w0')
+    w0 = shared_fn(0.01 * (my_rand(*shape_kern) - 0.5), 'w0')
     b0 = shared_fn(my_zeros((n_kern,)), 'b0')
-    w1 = shared_fn(0.01*(my_rand(*shape_kern1)-0.5), 'w1')
+    w1 = shared_fn(0.01 * (my_rand(*shape_kern1) - 0.5), 'w1')
     b1 = shared_fn(my_zeros((n_kern1,)), 'b1')
-    v = shared_fn(0.01*my_randn(n_hid, n_out), 'v')
+    v = shared_fn(0.01 * my_randn(n_hid, n_out), 'v')
     c = shared_fn(my_zeros(n_out), 'c')
 
     #print 'ALLOCATING ARCH: w0 shape', w0.get_value(borrow=True).shape
     #print 'ALLOCATING ARCH: w1 shape', w1.get_value(borrow=True).shape
     #print 'ALLOCATING ARCH: v shape', v.get_value(borrow=True).shape
 
-    x = tensor.Tensor(dtype='float32', broadcastable=(0,1,0,0))('x')
+    x = tensor.Tensor(dtype='float32', broadcastable=(0, 1, 0, 0))('x')
     y = tensor.fmatrix('y')
     lr = tensor.fscalar('lr')
 
     conv_op = conv.ConvOp(shape_img[2:], shape_kern[2:], n_kern,
                           n_batch, 1, 1, verbose=verbose, version=version)
     conv_op1 = conv.ConvOp(
-        (n_kern,logical_hid_shape[0]/2, logical_hid_shape[1]/2),
+        (n_kern, logical_hid_shape[0] / 2, logical_hid_shape[1] / 2),
         shape_kern1[2:], n_kern1, n_batch, 1, 1,verbose=verbose, version=version)
     conv_op.set_flops()
     conv_op1.set_flops()
 
-    ds_op = downsample.DownsampleFactorMax((2,2), ignore_border=False)
+    ds_op = downsample.DownsampleFactorMax((2, 2), ignore_border=False)
     if downsample_ops:
-        hid = tensor.tanh(ds_op(conv_op(x, w0)+b0.dimshuffle((0,'x','x'))))
+        hid = tensor.tanh(ds_op(conv_op(x, w0) + b0.dimshuffle((0, 'x', 'x'))))
     else:
-        hid = tensor.tanh((conv_op(x, w0)+b0.dimshuffle((0,'x','x')))[:,:,::2,::2])
-    hid1 = tensor.tanh(conv_op1(hid, w1) + b1.dimshuffle((0,'x','x')))
+        hid = tensor.tanh((conv_op(x, w0) + b0.dimshuffle((0, 'x', 'x')
+            ))[:, :, ::2, ::2])
+    hid1 = tensor.tanh(conv_op1(hid, w1) + b1.dimshuffle((0, 'x', 'x')))
     hid_flat = hid1.reshape((n_batch, n_hid))
-    out = tensor.nnet.softmax(tensor.dot(hid_flat, v)+c)
-    loss = tensor.sum(tensor.nnet.crossentropy_categorical_1hot(out, tensor.argmax(y, axis=1)) * lr)
+    out = tensor.nnet.softmax(tensor.dot(hid_flat, v) + c)
+    loss = tensor.sum(tensor.nnet.crossentropy_categorical_1hot(out,
+         tensor.argmax(y, axis=1)) * lr)
     #print 'loss type', loss.type
 
     params = [w0, b0, w1, b1, v, c]
-    gparams = tensor.grad(loss, params, warn_type=True)
+    gparams = tensor.grad(loss, params)
 
     mode = get_mode(use_gpu, check_isfinite)
 
     #print 'building pfunc ...'
-    train = pfunc([x,y,lr], [loss], mode=mode, updates=[(p, p-g) for p,g in zip(params, gparams)])
+    train = pfunc([x, y, lr], [loss], mode=mode, updates=[(p, p - g) for p,
+        g in zip(params, gparams)])
 
     if verbose:
         theano.printing.debugprint(train)
@@ -392,7 +401,7 @@ def build_conv_nnet2_classif(use_gpu, isize, ksize, n_batch,
         topo = train.maker.fgraph.toposort()
         assert len([n for n in topo if isinstance(n.op, tcn.blas.GpuConv)]) > 0
 
-    shape_target = (n_batch,n_out)
+    shape_target = (n_batch, n_out)
     return train, params, shape_img, shape_target, mode
 
 
@@ -405,7 +414,7 @@ def run_conv_nnet2_classif(use_gpu, seed, isize, ksize, bsize,
     """Run the train function returned by build_conv_nnet2_classif on one device.
     """
 
-    utt.seed_rng(seed) # Seeds numpy.random with seed
+    utt.seed_rng(seed)  # Seeds numpy.random with seed
     train, params, x_shape, y_shape, mode = build_conv_nnet2_classif(
             use_gpu=use_gpu,
             isize=isize,
@@ -488,7 +497,7 @@ def cmp_run_conv_nnet2_classif(seed, isize, ksize, bsize,
                     verbose=verbose,
                     version=version)
 
-        utt.seed_rng(seed) # Seeds numpy.random with seed
+        utt.seed_rng(seed)  # Seeds numpy.random with seed
         train_cpu, params_cpu, x_shape, y_shape, mode_cpu = \
                 build_conv_nnet2_classif(
                         use_gpu=False,
@@ -499,7 +508,7 @@ def cmp_run_conv_nnet2_classif(seed, isize, ksize, bsize,
                         version=version,
                         check_isfinite=check_isfinite)
 
-        utt.seed_rng(seed) # Seeds numpy.random with seed
+        utt.seed_rng(seed)  # Seeds numpy.random with seed
         train_gpu, params_gpu, x_shape_gpu, y_shape_gpu, mode_gpu = \
                 build_conv_nnet2_classif(
                         use_gpu=True,
@@ -525,28 +534,30 @@ def cmp_run_conv_nnet2_classif(seed, isize, ksize, bsize,
             t0 = time.time()
             rval_cpu = train_cpu(xval, yval, lr)[0]
             t1 = time.time()
-            time_cpu += (t1-t0)
+            time_cpu += (t1 - t0)
 
             # Train one batch on GPU
             t0 = time.time()
             rval_gpu = train_gpu(xval, yval, lr)[0]
             t1 = time.time()
-            time_gpu += (t1-t0)
+            time_gpu += (t1 - t0)
 
             # Compare results
             if (verbose or not
                     numpy.allclose(rval_cpu, rval_gpu, rtol=1e-5, atol=float_atol)):
-                print "At batch:", i+1
+                print "At batch:", i + 1
                 print "CPU:", rval_cpu
                 print "GPU:", rval_gpu
-                print "abs diff:", numpy.absolute(rval_gpu-rval_cpu)
-                print "rel diff:", numpy.absolute((rval_gpu-rval_cpu)/rval_gpu)
+                print "abs diff:", numpy.absolute(rval_gpu - rval_cpu)
+                print "rel diff:", numpy.absolute((
+                    rval_gpu - rval_cpu) / rval_gpu)
 
             if not ignore_error:
-                assert numpy.allclose(rval_cpu, rval_gpu, rtol=1e-5, atol=float_atol)
+                assert numpy.allclose(rval_cpu, rval_gpu,
+                     rtol=1e-5, atol=float_atol)
 
             # Synchronize parameters to start from the same point next time
-            if i < n_train-1:
+            if i < n_train - 1:
                 for cpu_p, gpu_p in zip(params_cpu, params_gpu):
                     cpu_p.set_value(gpu_p.get_value(borrow=False), borrow=True)
 
@@ -574,27 +585,27 @@ def cmp_run_conv_nnet2_classif(seed, isize, ksize, bsize,
 
 
 # Default parameters for all subsequent tests
-gpu_only=False
-cpu_only=False
-ignore_error=False
-verbose=0
-version=-1
+gpu_only = False
+cpu_only = False
+ignore_error = False
+verbose = 0
+version = -1
 seed = utt.fetch_seed()
 
 
-def test_lenet_28(): #MNIST
+def test_lenet_28():  # MNIST
     cmp_run_conv_nnet2_classif(seed, 28, 5, 60, n_train=10,
                                ignore_error=ignore_error, gpu_only=gpu_only,
                                cpu_only=cpu_only, verbose=verbose, version=version)
 
 
-def test_lenet_32(): #CIFAR10 / Shapeset
+def test_lenet_32():  # CIFAR10 / Shapeset
     cmp_run_conv_nnet2_classif(seed, 32, 5, 60, n_train=8,
                                ignore_error=ignore_error, gpu_only=gpu_only,
                                verbose=verbose, version=version)
 
 
-def test_lenet_32_long(): #CIFAR10 / Shapeset
+def test_lenet_32_long():  # CIFAR10 / Shapeset
     # this tests the gradient of downsample on the GPU,
     # which does not recieve specific testing
     cmp_run_conv_nnet2_classif(seed, 32, 5, 30, n_train=50,
@@ -602,7 +613,7 @@ def test_lenet_32_long(): #CIFAR10 / Shapeset
                                cpu_only=cpu_only, verbose=verbose, version=version)
 
 
-def test_lenet_64(): # ???
+def test_lenet_64():  # ???
     #float_atol need to pass in debug mode
     #needed as cpu use extended precision and gpu don't
     cmp_run_conv_nnet2_classif(seed, 64, 7, 10, n_train=10,
@@ -611,14 +622,14 @@ def test_lenet_64(): # ???
                                check_isfinite=True, version=version)
 
 
-def test_lenet_108(): # NORB
+def test_lenet_108():  # NORB
     cmp_run_conv_nnet2_classif(seed, 108, 7, 5, n_train=4,
                                ignore_error=ignore_error, gpu_only=gpu_only,
                                cpu_only=cpu_only, verbose=verbose,
                                check_isfinite=True, version=version)
 
 
-def test_lenet_256(): # ImageNet
+def test_lenet_256():  # ImageNet
     cmp_run_conv_nnet2_classif(seed, 256, 9, 2, n_train=5,
                                ignore_error=ignore_error, gpu_only=gpu_only,
                                cpu_only=cpu_only, verbose=verbose,
@@ -626,16 +637,16 @@ def test_lenet_256(): # ImageNet
 
 
 #I did a wanted error in the name as we don't want it to execute automatically for now as it don't work
-def tes_lenet_hd(): #HD 720p: 1280(wid)x720(len)
-    cmp_run_conv_nnet2_classif(seed, (720,1280), 9, 2, n_train=3,
+def tes_lenet_hd():  # HD 720p: 1280(wid)x720(len)
+    cmp_run_conv_nnet2_classif(seed, (720, 1280), 9, 2, n_train=3,
                                ignore_error=ignore_error, gpu_only=gpu_only,
                                cpu_only=cpu_only, verbose=verbose,
                                check_isfinite=True, version=version)
 
 
 #I did a wanted error in the name as we don't want it to execute automatically for now as it don't work
-def tes_lenet_full_hd(): #HD 1080p: 1920(wid)x1080(len)
-    cmp_run_conv_nnet2_classif(seed, (1080,1920), 9, 2, n_train=3,
+def tes_lenet_full_hd():  # HD 1080p: 1920(wid)x1080(len)
+    cmp_run_conv_nnet2_classif(seed, (1080, 1920), 9, 2, n_train=3,
                                ignore_error=ignore_error, gpu_only=gpu_only,
                                cpu_only=cpu_only, verbose=verbose,
                                check_isfinite=True, version=version)

theano/sandbox/cuda/tests/test_neighbours.py

 # Skip test if cuda_ndarray is not available.
 from nose.plugins.skip import SkipTest
-import numpy
-
-import theano
 
 import theano.sandbox.cuda as cuda_ndarray
 if cuda_ndarray.cuda_available == False:

theano/sandbox/neighbours.py

@@ -2,10 +2,10 @@
 TODO: implement Images2Neibs.{perform,infer_shape}() methods
 
 """
-import theano
 from theano import Op, Apply
 import theano.tensor as T
 from theano.gradient import grad_not_implemented
+from theano.gradient import grad_undefined
 
 
 class Images2Neibs(Op):
@@ -59,7 +59,8 @@ class Images2Neibs(Op):
                 for j in xrange(list 2 dim)
                     for k in <image column coordinates>
                         for l in <image row coordinates>
-                            output[idx,:] = flattened version of ten4[i,j,l:l+r,k:k+c]
+                            output[idx,:]
+                                 = flattened version of ten4[i,j,l:l+r,k:k+c]
                             idx += 1
             (note: the op isn't necessarily implemented internally with these
             for loops, they're just the easiest way to describe the output pattern)
@@ -90,8 +91,11 @@ class Images2Neibs(Op):
                 (hasattr(neib_shape, "equals") and
                  neib_shape.equals(neib_step))):
                 return [neibs2images(gz, neib_shape, x.shape, mode=self.mode),
-                        None, None]
-        return [grad_not_implemented(self, 0, x), None, None]
+                        grad_undefined(self, 1, neib_shape),
+                        grad_undefined(self, 2, neib_step)]
+        return [grad_not_implemented(self, 0, x),
+                grad_undefined(self, 1, neib_shape),
+                grad_undefined(self, 2, neib_step)]
 
     def c_code_cache_version(self):
         return (5,)
@@ -307,5 +311,3 @@ def neibs2images(neibs, neib_shape, original_shape, mode='valid'):
         raise NotImplementedError("neibs2images do not support mode=%s" % mode)
 
     return output_4d
-
-

theano/scalar/basic.py

@@ -26,6 +26,9 @@ from theano.gof import Op, utils, Variable, Constant, Type, Apply, FunctionGraph
 from theano.gof.python25 import partial, all, any
 from theano.configparser import config
 
+from theano.gradient import DisconnectedType
+from theano.gradient import grad_undefined
+
 builtin_complex = complex
 builtin_int = int
 builtin_float = float
@@ -332,7 +335,7 @@ class Scalar(Type):
                 return '''
                 template <> %(mytype)s & %(mytype)s::operator=<%(othertype)s>(const %(othertype)s & y)
                 { this->real=y; this->imag=0; return *this; }
-                ''' % dict(mytype = mytype, othertype = othertype)
+                ''' % dict(mytype=mytype, othertype=othertype)
 
             def operator_eq_cplx(mytype, othertype):
                 return '''
@@ -448,8 +451,11 @@ class _scalar_py_operators:
     ndim = 0
 
     #UNARY
-    def __abs__(self): return abs_(self)
-    def __neg__(self): return neg(self)
+    def __abs__(self):
+        return abs_(self)
+
+    def __neg__(self):
+        return neg(self)
 
     #CASTS
     #def __int__(self): return AsInt(self).out
@@ -457,39 +463,87 @@ class _scalar_py_operators:
     #def __complex__(self): return AsComplex(self).out
 
     #BITWISE
-    def __invert__(self): return invert(self)
-    def __and__(self,other): return and_(self, other)
-    def __or__(self,other): return or_(self, other)
-    def __xor__(self,other): return xor(self, other)
-    def __rand__(self,other): return and_(other,self)
-    def __ror__(self,other): return or_(other, self)
-    def __rxor__(self,other): return xor(other, self)
+    def __invert__(self):
+        return invert(self)
+
+    def __and__(self, other):
+        return and_(self, other)
+
+    def __or__(self, other):
+        return or_(self, other)
+
+    def __xor__(self, other):
+        return xor(self, other)
+
+    def __rand__(self, other):
+        return and_(other, self)
+
+    def __ror__(self, other):
+        return or_(other, self)
+
+    def __rxor__(self, other):
+        return xor(other, self)
 
     #COMPARISONS
-    def __lt__(self,other): return lt(self, other)
-    def __le__(self,other): return le(self, other)
-    def __gt__(self,other): return gt(self, other)
-    def __ge__(self,other): return ge(self, other)
+    def __lt__(self, other):
+        return lt(self, other)
+
+    def __le__(self, other):
+        return le(self, other)
+
+    def __gt__(self, other):
+        return gt(self, other)
+
+    def __ge__(self, other):
+        return ge(self, other)
 
     #ARITHMETIC - NORMAL
-    def __add__(self,other): return add(self,other)
-    def __sub__(self,other): return sub(self,other)
-    def __mul__(self,other): return mul(self,other)
-    def __div__(self,other): return div_proxy(self,other)
-    def __floordiv__(self, other): return int_div(self, other)
-    def __mod__(self, other): return mod_check(self, other)
-    def __pow__(self,other): return pow(self,other)
+    def __add__(self, other):
+        return add(self, other)
+
+    def __sub__(self, other):
+        return sub(self, other)
+
+    def __mul__(self, other):
+        return mul(self, other)
+
+    def __div__(self, other):
+        return div_proxy(self, other)
+
+    def __floordiv__(self, other):
+        return int_div(self, other)
+
+    def __mod__(self, other):
+        return mod_check(self, other)
+
+    def __pow__(self, other):
+        return pow(self, other)
 
     #ARITHMETIC - RIGHT-OPERAND
-    def __radd__(self,other): return add(other,self)
-    def __rsub__(self,other): return sub(other,self)
-    def __rmul__(self,other): return mul(other,self)
-    def __rdiv__(self,other): return div_proxy(other,self)
-    def __rmod__(self,other): return mod(other,self)
-    def __rpow__(self,other): return pow(other,self)
+    def __radd__(self, other):
+        return add(other, self)
+
+    def __rsub__(self, other):
+        return sub(other, self)
+
+    def __rmul__(self, other):
+        return mul(other, self)
+
+    def __rdiv__(self, other):
+        return div_proxy(other, self)
+
+    def __rmod__(self, other):
+        return mod(other, self)
+
+    def __rpow__(self, other):
+        return pow(other, self)
 
     def zeros_like(self):
-        return ScalarConstant(Scalar(str(self.type.dtype)), 0)
+        # The second is needed for Elemwise ops to work right
+        return second(self, ScalarConstant(Scalar(str(self.type.dtype)), 0))
+
+    def astype(self, dtype):
+        return cast(self, dtype)
 
 
 class ScalarVariable(_scalar_py_operators, Variable):
@@ -690,7 +744,8 @@ class ScalarOp(Op):
         self.name = name
         if output_types_preference is not None:
             if not callable(output_types_preference):
-                raise TypeError("Expected a callable for the 'output_types_preference' argument to %s. (got: %s)" % (self.__class__, output_types_preference))
+                raise TypeError(
+                    "Expected a callable for the 'output_types_preference' argument to %s. (got: %s)" % (self.__class__, output_types_preference))
             self.output_types_preference = output_types_preference
 
     def make_node(self, *inputs):
@@ -699,7 +754,8 @@ class ScalarOp(Op):
                 raise TypeError("Wrong number of inputs for %s.make_node (got %i(%s), expected %i)" \
                                     % (self, len(inputs), str(inputs), self.nin))
         inputs = [as_scalar(input) for input in inputs]
-        outputs = [t() for t in self.output_types([input.type for input in inputs])]
+        outputs = [t() for t in self.output_types([input.
+            type for input in inputs])]
         if len(outputs) != self.nout:
             raise TypeError("Not the right number of outputs produced for %s(%s). Expected %s, got %s."
                             % (self, ", ".join(str(input) for input in inputs), self.nout, len(outputs)))
@@ -709,7 +765,8 @@ class ScalarOp(Op):
         if hasattr(self, 'output_types_preference'):
             variables = self.output_types_preference(*types)
             if not isinstance(variables, (list, tuple)) or any(not isinstance(x, Type) for x in variables):
-                raise TypeError("output_types_preference should return a list or a tuple of types", self.output_types_preference, variables)
+                raise TypeError(
+                    "output_types_preference should return a list or a tuple of types", self.output_types_preference, variables)
             if len(variables) != self.nout:
                 raise TypeError("Not the right number of outputs types produced for %s(%s) by %s. Expected %s, got %s."
                                 % (self, ", ".join(str(type) for type in variables),
@@ -1092,11 +1149,15 @@ class Maximum(BinaryScalarOp):
     def grad(self, (x, y), (gz, )):
         assert gz.type not in complex_types
         # max is not defined for complex_types
-        gx, gy = None, None
-        if x.type in float_types:
-            gx = cast(eq(maximum(x, y), x) * gz, x.type.dtype)
-        if y.type in float_types:
-            gy = cast(eq(maximum(x, y), y) * gz, y.type.dtype)
+
+        output = self(x, y)
+
+        if output.type in discrete_types:
+            return [x.zeros_like().astype(theano.config.floatX),
+                    y.zeros_like().astype(theano.config.floatX)]
+
+        gx = eq(output, x) * gz
+        gy = eq(output, y) * gz
         return (gx, gy)
 maximum = Maximum(upcast_out, name='maximum')
 
@@ -1118,11 +1179,13 @@ class Minimum(BinaryScalarOp):
     def grad(self, (x, y), (gz, )):
         assert gz.type not in complex_types
         # max is not defined for complex_types
-        gx, gy = None, None
-        if x.type in float_types:
-            gx = cast(eq(minimum(x, y), x) * gz, x.type.dtype)
-        if y.type in float_types:
-            gy = cast(eq(minimum(x, y), y) * gz, y.type.dtype)
+
+        output = minimum(x, y)
+        if output.type in discrete_types:
+            return [x.zeros_like().astype(theano.config.floatX),
+                    y.zeros_like().astype(theano.config.floatX)]
+        gx = eq(output, x) * gz
+        gy = eq(output, y) * gz
         return (gx, gy)
 
 minimum = Minimum(upcast_out, name='minimum')
@@ -1143,23 +1206,21 @@ class Add(ScalarOp):
             return z + " = " + " + ".join(inputs) + ";"
 
     def grad(self, inputs, (gz, )):
-        retval = []
         if gz.type in complex_types:
-            for i in inputs:
-                if i.type in complex_types:
-                    retval += [cast(gz, i.type.dtype)]
-                elif i.type in float_types:
-                    retval += [cast(real(gz), i.type.dtype)]
-                else:
-                    retval += [None]
-        elif gz.type in float_types:
-            for i in inputs:
-                if i.type in float_types:
-                    retval += [cast(gz, i.type.dtype)]
+            raise NotImplementedError()
+        if self(*inputs).type in discrete_types:
+            assert gz is not None
+            retval = []
+            for ii, inp in enumerate(inputs):
+                if hasattr(inp, 'zeros_like'):
+                    retval.append(
+                            inp.zeros_like().astype(theano.config.floatX))
                 else:
-                    retval += [None]
+                    retval.append(grad_undefined(self, ii, inp))
         else:
-            retval += [None] * len(inputs)
+            retval = []
+            for i in inputs:
+                    retval += [gz]
         return retval
 add = Add(upcast_out, name='add')
 
@@ -1186,30 +1247,29 @@ class Mul(ScalarOp):
         output_type = self.output_types([i.type for i in inputs])[0]
         if output_type in complex_types:
             if not gz.type in complex_types:
-                raise TypeError('Mul with output_type '+str(output_type)+\
-                        ' expected gz type to be complex, got gz with type '+\
+                raise TypeError('Mul with output_type ' + str(output_type) +\
+                        ' expected gz type to be complex, got gz with type ' +\
                         str(gz.type))
 
+        if output_type in discrete_types:
+            return [ipt.zeros_like().astype(theano.config.floatX)
+                    for ipt in inputs]
+
         for input in inputs:
-            if input.type in continuous_types:
-                if gz.type in complex_types:
-                    # zr+zi = (xr + xi)(yr + yi)
-                    # zr+zi = (xr*yr - xi*yi) + (xr yi + xi yr )
-                    otherprod = mul(*(utils.difference(inputs, [input])))
-                    yr = real(otherprod)
-                    yi = imag(otherprod)
-                    if input.type in complex_types:
-                        retval += [complex(yr * real(gz) + yi * imag(gz),
-                                           yr * imag(gz) - yi * real(gz))]
-                    else:
-                        retval += [cast(yr * real(gz) + yi * imag(gz),
-                                        input.type.dtype)]
+            if gz.type in complex_types:
+                # zr+zi = (xr + xi)(yr + yi)
+                # zr+zi = (xr*yr - xi*yi) + (xr yi + xi yr )
+                otherprod = mul(*(utils.difference(inputs, [input])))
+                yr = real(otherprod)
+                yi = imag(otherprod)
+                if input.type in complex_types:
+                    retval += [complex(yr * real(gz) + yi * imag(gz),
+                                       yr * imag(gz) - yi * real(gz))]
                 else:
-                    retval += [cast(mul(*([gz] + utils.difference(inputs,
-                                                                  [input]))),
-                                    input.type.dtype)]
+                    retval += [yr * real(gz) + yi * imag(gz)]
             else:
-                retval += [None]
+                retval += [mul(*([gz] + utils.difference(inputs,
+                                                              [input])))]
         return retval
 
 
@@ -1227,15 +1287,13 @@ class Sub(BinaryScalarOp):
         if gz.type in complex_types:
             raise NotImplementedError()
 
-        if x.type in float_types:
-            first_part = cast(gz, x.type.dtype)
-        else:
-            first_part = None
+        if (x - y).type in discrete_types:
+            return [x.zeros_like().astype(theano.config.floatX),
+                    y.zeros_like().astype(theano.config.floatX)]
+
+        first_part = gz
+        second_part = -gz
 
-        if y.type in float_types:
-            second_part = cast(-gz, y.type.dtype)
-        else:
-            second_part = None
         return first_part, second_part
 sub = Sub(upcast_out, name='sub')
 
@@ -1313,22 +1371,28 @@ class TrueDiv(BinaryScalarOp):
         return "%(z)s = %(x)s / %(y)s;" % locals()
 
     def grad(self, (x, y), (gz, )):
+
         if x.type in complex_types:
             raise NotImplementedError()
-        if x.type in float_types:
-            first_part = cast(gz / y, x.type.dtype)
-        else:
-            assert x.type in discrete_types
-            first_part = None
+
+        # If the output of this op is discrete, then it
+        # it is locally flat everywhere, so the gradient
+        # through it is 0.
+        # This is different from it not being connected
+        # to the output; x/y is still a function of x
+        # and y; it's just a step function.
+        if (x / y).type in discrete_types:
+            return [x.zeros_like(), y.zeros_like()]
+
+        first_part = gz / y
 
         if y.type in complex_types:
             raise NotImplementedError()
-        if y.type in float_types:
-            second_part = cast(-(gz * x) / (y * y), y.type.dtype)
-        else:
-            assert y.type in discrete_types
-            second_part = None
+
+        second_part = -(gz * x) / (y * y)
+
         return first_part, second_part
+
 true_div = TrueDiv(upcast_out, name='true_div')
 
 
@@ -1501,15 +1565,14 @@ class Pow(BinaryScalarOp):
     def grad(self, (x, y), (gz, )):
         if gz.type in complex_types:
             raise NotImplementedError()
-        if x.type in float_types:
-            first_part = gz * y * x ** (y - 1)
-        else:
-            first_part = None
 
-        if y.type in float_types:
-            second_part = gz * log(x) * x ** y
-        else:
-            second_part = None
+        if self(x, y).type in discrete_types:
+            return [x.zeros_like().astype(theano.config.floatX),
+                    y.zeros_like().astype(theano.config.floatX)]
+
+        first_part = gz * y * x ** (y - 1)
+
+        second_part = gz * log(x) * x ** y
 
         return (first_part, second_part)
 
@@ -1549,11 +1612,25 @@ class Second(BinaryScalarOp):
     def c_code(self, node, name, (x, y), (z, ), sub):
         return "%(z)s = %(y)s;" % locals()
 
+    def connection_pattern(self, node):
+
+        # x is never connected because its elements are never used
+        # y is connected because its elements are copied over
+
+        return [[False], [True]]
+
     def grad(self, (x, y), (gz, )):
+
         if y.type in continuous_types:
-            return None, gz
+            # x is disconnected because the elements of x are not used
+            return DisconnectedType()(), gz
         else:
-            return None, None
+            #when y is discrete, we assume the function can be extended
+            #to deal with real-valued inputs by rounding them to the
+            #nearest integer. f(x+eps) thus equals f(x) so the gradient
+            #is zero, not disconnected or undefined
+            return DisconnectedType()(), y.zeros_like()
+
 second = Second(transfer_type(1), name='second')
 
 
@@ -1591,10 +1668,10 @@ class Cast(UnaryScalarOp):
         return "%s = (%s)%s;" % (z, node.outputs[0].type.dtype_specs()[1], x)
 
     def grad(self, (x, ), (gz, )):
-        if x.type in continuous_types and self.o_type in continuous_types:
-            return [cast(gz, x.type.dtype)]
+        if self.o_type in continuous_types:
+            return [gz]
         else:
-            return None,
+            return [x.zeros_like().astype(theano.config.floatX)]
 
     def c_code_cache_version(self):
         s = super(Cast, self).c_code_cache_version()
@@ -1684,7 +1761,13 @@ class Sgn(UnaryScalarOp):
         return numpy.sign(x)
 
     def grad(self, (x, ), (gz, )):
-        return None,
+
+        rval = x.zeros_like()
+
+        if rval.type.dtype in discrete_types:
+            rval = rval.astype(theano.config.floatX)
+
+        return [rval]
 
     def c_code(self, node, name, (x, ), (z, ), sub):
         #casting is done by compiler
@@ -1710,7 +1793,12 @@ class Ceil(UnaryScalarOp):
         return numpy.ceil(x)
 
     def grad(self, (x,), (gz,)):
-        return None,
+        rval = x.zeros_like()
+
+        if rval.type.dtype in discrete_types:
+            rval = rval.astype(theano.config.floatX)
+
+        return [rval]
 
     def c_code(self, node, name, (x,), (z,), sub):
         return "%(z)s = ceil(%(x)s);" % locals()
@@ -1722,7 +1810,12 @@ class Floor(UnaryScalarOp):
         return numpy.floor(x)
 
     def grad(self, (x,), (gz,)):
-        return None,
+        rval = x.zeros_like()
+
+        if rval.type.dtype in discrete_types:
+            rval = rval.astype(theano.config.floatX)
+
+        return [rval]
 
     def c_code(self, node, name, (x,), (z,), sub):
         return "%(z)s = floor(%(x)s);" % locals()
@@ -1734,7 +1827,7 @@ class Trunc(UnaryScalarOp):
         return numpy.trunc(x)
 
     def grad(self, (x,), (gz,)):
-        return None,
+        return [x.zeros_like().astype(theano.config.floatX)]
 
     def c_code(self, node, name, (x,), (z,), sub):
         return "%(z)s = %(x)s >= 0? floor(%(x)s): -floor(-%(x)s);" % locals()
@@ -2631,7 +2724,7 @@ class Composite(ScalarOp):
                      onames),
                  **sub)
         d['nodename'] = nodename
-        if not sub.has_key('id'):
+        if not 'id' in sub:
             #The use of a dummy id is safe as the code is in a separate block.
             #It won't generate conflicting variable name.
             d['id'] = '_DUMMY_ID_'

theano/scan_module/scan_op.py

@@ -260,12 +260,16 @@ class Scan(PureOp):
                                     zip(self.inner_seqs(self.inputs),
                                         self.outer_seqs(inputs))):
             if inner_seq.type.dtype != outer_seq[idx].type.dtype:
+                assert isinstance(idx, int)
+
                 raise ValueError(err_msg1 % ('sequence',
                                              str(outer_seq),
                                              idx,
                                              outer_seq.type.dtype,
+                                             outer_seq.ndim,
                                              str(inner_seq),
-                                             inner_seq.type.dtype))
+                                             inner_seq.type.dtype,
+                                             inner_seq.ndim))
         argoffset += len(self.outer_seqs(inputs))
         # Check that this 3 things have the same dtype for mit_mot:
         #   - initial state of the output
@@ -1260,7 +1264,7 @@ class Scan(PureOp):
         # the gradients with respect to all outputs)
         def compute_gradient(y, g_y):
             gmp = gradient.grad_sources_inputs(
-                        [(y, g_y)], diff_inputs, False)
+                        [(y, g_y)], diff_inputs)
             return [gmp.get(p, None) for p in diff_inputs]
 
         # 6. clean the outputs (i.e. remove update rules)
@@ -1301,7 +1305,13 @@ class Scan(PureOp):
 
         # 7.3. compute gradients of the inputs given one output
         for dx, out in enumerate(clean_outputs):
-            inner_g_out = safe_new(out)
+            if g_outs[dx] != None:
+                inner_g_out = safe_new(g_outs[dx][0])
+            else:
+                # We do not have a gradient on this output so we need a
+                # placeholder, which for now has the same dtype as the
+                # output
+                inner_g_out = safe_new(out)
             ###
             #### I need to clip the gradient HERE !!
 

theano/sparse/basic.py

@@ -18,6 +18,7 @@ from theano.gof.python25 import all
 from theano.gradient import DisconnectedType
 from theano.sparse.utils import hash_from_sparse
 import theano.tests.unittest_tools as utt
+from theano.gradient import grad_not_implemented
 
 sparse_formats = ['csc', 'csr']
 
@@ -255,11 +256,13 @@ def sp_zeros_like(x):
     :return: The same as `x` with zero entries
              for all element.
     """
+
     # TODO: don't restrict to CSM formats
     _, _, indptr, shape = csm_properties(x)
-    return CSM(format=x.format)(numpy.array([], dtype=x.type.dtype),
-                                numpy.array([]), tensor.zeros_like(indptr),
-                                shape)
+    return CSM(format=x.format)(data=numpy.array([], dtype=x.type.dtype),
+                                indices=numpy.array([]),
+                                indptr=tensor.zeros_like(indptr),
+                                shape=shape)
 
 
 class _sparse_py_operators:
@@ -670,7 +673,7 @@ class CSM(gof.Op):
     the sparse matrix. Fancy indexing with numpy.ndarray
     should be used for this purpose.
 
-    :param data: One dimensionnal tensor representing
+    :param data: One dimensional tensor representing
                  the data of the sparse to construct.
     :param indices: One dimensional tensor of integers
                     representing the indices of the sparse
@@ -678,7 +681,7 @@ class CSM(gof.Op):
     :param indptr: One dimensional tensor of integers
                    representing the indice pointer for
                    the sparse matrix to construct.
-    :param shape: One dimensionnal tensor of integers
+    :param shape: One dimensional tensor of integers
                   representing the shape of the sparse
                   matrix to construct.
 
@@ -782,6 +785,9 @@ class CSM(gof.Op):
                                               indptr.copy()), shape.copy(),
                                              copy=False)
 
+    def connection_pattern(self, node):
+        return [[True], [False], [False], [False]]
+
     def grad(self, (x_data, x_indices, x_indptr, x_shape), (g_out,)):
         g_data, g_indices, g_indptr, g_shape = csm_properties(g_out)
         # unpack the data vector and wrap it as a 1d TensorType
@@ -984,7 +990,19 @@ class DenseFromSparse(gof.op.Op):
 
     def grad(self, (x, ), (gz, )):
         if self.sparse_grad:
-            return [sp_ones_like(x) * gz]
+            left = sp_ones_like(x)
+            right = gz
+
+            # Do upcasting if necessary to avoid an unimplemented case
+            # of mul
+
+            if right.dtype == 'float64' and left.dtype == 'float32':
+                left = left.astype('float64')
+
+            if right.dtype == 'float32' and left.dtype == 'float64':
+                right = right.astype('float64')
+
+            return [left * right]
         else:
             return [SparseFromDense(x.type.format)(gz)]
 
@@ -1993,7 +2011,9 @@ class MulSS(gof.op.Op):
     def make_node(self, x, y):
         x, y = as_sparse_variable(x), as_sparse_variable(y)
         if x.type != y.type:
-            raise NotImplementedError()
+            raise NotImplementedError(
+                    "MulSS not supported for differing types. "
+                    "Got %s and %s." % (str(x.type), str(y.type)))
         return gof.Apply(self, [x, y], [x.type()])
 
     def perform(self, node, (x, y), (out, )):
@@ -2042,7 +2062,9 @@ class MulSD(gof.op.Op):
             y = tensor.cast(y, dtype)
 
         if x.type.dtype != y.type.dtype:
-            raise NotImplementedError()
+            raise NotImplementedError(
+                "MulSD not implemented for different input dtypes. "
+                "Got %s and %s." % (x.type.dtype, y.type.dtype))
         # The magic number two here arises because L{scipy.sparse}
         # objects must be matrices (have dimension 2)
         # Broadcasting of the sparse matrix is not supported.
@@ -2128,7 +2150,9 @@ class MulSV(gof.op.Op):
         assert y.type.ndim == 1
 
         if x.type.dtype != y.type.dtype:
-            raise NotImplementedError()
+            raise NotImplementedError(
+                    "MulSV not implemented for differing dtypes."
+                    "Got %s and %s." % (str(x.type.dtype), str(y.type.dtype)))
         return gof.Apply(self,
                          [x, y],
                          [SparseType(dtype=x.type.dtype,
@@ -2142,6 +2166,15 @@ class MulSV(gof.op.Op):
     def grad(self, (x, y), (gz,)):
         assert _is_sparse_variable(x) and _is_dense_variable(y)
         assert _is_sparse_variable(gz)
+
+        # mul_s_v is not implemented if the types vary
+
+        if gz.dtype == 'float64' and y.dtype == 'float32':
+            y = y.astype('float64')
+
+        if gz.dtype == 'float32' and y.dtype == 'float64':
+            gz = gz.astype('float64')
+
         return mul_s_v(gz, y), sp_sum(x * gz, axis=0, sparse_grad=True)
 
     def infer_shape(self, node, ins_shapes):
@@ -2176,8 +2209,18 @@ def mul(x, y):
 
     assert x_is_sparse_variable or y_is_sparse_variable
     if x_is_sparse_variable and y_is_sparse_variable:
+
+        # mul_s_s is not implemented if the types differ
+        if y.dtype == 'float64' and x.dtype == 'float32':
+            x = x.astype('float64')
+
         return mul_s_s(x, y)
     elif x_is_sparse_variable and not y_is_sparse_variable:
+
+        # mul is unimplemented if the dtypes differ
+        if y.dtype == 'float64' and x.dtype == 'float32':
+            x = x.astype('float64')
+
         return mul_s_d(x, y)
     elif y_is_sparse_variable and not x_is_sparse_variable:
         return mul_s_d(y, x)
@@ -3260,7 +3303,7 @@ class SamplingDot(gof.op.Op):
         rval = [
             dot(p * gz, y),
             dot((p * gz).T, x),
-            None
+            grad_not_implemented(self, 2, p)
         ]
 
         return rval

theano/tensor/basic.py

@@ -479,6 +479,11 @@ def get_constant_value(v):
             data = v.tag.unique_value
         else:
             data = v.data
+        # handle case where data is numpy.array([])
+        if hasattr(data, 'shape') and len(data.shape) == 0 or \
+            __builtins__['max'](data.shape) == 0:
+            assert numpy.all(numpy.array([]) == data)
+            return data
         try:
             numpy.complex(data)  # works for all numeric scalars
             return data
@@ -493,15 +498,19 @@ def get_constant_value(v):
             return get_constant_value(v.owner.inputs[0])
         if isinstance(v.owner.op, Rebroadcast):
             return get_constant_value(v.owner.inputs[0])
-        if v.owner.op == fill:
+        if isinstance(v.owner.op, Elemwise) and \
+                isinstance(v.owner.op.scalar_op, scal.Second):
             shape, val = v.owner.inputs
-            # fill(a,b) fills the shape of 'a' filled with 'b'
             return get_constant_value(val)
+        if isinstance(v.owner.op, scal.Second):
+            x, y = v.owner.inputs
+            return get_constant_value(y)
         # Don't act as the constant_folding optimization here as this
         # fct is used too early in the optimization phase.  This would
         # mess with the stabilization optimization.
-        if isinstance(v.owner.op, Elemwise) and isinstance(
-            v.owner.op.scalar_op, scal.Cast):
+        if (isinstance(v.owner.op, Elemwise) and isinstance(
+            v.owner.op.scalar_op, scal.Cast)) or \
+            isinstance(v.owner.op, scal.Cast):
             const = get_constant_value(v.owner.inputs[0])
             ret = [[None]]
             v.owner.op.perform(v.owner, [const], ret)
@@ -983,8 +992,10 @@ class TensorType(Type):
                          %(type_num)s, type_num_%(name)s);
             %(fail)s
         }
+        // This is a TypeError to be consistent with DEBUG_MODE
+        // Note: DEBUG_MODE also tells the name of the container
         if (type_num_%(name)s != %(type_num)s) {
-            PyErr_Format(PyExc_ValueError,
+            PyErr_Format(PyExc_TypeError,
                          "expected type_num %%d (%(type_num)s) got %%d",
                          %(type_num)s, type_num_%(name)s);
             %(fail)s
@@ -1910,6 +1921,9 @@ class TensorFromScalar(Op):
     def grad(self, inp, grads):
         s, = inp
         dt, = grads
+        assert dt.type.dtype.find('float') != -1
+        if s.type.dtype.find('int') != -1:
+            return [s.zeros_like().astype(theano.config.floatX)]
         return [scalar_from_tensor(dt)]
 
     def __str__(self):
@@ -2097,13 +2111,13 @@ class Shape(Op):
     def infer_shape(self, node, in_shapes):
         return [[len(in_shapes[0])]]
 
-    def connection_pattern(self):
+    def connection_pattern(self, node):
         # the grad returns the gradient with respect to the
         # elements of a tensor variable
         # the elements of the tensor variable do not participate
         # in the computation of the shape, so they are not really
         # part of the graph
-        return [False]
+        return [[False]]
 
     def grad(self, inp, grads):
         # the grad returns the gradient with respect to the
@@ -2111,7 +2125,7 @@ class Shape(Op):
         # the elements of the tensor variable do not participate
         # in the computation of the shape, so they are not really
         # part of the graph
-        return [None]
+        return [DisconnectedType()()]
 
     def R_op(self, inputs, eval_points):
         return [None]
@@ -2193,6 +2207,9 @@ class SpecifyShape(Op):
         assert len(new_shape) == len(xshape)
         return [new_shape]
 
+    def connection_pattern(self, node):
+        return [[True], [False]]
+
     def grad(self, inp, grads):
         x, s = inp
         gz, = grads
@@ -2201,8 +2218,8 @@ class SpecifyShape(Op):
         # to remove that op from the graph to don't block other optimization
         # Should I do an optimizer that will remove the SpecifyShape?
         # I think Yes
-        return [gz, None]
-        return [specify_shape(gz, s), None]
+        return [gz, DisconnectedType()()]
+        return [specify_shape(gz, s), DisconnectedType()()]
 
     def R_op(self, inputs, eval_points):
         if eval_points[0] is None:
@@ -2988,73 +3005,6 @@ def eye(n, m=None, k=0, dtype=None):
 def identity_like(x):
     return eye(x.shape[0], x.shape[1], k=0, dtype=x.dtype)
 
-if 0:
-    ## COMMENTED OUT FEB 17 2010
-    ## TODO (DOCUMENT AND WRITE TESTS) OR DELETE
-    class Filler(gof.Op):
-        """WRITEME"""
-        def __init__(self, value, ndim, dtype='float64'):
-            self.value = value
-            self.ndim = ndim
-            self.dtype = dtype
-            self.type = TensorType(dtype=dtype,
-                                   broadcastable=(False,) * ndim)
-
-        def make_node(self, dims):
-            dims = as_tensor_variable(dims)
-            return gof.Apply(self, [dims], [self.type()])
-
-        def perform(self, node, inp, out_):
-            dims, = inp
-            out, = out_
-            if out[0] is not None:
-                out[0].resize(dims, refcheck=0)
-                out[0].fill(self.value)
-            else:
-                if self.value == 0:
-                    out[0] = numpy.zeros(dims, dtype=self.dtype)
-                elif self.value == 1:
-                    out[0] = numpy.ones(dims, dtype=self.dtype)
-                else:
-                    out[0] = numpy.ones(dims, dtype=self.dtype) * self.value
-
-        def grad(self, inp, grads):
-            return None,
-
-        def __eq__(self, other):
-            return (type(self) == type(other) and self.ndim == other.ndim and
-                    self.dtype == other.dtype)
-
-        def __hash__(self):
-            return hash(self.ndim) ^ hash(self.dtype)
-
-    Zeros = partial(Filler, 0)
-    """WRITEME"""
-
-    Ones = partial(Filler, 1)
-    """WRITEME"""
-
-    @constructor
-    def zero():
-        """
-        Return a scalar zero, e.g. for initializing sums.
-        """
-        return Zeros(0)([])
-
-    @constructor
-    def one():
-        """WRITEME"""
-        return Ones(0)([])
-
-    pprint.assign(lambda pstate, r: r.owner and
-                                    isinstance(r.owner.op, Filler) and
-                                    r.owner.op.value == 0,
-                  printing.FunctionPrinter('zeros'))
-    pprint.assign(lambda pstate, r: r.owner and
-                                    isinstance(r.owner.op, Filler) and
-                                    r.owner.op.value == 1,
-                  printing.FunctionPrinter('ones'))
-
 
 class Alloc(gof.Op):
     """Create a Tensor from an initial value and a desired shape
@@ -3170,12 +3120,25 @@ class Alloc(gof.Op):
     def infer_shape(self, node, input_shapes):
         return [node.inputs[1:]]
 
+    def connection_pattern(self, node):
+
+        rval = [[True]]
+
+        for ipt in node.inputs[1:]:
+            rval.append([False])
+
+        return rval
+
     def grad(self, inputs, grads):
         x = inputs[0]
         gz = grads[0]
         n_axes_to_sum = gz.ndim - x.ndim
         gx = gz.sum(axis=range(n_axes_to_sum))
-        return [gx] + [None for i in inputs[1:]]
+        #The *elements* of the output are not connected to
+        #the inputs that specify the shape. If you grow the
+        #shape by epsilon, the existing elements do not
+        #change.
+        return [gx] + [DisconnectedType()() for i in inputs[1:]]
 
     def __call__(self, val, *shapes):
         """
@@ -3439,43 +3402,6 @@ def std(input, axis=None, keepdims=False):
 
     return sqrt(var(input=input, axis=axis, keepdims=keepdims))
 
-if 0:
-    ## COMMENTED OUT FEB 17 2010
-    ## TODO (DOCUMENT AND WRITE TESTS) OR DELETE
-    class Repeat(gof.Op):
-
-        def make_node(self, input, repeats, axis):
-            assert isinstance(input.type, TensorType)
-            assert repeats.type == iscalar
-            assert axis.type == iscalar
-            broadcastable = []
-            for i, x in enumerate(input.broadcastable):
-                if i == axis:
-                    broadcastable += [False]
-                else:
-                    broadcastable += [x]
-
-            type = TensorType(dtype=input.type.dtype,
-                              broadcastable=broadcastable)
-            # backport
-            # type = TensorType(dtype=input.type.dtype,
-            #                  broadcastable=[
-            #                      False if i==axis else x
-            #                      for i, x in enumerate(input.broadcastable)])
-            return gof.Apply(self, [inputs, repeats, axis], [type()])
-
-        def perform(self, node, inp, out_):
-            input, repeats, axis = inp
-            out, = out_
-            out[0] = numpy.repeat(input, repeats, axis)
-
-        def grad(self, inp, grads):
-            input, repeats, axis = inp
-            gout, = grads
-            return add.grad((input, gout), (gout,))[:1]
-
-    repeat = Repeat()
-
 
 class Default(gof.Op):
     """
@@ -3969,8 +3895,22 @@ class Subtensor(Op):
         gz, = grads
         x = inputs[0]
         rest = inputs[1:]
-        return ([IncSubtensor(self.idx_list)(zeros_like(x), gz, *rest)]
-                + [None] * len(rest))
+        output = self(*inputs)
+        if output.dtype.find('int') != -1:
+            first = x.zeros_like().astype(theano.config.floatX)
+        else:
+            first = IncSubtensor(self.idx_list)(zeros_like(x), gz, *rest)
+        return ([first]
+                + [DisconnectedType()()] * len(rest))
+
+    def connection_pattern(self, node):
+
+        rval = [[True]]
+
+        for ipt in node.inputs[1:]:
+            rval.append([False])
+
+        return rval
 
     def __eq__(self, other):
         return type(self) == type(other) and self.idx_list == other.idx_list
@@ -4624,6 +4564,15 @@ class IncSubtensor(Op):
         return self.make_node(eval_points[0], eval_points[1],
                             *inputs[2:]).outputs
 
+    def connection_pattern(self, node):
+
+        rval = [[True], [True]]
+
+        for ipt in node.inputs[2:]:
+            rval.append([False])
+
+        return rval
+
     def grad(self, inputs, grads):
         g_output, = grads
         x, y = inputs[:2]
@@ -4637,7 +4586,7 @@ class IncSubtensor(Op):
             gx = g_output
         gy = Subtensor(idx_list=self.idx_list)(g_output, *idx_list)
 
-        return [gx, gy] + [None] * len(idx_list)
+        return [gx, gy] + [DisconnectedType()()] * len(idx_list)
 
 
 def split(x, splits_size, n_splits, axis=0):
@@ -4755,8 +4704,10 @@ class Split(Op):
 
     def grad(self, inputs, g_outputs):
         """Join the gradients along the axis that was used to split x."""
-        _, axis, _ = inputs
-        return [join(axis, *g_outputs), None, None]
+        _, axis, n = inputs
+        return [join(axis, *g_outputs),
+                grad_undefined(self, 1, axis),
+                grad_undefined(self, 2, n)]
 
     def R_op(self, inputs, eval_points):
         if eval_points[0]  is None:
@@ -5024,6 +4975,9 @@ class Join(Op):
         """
         gz, = grads
         axis, tensors = axis_and_tensors[0], axis_and_tensors[1:]
+
+        rval = [grad_undefined(self, 0, axis)]
+
         if 'float' in tensors[0].dtype or 'complex' in tensors[0].dtype:
             # assume that this is differentiable
             split = Split(len(tensors))
@@ -5032,25 +4986,14 @@ class Join(Op):
             # If there is only one split, it might not be in a list.
             if not isinstance(split_gz, list):
                 split_gz = [split_gz]
-            return [None] + split_gz
+
+            rval = rval + split_gz
         else:
-            # assume that this isn't differentiable
-            return [None] * (1 + len(tensors))
+            # the output has integer type, so the gradient through it
+            # is 0
+            rval = rval + [tensor.zeros_like() for tensor in tensors]
 
-    def _native_grad(self, axis_and_tensors, grads):
-        """WRITEME"""
-        gz, = grads
-        axis, tensors = axis_and_tensors[0], axis_and_tensors[1:]
-        sizes_along_axis = [shape(x)[axis] for x in tensors]
-        n_dims = len(shape(tensors[0]))
-        idx = [0]
-        for s in sizes_along_axis:
-            idx.append(idx[-1] + s)
-        # The gradient w.r.t. the k-th tensor is a slice of gz along the
-        # 'axis' dimension.
-        return [gz[[slice(None)] * axis + [slice(idx[k], idx[k + 1])] + \
-                [slice(None)] * (n_dims - axis - 1)] \
-                for k in xrange(len(sizes_along_axis))]
+        return rval
 
     def infer_shape(self, node, ishapes):
         # ishapes[0] contains the size of the axis on which we join
@@ -5294,60 +5237,6 @@ def vertical_stack(*args):
     return concatenate(args, axis=0)
 
 
-# Vertical and horizontal stacking are deprecated. Better to use stack() and
-# join().
-if 0:
-    class VerticalStack(Op):
-        """
-        Vertically stack two L{TensorType}s.
-        Stack two L{TensorType}s along the first axis (row wise). These
-        L{TensorType}s must have the same shape along all dimensions but the
-        first.
-
-        @attention: Because we use vstack as the implementation, if the
-        inputs have 1-dimension, the output will have 2-dimensions.
-        """
-        def make_node(self, x, y):
-            x = as_tensor_variable(x)
-            y = as_tensor_variable(y)
-            assert x.type.dtype == y.type.dtype
-            if x.type.broadcastable[1:] != y.type.broadcastable[1:]:
-                raise NotImplementedError
-            inputs = [x, y]
-            bcastable = (False, ) + x.type.broadcastable[1:]
-            outputs = [tensor(dtype=x.type.dtype,
-                              broadcastable=bcastable)]
-            return Apply(self, inputs, outputs)
-
-        def perform(self, node, inp, out_):
-            x, y = inp
-            out, = out_
-            assert x.ndim == y.ndim
-            # Make sure every dimension (save the first) is the same
-            for i in xrange(x.ndim):
-                assert i == 0 or x.shape[i] == y.shape[i]
-            out[0] = numpy.vstack([x, y])
-
-        def grad(self, inp, grads):
-            """
-            @todo: Make VSplit (or this grad implementation) its own L{Op},
-            that way we can do more sanity-checking::
-                assert x.ndim == y.ndim
-                # Make sure every dimension (save the first) is the same
-                for i in xrange(x.data.ndim):
-                    assert i == 0 or x.data.shape[i] == y.shape[i]
-                etc...
-            """
-            x, y = inp
-            gz, = grads
-            xs = shape(x)
-            return gz[:xs[0]], gz[xs[0]:]
-    vertical_stack = VerticalStack()
-
-else:
-    pass
-
-
 class Reshape(Op):
     """Perform a reshape operation of the input x to the new shape shp.
 
@@ -5410,10 +5299,14 @@ class Reshape(Op):
             raise ValueError('Cannot reshape input of shape %s to shape %s' %
                              (x.shape, shp))
 
+    def connection_pattern(self, node):
+        return [[True], [False]]
+
     def grad(self, inp, grads):
         x, shp = inp
         g_out, = grads
-        return [reshape(g_out, shape(x), ndim=x.ndim), None]
+        return [reshape(g_out, shape(x), ndim=x.ndim),
+                DisconnectedType()()]
 
     def R_op(self, inputs, eval_points):
         if eval_points[0] is None:
@@ -5760,9 +5653,21 @@ class ARange(Op):
         step = step.item()
         out[0] = numpy.arange(start, stop, step, dtype=self.dtype)
 
+    def connection_pattern(self, node):
+
+        return [[True], [False], [True]]
+
     def grad(self, inputs, grads):
+        start, stop, step = inputs
         gz, = grads
-        return [None] * len(inputs)
+        # start and step affect the output values
+        # but the outputs are integers so there's
+        # no gradient through them
+        # stop does not affect the output values,
+        # just the output shape, so it is disconnected
+        return [start.zeros_like(),
+                DisconnectedType()(),
+                step.zeros_like()]
 
     def R_op(self, inputs, eval_points):
         return [None]
@@ -5983,7 +5888,22 @@ class PermuteRowElements(Op):
 
         gx = DimShuffle(gx.type.broadcastable, newdims)(gx)
         assert gx.type.broadcastable == x.type.broadcastable
-        return [gx, None, None]
+
+        # if x is an integer type, then so is the output.
+        # this means f(x+eps) = f(x) so the gradient with respect
+        # to x is zero
+        if x.type.dtype.find('int') != -1:
+            gx = x.zeros_like()
+
+        # The elements of y and of inverse both affect the output,
+        # so they are connected to the output,
+        # and the transformation isn't defined if their values
+        # are non-integer, so the gradient with respect to them is
+        # undefined
+
+        return [gx, grad_undefined(self, 1, y),
+                grad_undefined(self, 1, inverse)]
+
 _permute_row_elements = PermuteRowElements()
 
 
@@ -6046,11 +5966,21 @@ class AdvancedSubtensor1(Op):
 
         out[0] = x.take(i, axis=0, out=o)
 
+    def connection_pattern(self, node):
+
+        rval = [[True]]
+
+        for ipt in node.inputs[1:]:
+            rval.append([False])
+
+        return rval
+
     def grad(self, inputs, grads):
+
         gz, = grads
         assert len(inputs) == 2
         rval1 = [advanced_inc_subtensor1(zeros_like(inputs[0]), gz, inputs[1])]
-        return rval1 + [None] * (len(inputs) - 1)
+        return rval1 + [DisconnectedType()()] * (len(inputs) - 1)
 
     def R_op(self, inputs, eval_points):
         if eval_points[0] is None:
@@ -6149,6 +6079,15 @@ class AdvancedIncSubtensor1(Op):
         return self.make_node(eval_points[0], eval_points[1],
                               *inputs[2:]).outputs
 
+    def connection_pattern(self, node):
+
+        rval = [[True], [True]]
+
+        for ipt in node.inputs[2:]:
+            rval.append([False])
+
+        return rval
+
     def grad(self, inputs, grads):
         g_output, = grads
         x, y = inputs[:2]
@@ -6157,7 +6096,7 @@ class AdvancedIncSubtensor1(Op):
         gx = g_output
         gy = advanced_subtensor1(g_output, *idx_list)
 
-        return [gx, gy] + [None] * len(idx_list)
+        return [gx, gy] + [DisconnectedType()()] * len(idx_list)
 
 advanced_inc_subtensor1 = AdvancedIncSubtensor1()
 
@@ -6246,12 +6185,22 @@ class AdvancedSubtensor(Op):
         # return
         #raise NotImplementedError()
 
+    def connection_pattern(self, node):
+
+        rval = [[True]]
+
+        for ipt in node.inputs[1:]:
+            rval.append([False])
+
+        return rval
+
     def grad(self, inputs, grads):
         gz, = grads
         x = inputs[0]
         rest = inputs[1:]
         return [advanced_inc_subtensor(zeros_like(x), gz,
-                                       *rest)] + [None] * len(rest)
+                                       *rest)] + \
+            [DisconnectedType()()] * len(rest)
 
 
 class AdvancedIncSubtensor(Op):
@@ -6336,13 +6285,23 @@ class AdvancedIncSubtensor(Op):
     def infer_shape(self, node, ishapes):
         return [ishapes[0]]
 
+    def connection_pattern(self, node):
+
+        rval = [[True], [True]]
+
+        for ipt in node.inputs[2:]:
+            rval.append([False])
+
+        return rval
+
     def grad(self, inpt, output_gradients):
         x, y = inpt[:2]
         idxs = inpt[2:]
         outgrad, = output_gradients
         d_x_wrt_C = outgrad
         d_y_wrt_C = AdvancedSubtensor()(outgrad, *idxs)
-        return [d_x_wrt_C, d_y_wrt_C] + [None for _ in idxs]
+        return [d_x_wrt_C, d_y_wrt_C] + \
+            [DisconnectedType()() for _ in idxs]
 
     def R_op(self, inputs, eval_points):
         if None in eval_points[:2]:
@@ -6457,6 +6416,7 @@ class Dot(Op):
             raise
 
     def grad(self, inp, grads):
+
         x, y = inp
         gz, = grads
         if gz.type.ndim == 0:
@@ -6467,7 +6427,11 @@ class Dot(Op):
             rval = outer(gz, y.T), dot(x.T, gz)
         else:
             rval = dot(gz, y.T), dot(x.T, gz)
-        return cast(rval[0], x.dtype), cast(rval[1], y.dtype)
+
+        for elem in rval:
+            assert elem.dtype.find('float') != -1
+
+        return rval
 
     def R_op(self, inputs, eval_points):
         # R_op for a \dot b evaluted at c for a and d for b is

theano/tensor/elemwise.py

@@ -14,6 +14,7 @@ from theano.scalar import Scalar
 from theano.printing import min_informative_str, pprint
 from theano.gof.python25 import all, any
 from theano.tensor.utils import hash_from_dict
+from theano.gradient import DisconnectedType
 
 config = theano.config
 
@@ -277,7 +278,8 @@ class DimShuffle(Op):
         #get the copy / view of the input depending on whether we're doingi
         # things inplace or not.
         if self.inplace:
-            get_base = ['{ PyArrayObject * %(basename)s = %(input)s', 'Py_INCREF((PyObject*)%(basename)s)']
+            get_base = [
+                '{ PyArrayObject * %(basename)s = %(input)s', 'Py_INCREF((PyObject*)%(basename)s)']
         else:
             get_base = [('{ PyArrayObject * %(basename)s = (PyArrayObject*)PyArray_FromAny((PyObject*)%(input)s, NULL,'
                     '0, 0, NPY_ALIGNED|NPY_ENSURECOPY, NULL)')]
@@ -285,7 +287,8 @@ class DimShuffle(Op):
         shape_statements = ['npy_intp dimensions[%i]' % nd_out]
         for i, o in enumerate(self.new_order):
             if o != 'x':
-                shape_statements += [('dimensions[' + str(i) + '] = %(basename)s->dimensions[' + str(o) + ']')]
+                shape_statements += [('dimensions[' + str(
+                    i) + '] = %(basename)s->dimensions[' + str(o) + ']')]
             else:
                 shape_statements += [('dimensions[' + str(i) + '] = 1')]
 
@@ -294,7 +297,8 @@ class DimShuffle(Op):
         #set the strides of the non-broadcasted dimensions
         for i, o in enumerate(self.new_order):
             if o != 'x':
-                strides_statements += [('strides[' + str(i) + '] = %(basename)s->strides[' + str(o) + ']')]
+                strides_statements += [('strides[' + str(i)
+                     + '] = %(basename)s->strides[' + str(o) + ']')]
             else:
                 strides_statements += [('strides[' + str(i) + '] = 0')]
 
@@ -310,7 +314,8 @@ class DimShuffle(Op):
                 '-1] = %(basename)s->descr->elsize'
             )
         for i in xrange(nd_out - 2, -1, -1):
-            strides_statements.append("if (strides[%(i)s] == 0) strides[%(i)s] = strides[%(i)s+1] * dimensions[%(i)s+1]" % dict(i=str(i)))
+            strides_statements.append(
+                "if (strides[%(i)s] == 0) strides[%(i)s] = strides[%(i)s+1] * dimensions[%(i)s+1]" % dict(i=str(i)))
 
         #
         # PyObject* PyArray_New(PyTypeObject* subtype, int nd, npy_intp* dims, int type_num,
@@ -605,7 +610,8 @@ class Elemwise(Op):
                 # the right thing to do .. have to talk to Ian and James
                 # about it
 
-                if bgrads[jdx] is None:
+                if bgrads[jdx] is None or \
+                        isinstance(bgrads[jdx].type, DisconnectedType):
                     pass
                 elif eval_point is not None:
                     if rop_out is None:
@@ -617,6 +623,13 @@ class Elemwise(Op):
 
         return rval
 
+    def connection_pattern(self, node):
+
+        if hasattr(self.scalar_op, 'connection_pattern'):
+            return self.scalar_op.connection_pattern(node)
+
+        return [[True for output in node.outputs] for ipt in node.inputs]
+
     def grad(self, inputs, ograds):
 
         #compute grad with respect to broadcasted input
@@ -676,10 +689,16 @@ class Elemwise(Op):
 
             theano.config.compute_test_value = prev_setting
 
+        if not isinstance(scalar_igrads, (list, tuple)):
+            raise TypeError('%s.grad returned %s instead of list or tuple' %
+                    (str(self.scalar_op), str(type(scalar_igrads))))
+
         nd = len(inputs[0].type.broadcastable)  # this is the same for everyone
 
         def transform(r):
             # From a graph of ScalarOps, make a graph of Broadcast ops.
+            if isinstance(r.type, DisconnectedType):
+                return r
             if r in scalar_inputs:
                 return inputs[scalar_inputs.index(r)]
             if r in scalar_ograds:
@@ -803,7 +822,7 @@ class Elemwise(Op):
             errormsg = ('While computing ' + str(node.outputs) +
                         ': Failed calling ufunc for op ' +
                         str(self.scalar_op) +
-                        'for params of shape ' +
+                        ' for params of shape ' +
                         str([arg.shape for arg in ufunc_args]))
 
             if config.exception_verbosity == 'high':
@@ -1324,7 +1343,8 @@ class CAReduce(Op):
             alloc += """
 for(int i=0;i<%(iname)s->nd;i++){
   if(PyArray_DIMS(%(iname)s)[i]==0 && tosum[i]){
-    PyErr_Format(PyExc_ValueError, "Input of CAReduce{%(scal_name)s} has zero-size on axis %%d",i);
+    PyErr_Format(PyExc_ValueError,
+         "Input of CAReduce{%(scal_name)s} has zero-size on axis %%d",i);
     %(fail)s;
   }
 }
@@ -1585,6 +1605,12 @@ class Sum(CAReduceDtype):
 
     def grad(self, inp, grads):
         x, = inp
+
+        out = self(*inp)
+
+        if out.dtype.find('int') != -1:
+            return [x.zeros_like().astype(theano.config.floatX)]
+
         gz, = grads
         gz = as_tensor_variable(gz)
         axis = self.axis
@@ -1601,7 +1627,7 @@ class Sum(CAReduceDtype):
                 new_dims.append(i)
                 i += 1
         ds_op = DimShuffle(gz.type.broadcastable, new_dims)
-        gx = Elemwise(scalar.second)(x, ds_op(gz).astype(x.dtype))
+        gx = Elemwise(scalar.second)(x, ds_op(gz))
         return [gx]
 
     def R_op(self, inputs, eval_points):
@@ -1646,7 +1672,7 @@ class Prod(CAReduceDtype):
 
     def grad(self, inp, grads):
         '''
-        The grad of this Op could be very easy, it is was not for the case
+        The grad of this Op could be very easy, if it is was not for the case
         where zeros are present in a given "group" (ie. elements reduced
         together to form the product).
 
@@ -1692,8 +1718,11 @@ class Prod(CAReduceDtype):
         '''
         prod_in, = inp
         gz, = grads
-        if prod_in.dtype[0:3] in ('int', 'uin'):
-            return [None]
+
+        out = self(*inp)
+
+        if out.dtype[0:3] in ('int', 'uin'):
+            return [prod_in.zeros_like().astype(theano.config.floatX)]
 
         # Prepare the broadcasting that is used everywhere to broadcast
         # over the original groups (ie. broadcast over the elements of a given

theano/tensor/extra_ops.py

@@ -5,6 +5,7 @@ import theano
 import basic
 from theano import gof, scalar
 import basic as tensor
+from theano.gradient import DisconnectedType
 
 
 class DiffOp(theano.Op):
@@ -148,7 +149,13 @@ class BinCountOp(theano.Op):
         z[0] = np.bincount(x, weights=weights, minlength=self.minlength)
 
     def grad(self, inputs, outputs_gradients):
-        return [None for i in inputs]
+        output = self(*inputs)
+
+        if output.dtype.find('int') != -1:
+            return [inp.zeros_like().astype(theano.config.floatX)
+                    for inp in inputs]
+
+        raise NotImplementedError()
 
     def infer_shape(self, node, ins_shapes):
         x = node.inputs[0]
@@ -252,6 +259,10 @@ class RepeatOp(theano.Op):
         z = output_storage[0]
         z[0] = np.repeat(x, repeats=repeats, axis=self.axis)
 
+    def connection_pattern(self, node):
+
+        return [[True], [False]]
+
     def grad(self, (x, repeats), (gz, )):
         if repeats.ndim == 0:
             if self.axis is None:
@@ -265,7 +276,8 @@ class RepeatOp(theano.Op):
             shape = [x.shape[k] for k in range(x.ndim)]
             shape.insert(axis, repeats)
 
-            return [gz.reshape(shape, x.ndim + 1).sum(axis=axis), None]
+            return [gz.reshape(shape, x.ndim + 1).sum(axis=axis),
+                    DisconnectedType()()]
         elif repeats.ndim == 1:
             # For this implementation, we would need to specify the length
             # of repeats in order to split gz in the right way to sum
@@ -387,7 +399,6 @@ def bartlett(M):
     return bartlett_(M)
 
 
-
 class FillDiagonal(gof.Op):
     # See function fill_diagonal for docstring
     def __eq__(self, other):

theano/tensor/nnet/ConvGrad3D.py

@@ -2,6 +2,8 @@ import theano
 from theano.tensor import basic as T
 from theano.misc import strutil
 import numpy as N
+from theano.gradient import grad_undefined
+from theano.gradient import DisconnectedType
 
 
 #TODO: speed up by reordering loops. Should pass through the videos once, incrementing all weight gradients, rather
@@ -9,7 +11,7 @@ import numpy as N
 
 class ConvGrad3D(theano.Op):
     """ Gradient of Conv3D with respect to W """
-    def __eq__(self,other):
+    def __eq__(self, other):
         return type(self) == type(other)
 
     def __hash__(self):
@@ -27,20 +29,26 @@ class ConvGrad3D(theano.Op):
         return theano.Apply(self, inputs=[V_, d_, WShape_, dCdH_], outputs = [ T.TensorType(V_.dtype, (False,False,False,False,False))() ] )
 
     def infer_shape(self, node, input_shapes):
-        V,d,W_shape, dCdH = node.inputs
+        V, d, W_shape, dCdH = node.inputs
         return [ ( W_shape[0], W_shape[1], W_shape[2], W_shape[3], W_shape[4] ) ]
 
-    def grad(self,inputs, output_gradients):
-        C,d, WShape, B = inputs
-        dLdA ,= output_gradients
+    def connection_pattern(self, node):
 
-        z = T.zeros_like(C[0,0,0,0,:])
-        dLdC = convTransp3D( dLdA, z, d, B, C.shape[1:4])
-        dLdd = None #not differentiable, since d is not continuous
-        dLdWShape = None #not differentiable, since d is not continuous
-        dLdB = conv3D( C, dLdA, T.zeros_like(B[0,0,0,0,:]), d)
+        return [[True], [True], [False], [True]]
 
-        return [ dLdC, dLdd, dLdWShape, dLdB ]
+    def grad(self, inputs, output_gradients):
+        C, d, WShape, B = inputs
+        dLdA, = output_gradients
+
+        z = T.zeros_like(C[0, 0, 0, 0, :])
+        dLdC = convTransp3D(dLdA, z, d, B, C.shape[1:4])
+        # d actually does affect the outputs, so it's not disconnected
+        dLdd = grad_undefined(self, 1, d)
+        # The shape of the weights doesn't affect the output elements
+        dLdWShape = DisconnectedType()()
+        dLdB = conv3D(C, dLdA, T.zeros_like(B[0, 0, 0, 0, :]), d)
+
+        return [dLdC, dLdd, dLdWShape, dLdB]
 
     def perform(self, node, inputs, output_storage):
         V, d, WShape, dCdH = inputs
@@ -64,17 +72,15 @@ class ConvGrad3D(theano.Op):
 
         #print 'computing output of shape '+str(WShape)
 
-
-
-        for k in xrange(0,WShape[1]):
-            for l in xrange(0,WShape[2]):
-                for m in xrange(0,WShape[3]):
-                    for i in xrange(0,batchSize):
-                        for p in xrange(0,outputHeight):
-                            for q in xrange(0,outputWidth):
-                                for r in xrange(0,outputDur):
-                                    for j in xrange(0,WShape[0]):
-                                        for z in xrange(0,WShape[4]):
+        for k in xrange(0, WShape[1]):
+            for l in xrange(0, WShape[2]):
+                for m in xrange(0, WShape[3]):
+                    for i in xrange(0, batchSize):
+                        for p in xrange(0, outputHeight):
+                            for q in xrange(0, outputWidth):
+                                for r in xrange(0, outputDur):
+                                    for j in xrange(0, WShape[0]):
+                                        for z in xrange(0, WShape[4]):
                                             dCdW[j,k,l,m,z] +=  dCdH[i,p,q,r,j] * V[i,dr*p+k,dc*q+l,dt*r+m,z]
 
         output_storage[0][0] = dCdW
@@ -89,7 +95,7 @@ class ConvGrad3D(theano.Op):
 
         dCdW = outputs[0]
 
-        codeSource =  """
+        codeSource = """
             ///////////// < code generated by ConvGradW3D >
 
             //printf("\t\t\t\tConvGradW3D c code\\n");
@@ -269,7 +275,7 @@ class ConvGrad3D(theano.Op):
             ///////////// < /code generated by ConvGradW3D >
         """
 
-        return strutil.renderString(codeSource,locals())
+        return strutil.renderString(codeSource, locals())
 
 
 convGrad3D = ConvGrad3D()

theano/tensor/nnet/ConvTransp3D.py

@@ -2,10 +2,13 @@ import numpy as N
 from theano.tensor import basic as T
 from theano.misc import strutil
 import theano
+from theano.gradient import grad_undefined
+from theano.gradient import DisconnectedType
+
 
 class ConvTransp3D(theano.Op):
     """ "Transpose" of Conv3D (Conv3D implements multiplication by an implicitly defined matrix W. This implements multiplication by its transpose) """
-    def __eq__(self,other):
+    def __eq__(self, other):
         return type(self) == type(other)
 
     def __hash__(self):
@@ -14,7 +17,7 @@ class ConvTransp3D(theano.Op):
     def c_code_cache_version(self):
         return (3,)
 
-    def make_node(self, W, b, d, H, RShape = None):
+    def make_node(self, W, b, d, H, RShape=None):
         """
         :param W: Weights, filter
         :param b: bias, shape == (W.shape[0],)
@@ -28,7 +31,7 @@ class ConvTransp3D(theano.Op):
         if RShape:
             RShape_ = T.as_tensor_variable(RShape)
         else:
-            RShape_ = T.as_tensor_variable([-1,-1,-1])
+            RShape_ = T.as_tensor_variable([-1, -1, -1])
 
         return theano.Apply(self, inputs=[W_,b_,d_,H_, RShape_], outputs = [ T.TensorType(H_.dtype, (False,False,False,False,False))() ] )
 
@@ -36,22 +39,25 @@ class ConvTransp3D(theano.Op):
         flags = ['-Werror']
         return flags
 
-
     def infer_shape(self, node, input_shapes):
-        W,b,d,H,RShape = node.inputs
+        W, b, d, H, RShape = node.inputs
         W_shape, b_shape, d_shape, H_shape, RShape_shape = input_shapes
         return [(H_shape[0],  RShape[0], RShape[1], RShape[2], W_shape[4])]
 
-    def grad(self,inputs, output_gradients):
-        W,b,d,H, RShape = inputs
-        dCdR ,= output_gradients
-        dCdH = conv3D( dCdR, W, T.zeros_like(H[0,0,0,0,:]), d)
-        WShape = W.shape
-        dCdW = convGrad3D(dCdR,d,WShape,H)
-        dCdb = T.sum(dCdR,axis=(0,1,2,3))
-        dCdd = None #not differentiable, since d is not continuous
-        dCdRShape = None #not differentiable, since RShape is not continuous
+    def connection_pattern(self, node):
+        return [[True], [True], [True], [True], [False]]
 
+    def grad(self, inputs, output_gradients):
+        W, b, d, H, RShape = inputs
+        dCdR, = output_gradients
+        dCdH = conv3D(dCdR, W, T.zeros_like(H[0, 0, 0, 0, :]), d)
+        WShape = W.shape
+        dCdW = convGrad3D(dCdR, d, WShape, H)
+        dCdb = T.sum(dCdR, axis=(0, 1, 2, 3))
+        # not differentiable, since d affects the output elements
+        dCdd = grad_undefined(self, 2, d)
+        # disconnected, since RShape just determines the output shape
+        dCdRShape = DisconnectedType()()
 
         if 'name' in dir(dCdR) and dCdR.name is not None:
             dCdR_name = dCdR.name
@@ -76,15 +82,14 @@ class ConvTransp3D(theano.Op):
 
         dCdW.name = 'ConvTransp3D_dCdW.H='+H_name+',dCdR='+dCdR_name+',W='+W_name
         dCdb.name = 'ConvTransp3D_dCdb.H='+H_name+',dCdR='+dCdR_name+',W='+W_name+',b='+b_name
-        dCdH.name = 'ConvTransp3D_dCdH.H='+H_name+',dCdR='+dCdR_name
-
-        return [ dCdW,  dCdb, dCdd, dCdH, dCdRShape ]
+        dCdH.name = 'ConvTransp3D_dCdH.H=' + H_name + ',dCdR=' + dCdR_name
 
+        return [dCdW,  dCdb, dCdd, dCdH, dCdRShape]
 
     def perform(self, node, inputs, output_storage):
         W, b, d, H, RShape = inputs
 #        print "\t\t\t\tConvTransp3D python code"
-        output_storage[0][0] = computeR(W,b,d,H,RShape)
+        output_storage[0][0] = computeR(W, b, d, H, RShape)
 
     def c_code(self, node, nodename, inputs, outputs, sub):
         W, b, d, H, RShape = inputs
@@ -321,33 +326,35 @@ class ConvTransp3D(theano.Op):
                ///////////// < /code generated by ConvTransp3D >
                      """
 
-        return strutil.renderString(codeSource,locals())
+        return strutil.renderString(codeSource, locals())
 
 
 convTransp3D = ConvTransp3D()
 
 #If the input size wasn't a multiple of D we may need to cause some automatic padding to get the right size of reconstruction
-def computeR(W,b,d,H,Rshape = None):
+
+
+def computeR(W, b, d, H, Rshape=None):
     assert len(W.shape) == 5
     assert len(H.shape) == 5
     assert len(b.shape) == 1
     assert len(d) == 3
 
-
-    outputChannels,  filterHeight, filterWidth, filterDur, inputChannels = W.shape
-    batchSize, outputHeight, outputWidth, outputDur, outputChannelsAgain = H.shape
+    outputChannels,  filterHeight, filterWidth, filterDur, \
+        inputChannels = W.shape
+    batchSize, outputHeight, outputWidth, outputDur, \
+        outputChannelsAgain = H.shape
     assert outputChannelsAgain == outputChannels
     assert b.shape[0] == inputChannels
 
-
-    dr,dc,dt = d
+    dr, dc, dt = d
     assert dr > 0
     assert dc > 0
     assert dt > 0
 
-    videoHeight = (outputHeight-1) * dr + filterHeight
-    videoWidth = (outputWidth-1) * dc + filterWidth
-    videoDur = (outputDur-1) * dt + filterDur
+    videoHeight = (outputHeight - 1) * dr + filterHeight
+    videoWidth = (outputWidth - 1) * dc + filterWidth
+    videoDur = (outputDur - 1) * dt + filterDur
 
     if Rshape is not None and Rshape[0] != -1:
         if Rshape[0] < videoHeight:
@@ -364,24 +371,27 @@ def computeR(W,b,d,H,Rshape = None):
 
     #print "video size: "+str((videoHeight, videoWidth, videoDur))
 
-    R =  N.zeros( (batchSize, videoHeight,
-            videoWidth, videoDur, inputChannels ) , dtype=H.dtype)
+    R = N.zeros((batchSize, videoHeight,
+            videoWidth, videoDur, inputChannels), dtype=H.dtype)
 
     #R[i,j,r,c,t] = b_j + sum_{rc,rk | d \circ rc + rk = r} sum_{cc,ck | ...} sum_{tc,tk | ...} sum_k W[k, j, rk, ck, tk] * H[i,k,rc,cc,tc]
-    for i in xrange(0,batchSize):
+    for i in xrange(0, batchSize):
         #print '\texample '+str(i+1)+'/'+str(batchSize)
-        for j in xrange(0,inputChannels):
+        for j in xrange(0, inputChannels):
             #print '\t\tfeature map '+str(j+1)+'/'+str(inputChannels)
-            for r in xrange(0,videoHeight):
+            for r in xrange(0, videoHeight):
                 #print '\t\t\trow '+str(r+1)+'/'+str(videoHeight)
-                for c in xrange(0,videoWidth):
-                    for t in xrange(0,videoDur):
-                        R[i,r,c,t,j] = b[j]
+                for c in xrange(0, videoWidth):
+                    for t in xrange(0, videoDur):
+                        R[i, r, c, t, j] = b[j]
 
-                        ftc = max([0, int(N.ceil(float(t-filterDur +1  )/float(dt))) ])
-                        fcc = max([0, int(N.ceil(float(c-filterWidth +1)/float(dc))) ])
+                        ftc = max([0, int(N.ceil(
+                            float(t - filterDur + 1) / float(dt)))])
+                        fcc = max([0, int(N.ceil(
+                            float(c - filterWidth + 1) / float(dc)))])
 
-                        rc =  max([0, int(N.ceil(float(r-filterHeight+1)/float(dr))) ])
+                        rc = max([0, int(N.ceil(
+                            float(r - filterHeight + 1) / float(dr)))])
                         while rc < outputHeight:
                             rk = r - rc * dr
                             if rk < 0:
@@ -399,20 +409,21 @@ def computeR(W,b,d,H,Rshape = None):
                                     if tk < 0:
                                         break
 
-                                    R[i,r,c,t,j] += N.dot(W[:,rk,ck,tk,j], H[i,rc,cc,tc,:] )
+                                    R[
+                                        i,r,c,t,j] += N.dot(W[:,rk,ck,tk,j], H[i,rc,cc,tc,:] )
 
                                     tc += 1
-                                "" #close loop over tc
+                                ""  # close loop over tc
                                 cc += 1
-                            "" #close loop over cc
+                            ""  # close loop over cc
 
                             rc += 1
-                        "" #close loop over rc
-                    "" #close loop over t
-                "" #close loop over c
-            "" #close loop over r
-        "" #close loop over j
-    "" #close loop over i
+                        ""  # close loop over rc
+                    ""  # close loop over t
+                ""  # close loop over c
+            ""  # close loop over r
+        ""  # close loop over j
+    ""  # close loop over i
 
     return R
 

theano/tensor/nnet/nnet.py

@@ -15,6 +15,7 @@ from theano.gof import Apply
 
 from theano.tensor.nnet.sigm import sigmoid, softplus
 from theano.gradient import DisconnectedType
+from theano.gradient import grad_not_implemented
 
 
 ############
@@ -79,7 +80,7 @@ class SoftmaxWithBias(gof.Op):
         g_sm, = grads
 
         if isinstance(g_sm.type, DisconnectedType):
-            return [ DisconnectedType()(), DisconnectedType()() ]
+            return [DisconnectedType()(), DisconnectedType()()]
 
         sm = softmax_with_bias(x, b)
         dx = softmax_grad(g_sm, sm)
@@ -560,8 +561,8 @@ if 0:
                             axis = ds_input.owner.op.axis
                             sum_input = ds_input.owner.inputs[0]
 
-                        if ((ds_order!=(0,'x')) or
-                            (axis!=(1,)) or
+                        if ((ds_order != (0, 'x')) or
+                            (axis != (1,)) or
                             (sum_input is not prod_term)):
                             rest.append(add_in)
                             #print 'ds_order =', ds_order
@@ -712,16 +713,20 @@ class CrossentropySoftmaxArgmax1HotWithBias(gof.Op):
         am_shp = idx_shp
         return [nll_shp, sm_shp, am_shp]
 
+    def connection_pattern(self, node):
+
+        return [[True, True, True],  # x
+                [True, True, True],  # b
+                [False, False, True]]  # y_idx
+
     def grad(self, inp, grads):
         x, b, y_idx = inp
         g_nll, g_sm, g_am = grads
 
-
         dx_terms = []
         db_terms = []
         d_idx_terms = []
 
-
         if not isinstance(g_nll.type, DisconnectedType):
             nll, sm = crossentropy_softmax_1hot_with_bias(x, b, y_idx)
             dx = crossentropy_softmax_1hot_with_bias_dx(g_nll, sm, y_idx)
@@ -739,7 +744,7 @@ class CrossentropySoftmaxArgmax1HotWithBias(gof.Op):
             db_terms.append(b.zeros_like())
             d_idx_terms.append(y_idx.zeros_like())
 
-        def fancy_sum( terms ):
+        def fancy_sum(terms):
             if len(terms) == 0:
                 return DisconnectedType()()
             rval = terms[0]
@@ -747,8 +752,8 @@ class CrossentropySoftmaxArgmax1HotWithBias(gof.Op):
                 rval = rval + term
             return rval
 
-        return [ fancy_sum(terms) for terms in
-                [dx_terms, db_terms, d_idx_terms ] ]
+        return [fancy_sum(terms) for terms in
+                [dx_terms, db_terms, d_idx_terms]]
 
     def c_headers(self):
         return ['<iostream>', '<cmath>']
@@ -897,7 +902,7 @@ class CrossentropySoftmax1HotWithBiasDx (gof.Op):
                     sm, tensor.fill(dy, -1), y_idx_range, y_idx),
                 axis=1)
         g_sm = dy.dimshuffle(0, 'x') * g_dx
-        g_y_idx = None
+        g_y_idx = grad_not_implemented(self, 2, y_idx)
         return [g_dy, g_sm, g_y_idx]
 
     def c_code_cache_version(self):
@@ -1136,7 +1141,7 @@ class CrossentropyCategorical1Hot(gof.Op):
         coding, one_of_n = inp
         g_y, = grads
         return [crossentropy_categorical_1hot_grad(g_y, coding, one_of_n),
-                None]
+                grad_not_implemented(self, 1, one_of_n)]
 
 crossentropy_categorical_1hot = CrossentropyCategorical1Hot()
 
@@ -1325,7 +1330,6 @@ def local_advanced_indexing_crossentropy_onehot(node):
             except Exception:
                 pass
 
-
     if sm is not None and sm.owner and sm.owner.op in (softmax,
                                                        softmax_with_bias):
         sm_w_bias = local_softmax_with_bias.transform(sm.owner)
@@ -1481,7 +1485,8 @@ def local_advanced_indexing_crossentropy_onehot_grad(node):
 
             if adv_subtensor is not None:
                 try:
-                    maybe_sm, maybe_rows, maybe_labels = adv_subtensor.owner.inputs
+                    maybe_sm, maybe_rows, \
+                        maybe_labels = adv_subtensor.owner.inputs
                 except Exception:
                     return
 
@@ -1691,7 +1696,6 @@ class Prepend_scalar_constant_to_each_row(gof.Op):
         shp = (in_shapes[0][0], in_shapes[0][1] + 1)
         return [shp]
 
-
     def grad(self, inp, grads):
         mat, = inp
         goutput, = grads
@@ -1758,18 +1762,19 @@ prepend_1_to_each_row = Prepend_scalar_constant_to_each_row(1.)
 #numerically stabilize log softmax (X)
 # as  X-X.max(axis=1).dimshuffle(0,'x') - log(exp(X-X.max(axis=1).dimshuffle(0,'x')).sum(axis=1)).dimshuffle(0,'x)
 def make_out_pattern(X):
-    stabilized_X = X - X.max(axis=1).dimshuffle(0,'x')
-    out_var = stabilized_X - tensor.log(tensor.exp(stabilized_X).sum(axis=1)).dimshuffle(0,'x')
+    stabilized_X = X - X.max(axis=1).dimshuffle(0, 'x')
+    out_var = stabilized_X - tensor.log(tensor.exp(stabilized_X).sum(
+        axis=1)).dimshuffle(0, 'x')
     #tell DEBUG_MODE that it's OK if the original graph produced NaN and the optimized graph does not
     out_var.values_eq_approx = out_var.type.values_eq_approx_remove_nan
     return out_var
 
 
-local_log_softmax = gof.PatternSub( in_pattern = (tensor.log, (softmax, 'x')),
-                                    out_pattern = (make_out_pattern, 'x'),
+local_log_softmax = gof.PatternSub(in_pattern=(tensor.log, (softmax, 'x')),
+                                    out_pattern=(make_out_pattern, 'x'),
                                    allow_multiple_clients=True)
 
 #don't do register_stabilize, this is to make local_log_softmax run
 #only after another more specific optimization that stabilizes cross entropy
 #opt.register_stabilize(local_log_softmax, name = 'local_log_softmax')
-opt.register_specialize(local_log_softmax, name = 'local_log_softmax')
+opt.register_specialize(local_log_softmax, name='local_log_softmax')

theano/tensor/nnet/sigm.py

@@ -30,13 +30,20 @@ class ScalarSigmoid(scalar.UnaryScalarOp):
         if x > 30.0:
             return 1.0
         return 1.0 / (1.0 + numpy.exp(-x))
+
     def impl(self, x):
         return ScalarSigmoid.st_impl(x)
+
     def grad(self, inp, grads):
         x, = inp
         gz, = grads
         y = scalar_sigmoid(x)
-        return [gz * y * (1.0 - y)]
+        rval = gz * y * (1.0 - y)
+
+        assert rval.type.dtype.find('float') != -1
+
+        return [rval]
+
     def c_code(self, node, name, inp, out, sub):
         x, = inp
         z, = out
@@ -50,6 +57,7 @@ class ScalarSigmoid(scalar.UnaryScalarOp):
             return """%(z)s = %(x)s < -709.0 ? 0.0 : %(x)s > 19.0 ? 1.0 : 1.0 /(1.0+exp(-%(x)s));""" % locals()
         else:
             raise NotImplementedError('only floatingpoint is implemented')
+
     def c_code_cache_version(self):
         v = super(ScalarSigmoid, self).c_code_cache_version()
         if v:
@@ -61,7 +69,7 @@ sigmoid = elemwise.Elemwise(scalar_sigmoid, name='sigmoid')
 
 sigmoid_inplace = elemwise.Elemwise(
         ScalarSigmoid(scalar.transfer_type(0)),
-        inplace_pattern={0:0},
+        inplace_pattern={0: 0},
         name='sigmoid_inplace',
         )
 
@@ -76,12 +84,15 @@ class ScalarSoftplus(scalar.UnaryScalarOp):
         if x > 30.0:
             return x
         return numpy.log1p(numpy.exp(x))
+
     def impl(self, x):
         return ScalarSoftplus.static_impl(x)
+
     def grad(self, inp, grads):
         x, = inp
         gz, = grads
         return [gz * scalar_sigmoid(x)]
+
     def c_code(self, node, name, inp, out, sub):
         x, = inp
         z, = out
@@ -95,27 +106,29 @@ class ScalarSoftplus(scalar.UnaryScalarOp):
             return """%(z)s = %(x)s < -745.0 ? 0.0 : %(x)s > 16.0 ? %(x)s : log1p(exp(%(x)s));""" % locals()
         else:
             raise NotImplementedError('only floatingpoint is implemented')
+
     def c_code_cache_version(self):
         v = super(ScalarSoftplus, self).c_code_cache_version()
         if v:
             return (2,) + v
         else:
             return v
-scalar_softplus = ScalarSoftplus(scalar.upgrade_to_float, name='scalar_softplus')
+scalar_softplus = ScalarSoftplus(scalar.upgrade_to_float, name=                                                                                                                                                                                                        'scalar_softplus')
 softplus = elemwise.Elemwise(scalar_softplus, name='softplus')
 
 pprint.assign(softplus, printing.FunctionPrinter('softplus'))
 
+
 def _skip_mul_1(r):
     if r.owner and r.owner.op == tensor.mul:
-        not_is_1 = [i for i in r.owner.inputs if not _is_1(i) ]
-        if len(not_is_1)==1:
+        not_is_1 = [i for i in r.owner.inputs if not _is_1(i)]
+        if len(not_is_1) == 1:
             return not_is_1[0]
 
 logsigm_to_softplus = gof.PatternSub(
     (tensor.log, (sigmoid, 'x')),
     (tensor.neg, (softplus, (tensor.neg, 'x'))),
-    allow_multiple_clients = True,
+    allow_multiple_clients=True,
     skip_identities_fn=_skip_mul_1)
 
 
@@ -131,21 +144,22 @@ def _is_1(expr):
 log1msigm_to_softplus = gof.PatternSub(
     (tensor.log,
         (tensor.sub,
-            dict(pattern='y', constraint = _is_1),
+            dict(pattern='y', constraint=_is_1),
             (sigmoid, 'x'))),
     (tensor.neg, (softplus, 'x')),
-    allow_multiple_clients = True,
+    allow_multiple_clients=True,
     skip_identities_fn=_skip_mul_1)
 
 log1pexp_to_softplus = gof.PatternSub(
     (tensor.log1p,
      (tensor.exp, 'x')),
     (softplus, 'x'),
-    allow_multiple_clients = True)
+    allow_multiple_clients=True)
+
+opt.register_stabilize(logsigm_to_softplus, name='logsigm_to_softplus')
+opt.register_stabilize(log1msigm_to_softplus, name='log1msigm_to_softplus')
+opt.register_stabilize(log1pexp_to_softplus, name='log1pexp_to_softplus')
 
-opt.register_stabilize(logsigm_to_softplus, name = 'logsigm_to_softplus')
-opt.register_stabilize(log1msigm_to_softplus, name = 'log1msigm_to_softplus')
-opt.register_stabilize(log1pexp_to_softplus, name = 'log1pexp_to_softplus')
 
 def is_1pexp(t):
     """
@@ -239,7 +253,7 @@ def partition_num_or_denom(r, f):
         else:
             neg_t, f_t = f_t
             f_terms.append(f_t)
-            neg ^= neg_t #bit flip if neg_t is true
+            neg ^= neg_t  # bit flip if neg_t is true
     return f_terms, rest, neg
 
 
@@ -291,7 +305,8 @@ def local_exp_over_1_plus_exp(node):
         #find all the exp() terms in the numerator
         num, denom = node.inputs
         num_exp_x, num_rest, num_neg = partition_num_or_denom(num, is_exp)
-        denom_1pexp, denom_rest, denom_neg = partition_num_or_denom(denom, is_1pexp)
+        denom_1pexp, denom_rest, \
+            denom_neg = partition_num_or_denom(denom, is_1pexp)
 
         sigmoids = []
         for t in denom_1pexp:
@@ -303,7 +318,7 @@ def local_exp_over_1_plus_exp(node):
                 # case: 1/(1+exp(x))
                 sigmoids.append(sigmoid(-t))
 
-        if not sigmoids: # we didn't find any.  abort
+        if not sigmoids:  # we didn't find any.  abort
             return
         # put the new numerator together
         new_num = sigmoids + [tensor.exp(t) for t in num_exp_x] + num_rest
@@ -322,6 +337,7 @@ def local_exp_over_1_plus_exp(node):
         else:
             return [new_num / tensor.mul(*denom_rest)]
 
+
 def parse_mul_tree(root):
     """
     Parse a tree of multiplications starting at the given root.
@@ -504,7 +520,7 @@ def perform_sigm_times_exp(tree, exp_x=None, exp_minus_x=None, sigm_x=None,
         sigm_minus_x = []
     if full_tree is None:
         full_tree = tree
-    if False: # Debug code.
+    if False:  # Debug code.
         print '<perform_sigm_times_exp>'
         print '  full_tree   = %s' % full_tree
         print '  tree        = %s' % tree
@@ -613,10 +629,13 @@ def local_inv_1_plus_exp(node):
                 if nonconsts[0].owner and nonconsts[0].owner.op == tensor.exp:
                     if scalars and numpy.allclose(numpy.sum(scalars), 1):
                         return opt._fill_chain(
-                                sigmoid(tensor.neg(nonconsts[0].owner.inputs[0])),
+                                sigmoid(
+                                    tensor.neg(nonconsts[0].owner.inputs[0])),
                                 scalar_inputs)
 
 # Registration is below, and conditional.
+
+
 @gof.local_optimizer([tensor.sub])
 def local_1msigmoid(node):
     """
@@ -625,7 +644,7 @@ def local_1msigmoid(node):
     if node.op == tensor.sub:
         sub_l, sub_r = node.inputs
         if len(sub_r.clients) > 1:
-            return # graph is using both sigm and 1-sigm
+            return  # graph is using both sigm and 1-sigm
         if sub_r.owner and sub_r.owner.op == sigmoid:
             try:
                 val_l = opt.get_constant_value(sub_l)
@@ -678,13 +697,14 @@ if 0:
                         assert t0.owner.op == div
                         t0top, t0bot = t0.owner.inputs
                         t1top, t1bot = t1.owner.inputs
-                        rval.append(div(mul(*(t0top+t1top)), mul(*(t0bot+t1bot))))
+                        rval.append(div(mul(*(
+                            t0top + t1top)), mul(*(t0bot + t1bot))))
 
                         if len(rval) > 100:
                             # This loop can be exponentially long.
                             # aborting
                             return []
-        elif len(node.outputs)>1:
+        elif len(node.outputs) > 1:
             return []
         else:
             return [node.outputs[0]]

theano/tensor/opt.py

@@ -542,15 +542,12 @@ class MakeVector(T.Op):
     def grad(self, inputs, output_gradients):
         # If the output is of an integer dtype, no gradient shall pass
         if 'int' in self.dtype:
-            return [None] * len(inputs)
+            return [ipt.zeros_like().astype(theano.config.floatX)
+                    for ipt in inputs]
 
         grads = []
         for i, inp in enumerate(inputs):
-            if 'int' in inp.dtype:
-                # No gradient wrt integer inputs
-                grads.append(None)
-            else:
-                grads.append(output_gradients[0][i])
+            grads.append(output_gradients[0][i])
         return grads
 
     def R_op(self, inputs, eval_points):
@@ -1914,6 +1911,8 @@ def local_subtensor_of_alloc(node):
 
     nw_val = val[tuple(val_slices)]
     nw_dims += dims[len(slices):]
+    if nw_val.ndim > len(nw_dims):
+        return False
     rval = T.alloc(nw_val, *nw_dims)
     if type(rval) not in (list, tuple):
         rval = [rval]

theano/tensor/randomstreams.py

@@ -136,7 +136,7 @@ class RandomStreams(Component, raw_random.RandomStreamsBase):
 
     """
 
-    def __init__(self, seed=None, no_warn = False):
+    def __init__(self, seed=None, no_warn=False):
         """:type seed: None or int
 
         :param seed: a default seed to initialize the RandomState
@@ -146,7 +146,7 @@ class RandomStreams(Component, raw_random.RandomStreamsBase):
         """
         if not no_warn:
             deprecation_warning()
-        super(RandomStreams, self).__init__(no_warn = True)
+        super(RandomStreams, self).__init__(no_warn=True)
         self.random_state_variables = []
         self.default_instance_seed = seed
 
@@ -164,7 +164,6 @@ class RandomStreams(Component, raw_random.RandomStreamsBase):
     def build(self, mode, memo):
         """override `Component.build` """
         if self not in memo:
-            print 'creating RandomStreamsInstance'
             memo[self] = RandomStreamsInstance(self, memo,
                                                self.default_instance_seed)
         return memo[self]

theano/tensor/tests/test_elemwise.py

@@ -47,7 +47,8 @@ class test_DimShuffle(unittest_tools.InferShapeTester):
             #test that DimShuffle.infer_shape work correctly
             x = TensorType('float64', ib)('x')
             e = DimShuffle(ib, shuffle)(x)
-            f = copy(linker).accept(FunctionGraph([x], [e.shape])).make_function()
+            f = copy(linker).accept(FunctionGraph([x], [e.
+                shape])).make_function()
             assert all(f(numpy.ones(xsh))) == all(zsh)
 
         # Test when we drop a axis that is not broadcastable
@@ -125,7 +126,8 @@ class test_Broadcast(unittest.TestCase):
                 x = TensorType('float64', [(entry == 1) for entry in xsh])('x')
                 y = TensorType('float64', [(entry == 1) for entry in ysh])('y')
                 e = Elemwise(scalar.add)(x, y)
-                f = copy(linker).accept(FunctionGraph([x, y], [e.shape])).make_function()
+                f = copy(linker).accept(FunctionGraph([x,
+                     y], [e.shape])).make_function()
                 assert tuple(f(xv, yv)) == tuple(zv.shape)
 
     def with_linker_inplace(self, linker):
@@ -154,7 +156,8 @@ class test_Broadcast(unittest.TestCase):
                 x = TensorType('float64', [(entry == 1) for entry in xsh])('x')
                 y = TensorType('float64', [(entry == 1) for entry in ysh])('y')
                 e = Elemwise(scalar.Add(scalar.transfer_type(0)), {0: 0})(x, y)
-                f = copy(linker).accept(FunctionGraph([x, y], [e.shape])).make_function()
+                f = copy(linker).accept(FunctionGraph([x,
+                     y], [e.shape])).make_function()
                 xv = numpy.asarray(numpy.random.rand(*xsh))
                 yv = numpy.asarray(numpy.random.rand(*ysh))
                 zv = xv + yv
@@ -349,7 +352,8 @@ class test_CAReduce(unittest_tools.InferShapeTester):
                     e = tensor_op(x, axis=tosum)
                 if tosum is None:
                     tosum = range(len(xsh))
-                f = copy(linker).accept(FunctionGraph([x], [e.shape])).make_function()
+                f = copy(linker).accept(FunctionGraph([x],
+                     [e.shape])).make_function()
                 if not(scalar_op in [scalar.maximum, scalar.minimum] and
                        ((xsh == () or numpy.prod(xsh) == 0))):
                     assert all(f(xv) == zv.shape)
@@ -459,7 +463,8 @@ class test_Prod(unittest.TestCase):
 
         # including zeros, as the case with zeros is important
         # (and special cases: 1 zero in the row, more than 1 zero in the row)
-        x_val = numpy.asarray([[1,2,3],[4,5,6],[7,8,9]], dtype='float32')
+        x_val = numpy.asarray([[1, 2, 3], [4, 5, 6], [7, 8, 9]],
+             dtype='float32')
         x = theano.tensor.dmatrix()
         # now with verify_grad
         unittest_tools.verify_grad(Prod(axis=1), [x_val], mode=self.mode)
@@ -471,26 +476,28 @@ class test_Prod(unittest.TestCase):
 
         unittest_tools.verify_grad(fn, [x_val], mode=self.mode)
 
-
     def test_verify_grad_with_zeros(self):
         # including zeros, as the case with zeros is important
         # (and special cases: 1 zero in the row, more than 1 zero in the row)
-        x_val = numpy.asarray([[1.,2.,3.],[0.,5.,6.],[0.,0.,9.]], dtype='float32')
+        x_val = numpy.asarray([[1., 2., 3.], [0., 5., 6.], [0., 0., 9.]],
+             dtype='float32')
         x = theano.tensor.dmatrix()
 
         # sanity check
         x2 = theano.tensor.dmatrix()
         p = Prod(axis=1)(x)
         p2 = Prod(axis=1)(x2)
-        fn = theano.function([x,x2],[p-p2], mode=self.mode)
+        fn = theano.function([x, x2], [p - p2], mode=self.mode)
         #print "hand computed diff for each row"
-        x2_val = numpy.asarray([[1., 2., 3.003], [0.003,5.,6], [0.,0.,9.01]])
+        x2_val = numpy.asarray([[1., 2., 3.003], [0.003, 5., 6], [
+            0., 0., 9.01]])
         #print fn(x_val, x2_val)
-        fn2 = theano.function([x],[theano.tensor.grad(p.sum(),x)], mode=self.mode)
+        fn2 = theano.function([x], [theano.tensor.grad(p.sum(), x)],
+             mode=self.mode)
         #print "real grad"
         #print fn2(x_val)
-        fn3 = theano.function([x],[p], mode=self.mode)
-        assert numpy.allclose(fn3(x_val), [6.,0.,0.])
+        fn3 = theano.function([x], [p], mode=self.mode)
+        assert numpy.allclose(fn3(x_val), [6., 0., 0.])
 
         # now with verify_grad
         unittest_tools.verify_grad(Prod(axis=1), [x_val], mode=self.mode)
@@ -511,10 +518,10 @@ class test_Prod(unittest.TestCase):
 
     def test_prod_without_zeros(self):
         x = theano.tensor.dmatrix()
-        x_val = numpy.array([[1,2,3],[0,5,6],[0,0,9]], dtype='float32')
+        x_val = numpy.array([[1, 2, 3], [0, 5, 6], [0, 0, 9]], dtype='float32')
         pwz = ProdWithoutZeros(axis=1)(x)
         fn = theano.function([x], pwz, mode=self.mode)
-        assert numpy.allclose(fn(x_val), [6,30,9])
+        assert numpy.allclose(fn(x_val), [6, 30, 9])
 
         pwz_a0 = ProdWithoutZeros(axis=0)(x)
         fn_a0 = theano.function([x], pwz_a0, mode=self.mode)
@@ -522,25 +529,30 @@ class test_Prod(unittest.TestCase):
 
     def test_other_grad_tests(self):
         x = theano.tensor.dmatrix()
-        x_val1 = numpy.array([[1,2,3],[0,5,6],[0,0,9]], dtype='float32')
-        x_val2 = numpy.array([[1,2,0],[0,5,6],[7,8,9],[9,10,0]], dtype='float32')
+        x_val1 = numpy.array([[1, 2, 3], [0, 5, 6], [0, 0, 9]],
+             dtype='float32')
+        x_val2 = numpy.array([[1, 2, 0], [0, 5, 6], [7, 8, 9], [9, 10, 0]],
+             dtype='float32')
         rng = rng = numpy.random.RandomState(43)
 
         p = Prod(axis=1)
         grad_p = theano.tensor.grad(p(x).sum(), x)
         grad_fn = theano.function([x], grad_p, mode=self.mode)
-        assert numpy.allclose(grad_fn(x_val1), [[6.,3.,2.],[30.,0.,0.],[0.,0.,0.]])
-        assert numpy.allclose(grad_fn(x_val2), [[0., 0., 2.], [30., 0., 0.], [72., 63., 56.], [0., 0., 90.]])
+        assert numpy.allclose(grad_fn(x_val1), [[6., 3., 2.], [30., 0.,
+            0.], [0., 0., 0.]])
+        assert numpy.allclose(grad_fn(x_val2), [[0., 0., 2.], [30.,
+             0., 0.], [72., 63., 56.], [0., 0., 90.]])
 
         p_axis0 = Prod(axis=0)
         grad_p_axis0 = theano.tensor.grad(p_axis0(x).sum(), x)
         grad_fn_axis0 = theano.function([x], grad_p_axis0, mode=self.mode)
-        assert numpy.allclose(grad_fn_axis0(x_val2), [[0., 400., 0.],[63., 160., 0.], [0., 100., 0.], [0., 80., 0.]])
+        assert numpy.allclose(grad_fn_axis0(x_val2), [[0., 400.,
+             0.], [63., 160., 0.], [0., 100., 0.], [0., 80., 0.]])
 
         tensor.verify_grad(p, [x_val1], rng=rng, mode=self.mode)
 
     def test_mul_without_zeros_zeros(self):
-        a = numpy.zeros((3,3))
+        a = numpy.zeros((3, 3))
 
         x = theano.tensor.dmatrix()
 
@@ -655,6 +667,7 @@ class T_sum_dtype(unittest.TestCase):
 
                 idx += 1
 
+
 class T_mean_dtype(unittest.TestCase):
     def test_mean_default_dtype(self):
         """
@@ -671,6 +684,7 @@ class T_mean_dtype(unittest.TestCase):
                 assert x.dtype == dtype, (x, x.dtype, dtype)
 
     def test_mean_custom_dtype(self):
+
         """
         Test the ability to provide your own output dtype for a mean.
         """
@@ -709,6 +723,7 @@ class T_mean_dtype(unittest.TestCase):
 
                 idx += 1
 
+
 class T_prod_dtype(unittest.TestCase):
     def test_prod_default_dtype(self):
         """
@@ -760,6 +775,7 @@ class T_prod_dtype(unittest.TestCase):
 
                 idx += 1
 
+
 class T_prod_without_zeros_dtype(unittest.TestCase):
     def test_prod_without_zeros_default_dtype(self):
         """
@@ -843,11 +859,8 @@ if __name__ == '__main__':
 """
 
 
-
 if __name__ == '__main__':
 
     t = TestElemwise('setUp')
     t.setUp()
     t.test_infer_shape()
-
-

theano/tensor/tests/test_naacl09.py

@@ -10,6 +10,8 @@ from theano import tensor as T, sparse as S
 import numpy as N
 import sys
 from theano.tests import unittest_tools
+from numpy.testing.noseclasses import KnownFailureTest
+
 
 def cross_entropy(target, output, axis=1):
     """
@@ -17,9 +19,12 @@ def cross_entropy(target, output, axis=1):
     @warning: OUTPUT and TARGET are reversed in tensor.nnet.binary_crossentropy
     """
     return -T.mean(target * T.log(output) + (1 - target) * T.log(1 - output), axis=axis)
+
+
 def quadratic(target, output, axis=1):
     return T.mean(T.sqr(target - output), axis=axis)
 
+
 class QuadraticDenoisingAA(module.Module):
     """Quadratic de-noising Auto-encoder
 
@@ -34,15 +39,15 @@ class QuadraticDenoisingAA(module.Module):
     """
 
     def __init__(self,
-            input = None,
+            input=None,
 #            regularize = False,
-            tie_weights = False,
-            n_quadratic_filters = 1,
-            _w1 = None,
-            _w2 = None,
-            _b1 = None,
-            _b2 = None,
-            _qfilters = None,
+            tie_weights=False,
+            n_quadratic_filters=1,
+            _w1=None,
+            _w2=None,
+            _b1=None,
+            _b2=None,
+            _qfilters=None,
             activation_function=NN.sigmoid,
             reconstruction_cost_function=cross_entropy):
         """
@@ -82,7 +87,8 @@ class QuadraticDenoisingAA(module.Module):
         # PARAMETERS
         if _qfilters is None:
             #self.qfilters = [theano.Member(T.dmatrix('q%i'%i)) for i in xrange(n_quadratic_filters)]
-            self.qfilters = [(T.dmatrix('q%i'%i)) for i in xrange(n_quadratic_filters)]
+            self.qfilters = [(T.dmatrix('q%i' % i))
+                 for i in xrange(n_quadratic_filters)]
         else:
             #self.qfilters = [theano.Member(q) for q in _qfilters]
             self.qfilters = [(q) for q in _qfilters]
@@ -90,7 +96,8 @@ class QuadraticDenoisingAA(module.Module):
         #self.w1 = theano.Member(T.matrix('w1')) if _w1 is None else theano.Member(_w1)
         if _w1 is None:
             self.w1 = (T.matrix('w1'))
-        else: self.w1 = (_w1)
+        else:
+            self.w1 = (_w1)
         if _w2 is None:
             if not tie_weights:
                 #self.w2 = theano.Member(T.matrix())
@@ -103,30 +110,30 @@ class QuadraticDenoisingAA(module.Module):
         #self.b1 = theano.Member(T.vector('b1')) if _b1 is None else theano.Member(_b1)
         if _b1 is None:
             self.b1 = (T.vector('b1'))
-        else: self.b1 = (_b1)
+        else:
+            self.b1 = (_b1)
         #self.b2 = theano.Member(T.vector('b2')) if _b2 is None else theano.Member(_b2)
         if _b2 is None:
             self.b2 = (T.vector('b2'))
-        else: self.b2 = (_b2)
+        else:
+            self.b2 = (_b2)
 
 #        # REGULARIZATION COST
 #        self.regularization = self.build_regularization()
 
-
         ### NOISELESS ###
-
         # HIDDEN LAYER
         def _act(x):
             if len(self.qfilters) > 0:
                 qsum = 10e-10   # helps to control the gradient in the square-root below
                 for qf in self.qfilters:
-                    qsum = qsum + T.dot(x, qf)**2
+                    qsum = qsum + T.dot(x, qf) ** 2
 
                 return T.dot(x, self.w1) + self.b1 + T.sqrt(qsum)
             else:
                 return T.dot(x, self.w1) + self.b1
 
-        self.hidden_activation = _act(self.input) #noise-free hidden
+        self.hidden_activation = _act(self.input)  # noise-free hidden
 
         self.hidden = self.hid_activation_function(self.hidden_activation)
 
@@ -143,7 +150,6 @@ class QuadraticDenoisingAA(module.Module):
 #        if self.regularize:
 #            self.cost = self.cost + self.regularization
 
-
         ### WITH NOISE ###
         self.corrupted_input = self.build_corrupted_input()
 
@@ -164,7 +170,6 @@ class QuadraticDenoisingAA(module.Module):
 #        if self.regularize:
 #            self.ncost = self.ncost + self.regularization
 
-
         # GRADIENTS AND UPDATES
         if self.tie_weights:
             self.params = [self.w1, self.b1, self.b2] + self.qfilters
@@ -172,7 +177,8 @@ class QuadraticDenoisingAA(module.Module):
             self.params = [self.w1, self.w2, self.b1, self.b2] + self.qfilters
 
         gradients = T.grad(self.ncost, self.params)
-        updates = dict((p, p - self.lr * g) for p, g in zip(self.params, gradients))
+        updates = dict((p, p - self.lr * g) for p, g in zip(self.
+            params, gradients))
 
         # INTERFACE METHODS
         #self.update = theano.Method(self.input, self.ncost, updates)
@@ -191,16 +197,17 @@ class QuadraticDenoisingAA(module.Module):
         filter's initial range)
         """
         if (input_size is None) ^ (hidden_size is None):
-            raise ValueError("Must specify input_size and hidden_size or neither.")
+            raise ValueError(
+                "Must specify input_size and hidden_size or neither.")
         super(QuadraticDenoisingAA, self)._instance_initialize(obj, {})
 
         obj.random.initialize()
         R = N.random.RandomState(unittest_tools.fetch_seed(seed))
         if input_size is not None:
             sz = (input_size, hidden_size)
-            inf = 1/N.sqrt(input_size)
-            hif = 1/N.sqrt(hidden_size)
-            obj.w1 = N.asarray(R.uniform(size = sz, low = -inf, high = inf),
+            inf = 1 / N.sqrt(input_size)
+            hif = 1 / N.sqrt(hidden_size)
+            obj.w1 = N.asarray(R.uniform(size=sz, low=-inf, high=inf),
                     dtype=config.floatX)
             if not self.tie_weights:
                 obj.w2 = N.asarray(
@@ -256,14 +263,17 @@ class SigmoidXEQuadraticDenoisingAA(QuadraticDenoisingAA):
     def _instance_initialize(self, obj, input_size, hidden_size, noise_level, seed, lr, qfilter_relscale):
 #        obj.l2_coef = 0.0
         obj.noise_level = N.asarray(noise_level, dtype=config.floatX)
-        super(SigmoidXEQuadraticDenoisingAA, self)._instance_initialize(obj, input_size, hidden_size, seed, lr, qfilter_relscale)
+        super(SigmoidXEQuadraticDenoisingAA, self)._instance_initialize(
+                obj, input_size, hidden_size, seed, lr, qfilter_relscale)
 
 QDAA = SigmoidXEQuadraticDenoisingAA
 
+
 class Loss01(object):
     def loss_01(self, x, targ):
         return N.mean(self.classify(x) != targ)
 
+
 class Module_Nclass(module.FancyModule):
     def _instance_initialize(mod_self, self, n_in, n_out, lr, seed):
         #self.component is the LogisticRegressionTemplate instance that built this guy.
@@ -279,29 +289,34 @@ class Module_Nclass(module.FancyModule):
         self.output_dimension = n_out
 
     def __init__(self, x=None, targ=None, w=None, b=None, lr=None, regularize=False):
-        super(Module_Nclass, self).__init__() #boilerplate
+        super(Module_Nclass, self).__init__()  # boilerplate
 
         #self.x = module.Member(x) if x is not None else T.matrix('input')
         if x is not None:
             self.x = (x)
-        else: self.x = T.matrix('input')
+        else:
+            self.x = T.matrix('input')
         #self.targ = module.Member(targ) if targ is not None else T.lvector()
         if targ is not None:
             self.targ = (targ)
-        else: self.targ = T.lvector()
+        else:
+            self.targ = T.lvector()
 
         #self.w = module.Member(w) if w is not None else module.Member(T.dmatrix())
         if w is not None:
             self.w = (w)
-        else: self.w = (T.dmatrix())
+        else:
+            self.w = (T.dmatrix())
         #self.b = module.Member(b) if b is not None else module.Member(T.dvector())
         if b is not None:
             self.b = (b)
-        else: self.b = (T.dvector())
+        else:
+            self.b = (T.dvector())
         #self.lr = module.Member(lr) if lr is not None else module.Member(T.dscalar())
         if lr is not None:
             self.lr = (lr)
-        else: self.lr = (T.dscalar())
+        else:
+            self.lr = (T.dscalar())
 
         self.params = [p for p in [self.w, self.b] if p.owner is None]
 
@@ -340,13 +355,14 @@ class Module_Nclass(module.FancyModule):
             #self.update = module.Method([self.input, self.targ], sum_xent,
                     #updates = dict((p, p - self.lr * g) for p, g in zip(self.params, gparams)))
 
+
 class ConvolutionalMLP(module.FancyModule):
     def __init__(self,
             window_size,
             n_quadratic_filters,
             activation_function,
             reconstruction_cost_function,
-            tie_weights = False,
+            tie_weights=False,
 #            _input,
 #            _targ
             ):
@@ -361,9 +377,9 @@ class ConvolutionalMLP(module.FancyModule):
         self.input_representations = []
         self.input_representations.append(QDAA(
                             input=self.inputs[0],
-                            tie_weights = tie_weights,
-                            n_quadratic_filters = n_quadratic_filters,
-                            activation_function = activation_function,
+                            tie_weights=tie_weights,
+                            n_quadratic_filters=n_quadratic_filters,
+                            activation_function=activation_function,
                             reconstruction_cost_function = reconstruction_cost_function
                         )
         )
@@ -372,9 +388,9 @@ class ConvolutionalMLP(module.FancyModule):
             self.input_representations.append(
                             QDAA(
                                 input=i,
-                                tie_weights = tie_weights,
-                                n_quadratic_filters = n_quadratic_filters,
-                                activation_function = activation_function,
+                                tie_weights=tie_weights,
+                                n_quadratic_filters=n_quadratic_filters,
+                                activation_function=activation_function,
                                 reconstruction_cost_function = reconstruction_cost_function,
                                 _w1 = self.input_representations[0].w1,
                                 _w2 = self.input_representations[0].w2,
@@ -383,14 +399,16 @@ class ConvolutionalMLP(module.FancyModule):
                                 _qfilters = self.input_representations[0].qfilters
                             )
             )
-            assert self.input_representations[-1].w1 is self.input_representations[0].w1
+            assert self.input_representations[-1].w1 is \
+                    self.input_representations[0].w1
 
-        self.input_representation = T.concatenate([i.hidden for i in self.input_representations], axis=1)
+        self.input_representation = T.concatenate([i.
+            hidden for i in self.input_representations], axis=1)
         self.hidden = QDAA(
-                        input = self.input_representation,
-                        tie_weights = tie_weights,
-                        n_quadratic_filters = n_quadratic_filters,
-                        activation_function = activation_function,
+                        input=self.input_representation,
+                        tie_weights=tie_weights,
+                        n_quadratic_filters=n_quadratic_filters,
+                        activation_function=activation_function,
                         reconstruction_cost_function = reconstruction_cost_function
                     )
         self.output = Module_Nclass(x=self.hidden.hidden, targ=self.targ)
@@ -407,11 +425,13 @@ class ConvolutionalMLP(module.FancyModule):
                         self.hidden.b1,
                         self.hidden.b2
                         ] + self.hidden.qfilters
-        input_pretraining_cost = sum(i.ncost for i in self.input_representations)
+        input_pretraining_cost = sum(i.ncost for i in self.
+            input_representations)
         hidden_pretraining_cost = self.hidden.ncost
         input_pretraining_gradients = T.grad(input_pretraining_cost,
                 input_pretraining_params)
-        hidden_pretraining_gradients = T.grad(hidden_pretraining_cost, hidden_pretraining_params)
+        hidden_pretraining_gradients = T.grad(
+            hidden_pretraining_cost, hidden_pretraining_params)
         pretraining_updates = \
                 dict((p, p - self.lr * g) for p, g in \
                 zip(input_pretraining_params, input_pretraining_gradients) \
@@ -427,8 +447,10 @@ class ConvolutionalMLP(module.FancyModule):
                         [self.output.w, self.output.b]
         finetuning_cost = self.output.cost
         finetuning_gradients = T.grad(finetuning_cost, finetuning_params)
-        finetuning_updates = dict((p, p - self.lr * g) for p, g in zip(finetuning_params, finetuning_gradients))
-        self.finetuning_update = module.Method(self.inputs + [self.targ], self.output.cost, finetuning_updates)
+        finetuning_updates = dict((p, p - self.lr * g) for p,
+             g in zip(finetuning_params, finetuning_gradients))
+        self.finetuning_update = module.Method(self.inputs + [self.
+            targ], self.output.cost, finetuning_updates)
 
         #self.validate = module.Method(self.inputs + [self.targ], [self.output.cost, self.output.argmax, self.output.max_pr])
         #self.softmax_output = module.Method(self.inputs, self.output.softmax_unsupervised)
@@ -446,8 +468,10 @@ class ConvolutionalMLP(module.FancyModule):
 #        for layer in obj.layers:
 #            if layer.lr is None:
 #                layer.lr = lr
-        assert self.input_representations[-1] is not self.input_representations[0]
-        assert self.input_representations[-1].w1 is self.input_representations[0].w1
+        assert self.input_representations[-1] \
+            is not self.input_representations[0]
+        assert self.input_representations[-1].w1 is\
+                self.input_representations[0].w1
 
         for i in self.input_representations:
 #            i.initialize(input_size=self.input_size, hidden_size=self.input_representation_size, seed=R.random_integers(2**30), noise_level=noise_level, qfilter_relscale=qfilter_relscale)
@@ -464,13 +488,16 @@ class ConvolutionalMLP(module.FancyModule):
             assert (i.w2 == self.input_representations[0].w2).all()
             assert (i.b1 == self.input_representations[0].b1).all()
             assert (i.b2 == self.input_representations[0].b2).all()
-            assert N.all((a==b).all() for a, b in zip(i.qfilters, self.input_representations[0].qfilters))
+            assert N.all((a == b).all() for a, b in zip(i.
+                qfilters, self.input_representations[0].qfilters))
 
         self.hidden.initialize(input_size=(len(self.inputs) * self.input_representation_size),
                 hidden_size=self.hidden_representation_size, noise_level=noise_level,
                 seed=int(R.random_integers(2**30)), lr=lr, qfilter_relscale=qfilter_relscale)
 
-        self.output.initialize(n_in=self.hidden_representation_size, n_out=self.output_size, lr=lr, seed=R.random_integers(2**30))
+        self.output.initialize(n_in=self.
+            hidden_representation_size, n_out=self.output_size, lr=lr, seed=R.random_integers(2**30))
+
 
 def create(window_size=3,
         input_dimension=9,
@@ -487,22 +514,24 @@ def create(window_size=3,
     activation_function = T.tanh
 
     architecture = ConvolutionalMLP( \
-                window_size = window_size,
-                n_quadratic_filters = n_quadratic_filters,
-                activation_function = activation_function,
-                reconstruction_cost_function = quadratic,
-                tie_weights = False
+                window_size=window_size,
+                n_quadratic_filters=n_quadratic_filters,
+                activation_function=activation_function,
+                reconstruction_cost_function=quadratic,
+                tie_weights=False
             )
 
     backup = config.warn.sum_div_dimshuffle_bug
     config.warn.sum_div_dimshuffle_bug = False
     try:
-        model = architecture.make(input_size=input_dimension, input_representation_size=token_representation_size, hidden_representation_size=concatenated_representation_size, output_size=output_vocabsize, lr=lr, seed=seed, noise_level=noise_level, qfilter_relscale=qfilter_relscale, mode=compile_mode)
+        model = architecture.make(input_size=input_dimension,
+             input_representation_size=token_representation_size, hidden_representation_size=concatenated_representation_size, output_size=output_vocabsize, lr=lr, seed=seed, noise_level=noise_level, qfilter_relscale=qfilter_relscale, mode=compile_mode)
     finally:
         config.warn.sum_div_dimshuffle_bug = backup
     return model
 
-def create_realistic(window_size=3,#7,
+
+def create_realistic(window_size=3,  # 7,
         input_dimension=200,
         output_vocabsize=23,
         n_quadratic_filters=2,
@@ -517,15 +546,17 @@ def create_realistic(window_size=3,#7,
     activation_function = T.tanh
 
     architecture = ConvolutionalMLP( \
-                window_size = window_size,
-                n_quadratic_filters = n_quadratic_filters,
-                activation_function = activation_function,
-                reconstruction_cost_function = quadratic,
-                tie_weights = False
+                window_size=window_size,
+                n_quadratic_filters=n_quadratic_filters,
+                activation_function=activation_function,
+                reconstruction_cost_function=quadratic,
+                tie_weights=False
             )
-    model = architecture.make(input_size=input_dimension, input_representation_size=token_representation_size, hidden_representation_size=concatenated_representation_size, output_size=output_vocabsize, lr=lr, seed=seed, noise_level=noise_level, qfilter_relscale=qfilter_relscale, mode=compile_mode)
+    model = architecture.make(input_size=input_dimension,
+         input_representation_size=token_representation_size, hidden_representation_size=concatenated_representation_size, output_size=output_vocabsize, lr=lr, seed=seed, noise_level=noise_level, qfilter_relscale=qfilter_relscale, mode=compile_mode)
     return model
 
+
 def test_naacl_model(iters_per_unsup=3, iters_per_sup=3,
         optimizer=None, realistic=False):
     #print "BUILDING MODEL"
@@ -534,11 +565,12 @@ def test_naacl_model(iters_per_unsup=3, iters_per_sup=3,
 
     if optimizer:
         mode = theano.Mode(linker='c|py', optimizer=optimizer)
-    else: mode = get_default_mode()
+    else:
+        mode = get_default_mode()
 
     if mode.__class__.__name__ == 'DebugMode':
-        iters_per_unsup=1
-        iters_per_sup =1
+        iters_per_unsup = 1
+        iters_per_sup = 1
 
     if realistic:
         m = create_realistic(compile_mode=mode)
@@ -551,7 +583,8 @@ def test_naacl_model(iters_per_unsup=3, iters_per_sup=3,
     for i, node in enumerate(m.pretraining_update.maker.fgraph.toposort()):
         idx_of_node[node] = i
         if False and i > -1:
-            print '   ', i, node, [(ii, idx_of_node.get(ii.owner, 'IN')) for ii in node.inputs]
+            print '   ', i, node, [(ii, idx_of_node.get(ii.
+                owner, 'IN')) for ii in node.inputs]
         prog_str.append(str(node))
     #print input_pretraining_gradients[4].owner.inputs
     #print input_pretraining_gradients[4].owner.inputs[1].owner.inputs
@@ -561,20 +594,30 @@ def test_naacl_model(iters_per_unsup=3, iters_per_sup=3,
 
     rng = N.random.RandomState(unittest_tools.fetch_seed(23904))
 
-    inputs = [rng.rand(10,m.input_size) for i in 1,2,3]
-    targets = N.asarray([0,3,4,2,3,4,4,2,1,0])
+    inputs = [rng.rand(10, m.input_size) for i in 1, 2, 3]
+    targets = N.asarray([0, 3, 4, 2, 3, 4, 4, 2, 1, 0])
     #print inputs
 
     #print 'UNSUPERVISED PHASE'
     t = time.time()
     for i in xrange(3):
         for j in xrange(iters_per_unsup):
-            m.pretraining_update(*inputs)
+            try:
+                known_fail = False
+                m.pretraining_update(*inputs)
+            except ValueError:
+                known_fail = True
+            except TypeError:
+                known_fail = True
+            if known_fail:
+                raise KnownFailureTest("Deprecated compile.module fails to "
+                    "give a sensible warning when updates to a variable "
+                    "have the wrong type")
         s0, s1 = [str(j) for j in m.pretraining_update(*inputs)]
         #print 'huh?', i, iters_per_unsup, iters_per_unsup * (i+1), s0, s1
     if iters_per_unsup == 3:
-        assert s0.startswith('0.927793')#'0.403044')
-        assert s1.startswith('0.068035')#'0.074898')
+        assert s0.startswith('0.927793')  # '0.403044')
+        assert s1.startswith('0.068035')  # '0.074898')
     #print 'UNSUPERVISED took %.3fs'%(time.time() - t)
 
     #print 'FINETUNING GRAPH'
@@ -590,6 +633,7 @@ def test_naacl_model(iters_per_unsup=3, iters_per_sup=3,
         assert 19.7042 < s0f and s0f < 19.7043
     #print 'SUPERVISED took %.3fs'%( time.time() - t)
 
+
 def jtest_main():
     from theano import gof
     JTEST = theano.compile.mode.optdb.query(*sys.argv[2:])
@@ -598,13 +642,17 @@ def jtest_main():
     optimizer = eval(sys.argv[1])
     test_naacl_model(optimizer, 10, 10, realistic=False)
 
+
 def real_main():
     test_naacl_model()
 
+
 def profile_main():
     # This is the main function for profiling
     # We've renamed our original main() above to real_main()
-    import cProfile, pstats, StringIO
+    import cProfile
+    import pstats
+    import StringIO
     prof = cProfile.Profile()
     prof = prof.runctx("real_main()", globals(), locals())
     stream = StringIO.StringIO()

theano/tests/test_gradient.py

@@ -11,14 +11,13 @@ from theano import gradient
 from theano.tensor.nnet.Conv3D import conv3D
 from theano import config
 import numpy as np
+from theano.gradient import DisconnectedType
+from theano.gof.null_type import NullType
 
 one = theano.tensor.as_tensor_variable(1.)
 
-def _grad_sources_inputs(*args):
-    # warn_type was introduced after this code, it complains throughout for nothing.
-    return grad_sources_inputs(warn_type=False, *args)
 
-class test_grad_sources_inputs(unittest.TestCase):
+class testgrad_sources_inputs(unittest.TestCase):
 
     def test_retNone1(self):
         """Test that it is not ok to return None from op.grad()"""
@@ -27,33 +26,35 @@ class test_grad_sources_inputs(unittest.TestCase):
                 inputs = [theano.tensor.vector()]
                 outputs = [theano.tensor.vector()]
                 return gof.Apply(self, inputs, outputs)
+
             def grad(self, inp, grads):
                 x, = inp
                 gz, = grads
                 pass
         a = retNone().make_node()
         try:
-            _grad_sources_inputs([(a.out, one)], None)
+            grad_sources_inputs([(a.out, one)], None)
         except TypeError, e:
             return
         self.fail()
 
     def test_wrong_rval_len1(self):
         """Test that it is not ok to return the wrong number of gradient terms"""
-        class retNone(gof.op.Op):
+        class retOne(gof.op.Op):
             def make_node(self, *inputs):
                 outputs = [theano.tensor.vector()]
                 return gof.Apply(self, inputs, outputs)
+
             def grad(self, inputs, grads):
-                return [None]
+                return [inputs[0].zeros_like()]
 
         i = theano.tensor.vector()
         j = theano.tensor.vector()
-        a1 = retNone().make_node(i)
-        g = _grad_sources_inputs([(a1.out, one)], None)
-        a2 = retNone().make_node(i,j)
+        a1 = retOne().make_node(i)
+        g = grad_sources_inputs([(a1.out, one)], None)
+        a2 = retOne().make_node(i, j)
         try:
-            g = _grad_sources_inputs([(a2.out, one)], None)
+            g = grad_sources_inputs([(a2.out, one)], None)
         except ValueError, e:
             return
         self.fail()
@@ -61,48 +62,54 @@ class test_grad_sources_inputs(unittest.TestCase):
     def test_1in_1out(self):
         """Test grad is called correctly for a 1-to-1 op"""
         gval = theano.tensor.matrix()
+
         class O(gof.op.Op):
             def make_node(self):
                 inputs = [theano.tensor.matrix()]
                 outputs = [theano.tensor.matrix()]
                 return gof.Apply(self, inputs, outputs)
+
             def grad(self, inp, grads):
                 return gval,
         a1 = O().make_node()
-        g = _grad_sources_inputs([(a1.outputs[0], one)], None)
+        g = grad_sources_inputs([(a1.outputs[0], one)], None)
         self.assertTrue(g[a1.inputs[0]] is gval)
 
     def test_1in_Nout(self):
         """Test grad is called correctly for a 1-to-many op"""
         gval = theano.tensor.matrix()
+
         class O(gof.op.Op):
             def make_node(self):
                 inputs = [theano.tensor.matrix()]
-                outputs = [theano.tensor.scalar(),theano.tensor.scalar()]
+                outputs = [theano.tensor.scalar(), theano.tensor.scalar()]
                 return gof.Apply(self, inputs, outputs)
+
             def grad(self, inp, grads):
                 x, = inp
                 gz1, gz2 = grads
                 return gval,
         a1 = O().make_node()
-        g = _grad_sources_inputs([(a1.outputs[0], one)], None)
+        g = grad_sources_inputs([(a1.outputs[0], one)], None)
         self.assertTrue(g[a1.inputs[0]] is gval)
 
     def test_Nin_1out(self):
         """Test grad is called correctly for a many-to-1 op"""
         gval0 = theano.tensor.scalar()
         gval1 = theano.tensor.scalar()
+
         class O(gof.op.Op):
             def make_node(self):
                 inputs = [theano.tensor.scalar(), theano.tensor.scalar()]
                 outputs = [theano.tensor.matrix()]
                 return gof.Apply(self, inputs, outputs)
+
             def grad(self, inp, grads):
                 x0, x1 = inp
                 gz, = grads
                 return (gval0, gval1)
         a1 = O().make_node()
-        g = _grad_sources_inputs([(a1.outputs[0], one)], None)
+        g = grad_sources_inputs([(a1.outputs[0], one)], None)
         self.assertTrue(g[a1.inputs[0]] is gval0)
         self.assertTrue(g[a1.inputs[1]] is gval1)
 
@@ -110,15 +117,17 @@ class test_grad_sources_inputs(unittest.TestCase):
         """Test grad is called correctly for a many-to-many op"""
         gval0 = theano.tensor.matrix()
         gval1 = theano.tensor.matrix()
+
         class O(gof.op.Op):
             def make_node(self):
-                inputs = [theano.tensor.matrix(),theano.tensor.matrix()]
-                outputs = [theano.tensor.matrix(),theano.tensor.matrix()]
+                inputs = [theano.tensor.matrix(), theano.tensor.matrix()]
+                outputs = [theano.tensor.matrix(), theano.tensor.matrix()]
                 return gof.Apply(self, inputs, outputs)
+
             def grad(self, inp, grads):
                 return gval0, gval1
         a1 = O().make_node()
-        g = _grad_sources_inputs([(a1.outputs[0], one)], None)
+        g = grad_sources_inputs([(a1.outputs[0], one)], None)
         self.assertTrue(g[a1.inputs[0]] is gval0)
         self.assertTrue(g[a1.inputs[1]] is gval1)
 
@@ -127,36 +136,41 @@ class test_grad_sources_inputs(unittest.TestCase):
         class O(gof.op.Op):
             def __init__(self, tst):
                 self.tst = tst
+
             def make_node(self, *inputs):
-                outputs = [theano.tensor.matrix(),theano.tensor.matrix()]
+                outputs = [theano.tensor.matrix(), theano.tensor.matrix()]
                 return gof.Apply(self, inputs, outputs)
+
             def grad(self, inputs, g_out):
                 return [one]
         i = theano.tensor.matrix()
         a1 = O(self).make_node(i)
-        g = grad_sources_inputs([(a1.outputs[0], one)], None, warn_type=False)
+        g = grad_sources_inputs([(a1.outputs[0], one)], None)
         self.assertTrue(g[i] is one)
 
+
 def test_unimplemented_grad_func():
     # tests that function compilation catches unimplemented grads in the graph
     a = theano.tensor.vector()
     b = theano.gradient.grad_not_implemented(theano.tensor.add, 0, a)
     try:
-        f = theano.function([a], b, on_unused_input = 'ignore')
+        f = theano.function([a], b, on_unused_input='ignore')
         assert 0
     except TypeError:
         pass
 
+
 def test_undefined_grad_func():
     #tests that function compilation catches undefined grads in the graph
     a = theano.tensor.vector()
     b = theano.gradient.grad_undefined(theano.tensor.add, 0, a)
     try:
-        f = theano.function([a],b, on_unused_input = 'ignore')
+        f = theano.function([a], b, on_unused_input='ignore')
         assert 0
     except TypeError:
         pass
 
+
 def test_unimplemented_grad_grad():
     #tests that unimplemented grads are caught in the grad method
 
@@ -165,134 +179,251 @@ def test_unimplemented_grad_grad():
             return gof.Apply(self, [x], [x.type()])
 
         def grad(self, inputs, output_grads):
-            return [ theano.gradient.grad_not_implemented(self, 0, inputs[0]) ]
+            return [theano.gradient.grad_not_implemented(self, 0, inputs[0])]
 
     a = theano.tensor.scalar()
     b = DummyOp()(a)
 
     try:
-        g = theano.gradient.grad(b,a)
+        g = theano.gradient.grad(b, a)
         assert False
     except TypeError:
         pass
 
+
 def test_undefined_grad_grad():
     #tests that undefined grads are caught in the grad method
 
     V = theano.tensor.TensorType(dtype=config.floatX,
-            broadcastable = (False,False,False,False,False))()
+            broadcastable=(False, False, False, False, False))()
     W = theano.tensor.TensorType(dtype=config.floatX,
-            broadcastable = (False, False, False, False, False))()
+            broadcastable=(False, False, False, False, False))()
     b = theano.tensor.vector()
     d = theano.tensor.ivector()
 
-    Z = conv3D(V,W,b,d)
+    Z = conv3D(V, W, b, d)
 
     try:
-        g = theano.gradient.grad(Z.sum(),d)
+        g = theano.gradient.grad(Z.sum(), d)
         assert False
     except TypeError:
         pass
 
+
 def test_grad_name():
     A = theano.tensor.matrix('A')
     x = theano.tensor.vector('x')
-    f = theano.tensor.dot(x,theano.tensor.dot(A,x))
+    f = theano.tensor.dot(x, theano.tensor.dot(A, x))
     f.name = 'f'
-    g = theano.tensor.grad(f,x)
+    g = theano.tensor.grad(f, x)
     assert g.name == '(df/dx)'
 
 
-
 def test_grad_duplicate_input():
 
     #test that the grad works when a variable
     #appears in more than one place in a node's input list
 
     def output(x):
-        return (x*x)
+        return (x * x)
 
-    rng = np.random.RandomState([2012,8,28])
+    rng = np.random.RandomState([2012, 8, 28])
 
     vx = rng.randn(2)
 
-    theano.tests.unittest_tools.verify_grad(output,[vx])
+    theano.tests.unittest_tools.verify_grad(output, [vx])
+
 
 def test_grad_quadratic():
 
     #test the gradient on a tiny graph
 
-    def cost(x,A):
-        return theano.tensor.dot(x,theano.tensor.dot(A,x))
+    def cost(x, A):
+        return theano.tensor.dot(x, theano.tensor.dot(A, x))
 
-    rng = np.random.RandomState([2012,8,28])
+    rng = np.random.RandomState([2012, 8, 28])
 
     vx = rng.randn(2)
-    vA = rng.randn(2,2)
+    vA = rng.randn(2, 2)
 
-    theano.tests.unittest_tools.verify_grad(cost,[vx,vA])
+    theano.tests.unittest_tools.verify_grad(cost, [vx, vA])
 
 
 def test_grad_quadratic_vector():
 
     #test the gradient on a small graph
 
-    def output(x,A):
-        return theano.tensor.dot(x*x,A)
+    def output(x, A):
+        return theano.tensor.dot(x * x, A)
 
-    rng = np.random.RandomState([2012,8,28])
+    rng = np.random.RandomState([2012, 8, 28])
 
     vx = rng.randn(2)
-    vA = rng.randn(2,2)
+    vA = rng.randn(2, 2)
 
-    theano.tests.unittest_tools.verify_grad(output,[vx,vA])
+    theano.tests.unittest_tools.verify_grad(output, [vx, vA])
 
 
 def test_grad_cubic():
 
     #test the gradient on a bigger graph
 
-    def cost(x,A):
-        return theano.tensor.dot(x*x,theano.tensor.dot(A,x))
+    def cost(x, A):
+        return theano.tensor.dot(x * x, theano.tensor.dot(A, x))
 
-    rng = np.random.RandomState([2012,8,28])
+    rng = np.random.RandomState([2012, 8, 28])
 
     vx = rng.randn(2)
-    vA = rng.randn(2,2)
+    vA = rng.randn(2, 2)
+
+    theano.tests.unittest_tools.verify_grad(cost, [vx, vA])
 
-    theano.tests.unittest_tools.verify_grad(cost,[vx,vA])
 
 def test_grad_grad_quadratic():
 
     #test the gradient on a graph constructed using the gradient
 
-    def output(x,A):
-        orig_cost = theano.tensor.dot(x,theano.tensor.dot(A,x))
+    def output(x, A):
+        orig_cost = theano.tensor.dot(x, theano.tensor.dot(A, x))
         return theano.gradient.grad(orig_cost, x)
 
-    rng = np.random.RandomState([2012,8,28])
+    rng = np.random.RandomState([2012, 8, 28])
 
     vx = rng.randn(2)
-    vA = rng.randn(2,2)
+    vA = rng.randn(2, 2)
+
+    theano.tests.unittest_tools.verify_grad(output, [vx, vA])
 
-    theano.tests.unittest_tools.verify_grad(output,[vx,vA])
 
 def test_grad_grad_cubic():
 
     #test the gradient on a bigger graph constructed using the gradient
 
-    def output(x,A):
-        orig_cost = theano.tensor.dot(x*x,theano.tensor.dot(A,x))
+    def output(x, A):
+        orig_cost = theano.tensor.dot(x * x, theano.tensor.dot(A, x))
         return theano.gradient.grad(orig_cost, x)
 
-    rng = np.random.RandomState([2012,8,28])
+    rng = np.random.RandomState([2012, 8, 28])
 
     vx = rng.randn(2)
-    vA = rng.randn(2,2)
+    vA = rng.randn(2, 2)
+
+    theano.tests.unittest_tools.verify_grad(output, [vx, vA])
+
+
+def test_grad_int():
+
+    # tests that the gradient with respect to an integer
+    # is the same as the gradient with respect to a float
+
+    W = theano.tensor.matrix()
+    b = theano.tensor.vector()
+
+    def make_grad_func(X):
+        Z = theano.tensor.dot(X, W) + b
+        H = theano.tensor.nnet.sigmoid(Z)
+        cost = H.sum()
+        g = gradient.grad(cost, X)
+        return theano.function([X, W, b], g, on_unused_input='ignore')
+
+    int_func = make_grad_func(theano.tensor.imatrix())
+    #we have to use float64 as the float type to get the results to match
+    #using an integer for the input makes all the later functions use float64
+    float_func = make_grad_func(theano.tensor.matrix(dtype='float64'))
+
+    m = 5
+    d = 3
+    n = 4
+    rng = np.random.RandomState([2012, 9, 5])
+
+    int_type = theano.tensor.imatrix().dtype
+    float_type = 'float64'
+
+    X = np.cast[int_type](rng.randn(m, d) * 127.)
+    W = np.cast[W.dtype](rng.randn(d, n))
+    b = np.cast[b.dtype](rng.randn(n))
+
+    int_result = int_func(X, W, b)
+    float_result = float_func(np.cast[float_type](X), W, b)
+
+    assert np.allclose(int_result, float_result)
+
+
+def test_grad_disconnected():
+
+    #tests corner cases of gradient for shape and alloc
+
+    x = theano.tensor.vector(name='x')
+    total = x.sum()
+    total.name = 'total'
+    num_elements = x.shape[0]
+    num_elements.name = 'num_elements'
+    silly_vector = theano.tensor.alloc(total / num_elements, num_elements)
+    silly_vector.name = 'silly_vector'
+    cost = silly_vector.sum()
+    cost.name = 'cost'
+    #note that cost simplifies to be the same as "total"
+    g = gradient.grad(cost, x, add_names=False)
+    #we still need to pass in x because it determines the shape of the output
+    f = theano.function([x], g)
+    rng = np.random.RandomState([2012, 9, 5])
+    x = np.cast[x.dtype](rng.randn(3))
+    g = f(x)
+    assert np.allclose(g, np.ones(x.shape, dtype=x.dtype))
+
+
+def test_disconnected_nan():
+
+    # test that connection_pattern can prevent getting NaN
+
+    # Op1 has two outputs, f and g
+    # x is connected to f but not to g
+    class Op1(theano.gof.Op):
+        def make_node(self, x):
+            return theano.Apply(self, inputs=[x],
+                    outputs=[x.type(), theano.tensor.scalar()])
+
+        def connection_pattern(self, node):
+            return [[True, False]]
+
+        def grad(self, inputs, output_grads):
+            return [inputs[0].zeros_like()]
+
+    # Op2 has two inputs, f and g
+    # Its gradient with respect to g is not defined
+    class Op2(theano.gof.Op):
+        def make_node(self, f, g):
+            return theano.Apply(self, inputs=[f, g],
+                    outputs=[theano.tensor.scalar()])
+
+        def grad(self, inputs, output_grads):
+            return [inputs[0].zeros_like(), NullType()()]
+
+    x = theano.tensor.vector()
+    f, g = Op1()(x)
+    cost = Op2()(f, g)
+
+    # cost is differentiable wrt x
+    # but we can't tell that without using Op1's connection pattern
+    # looking at the theano graph alone, g is an ancestor of cost
+    # and has x as an ancestor, so we must compute its gradient
+
+    g = gradient.grad(cost, x)
+
+    # If we made it to here without an exception, then the
+    # connection_pattern functionality worked correctly
 
-    theano.tests.unittest_tools.verify_grad(output,[vx,vA])
 
+def test_sum_disconnected():
 
+    # Tests that we can add DisconnectedType to other terms correctly
+    x = theano.tensor.scalar()
+    y = x * 2.
+    z = x + 1.
+    cost = y + z
+    theano.tensor.grad(cost, x, consider_constant=[y, z])
+    # In an earlier version of theano, the above line would have failed
+    # while trying to add two DisconnectedTypes
 
 if __name__ == '__main__':
     unittest.main()

theano/tests/test_rop.py

@@ -19,6 +19,8 @@ import theano
 from theano import tensor
 import numpy
 from theano.gof import Op, Apply
+from theano.gradient import grad_undefined
+from numpy.testing.noseclasses import KnownFailureTest
 
 '''
 Special Op created to test what happens when you have one op that is not
@@ -45,7 +47,7 @@ class BreakRop(Op):
         out[0] = x
 
     def grad(self, inp, grads):
-        return [None]
+        return [grad_undefined(self, 0, inp[0])]
 
     def R_op(self, inputs, eval_points):
         return [None]
@@ -71,7 +73,7 @@ class RopLop_checker(unittest.TestCase):
                              5 + self.rng.randint(30))
 
     def check_nondiff_rop(self, y):
-        """ If you op is not differentiable(so you can't define Rop)
+        """ If your op is not differentiable(so you can't define Rop)
         test that an error is raised."""
         raised = False
         try:
@@ -80,7 +82,7 @@ class RopLop_checker(unittest.TestCase):
             raised = True
         if not raised:
             self.fail((
-                'Op did not raised an error even though the function'
+                'Op did not raise an error even though the function'
                 ' is not differentiable'))
 
     def check_mat_rop_lop(self, y, out_shape):
@@ -136,7 +138,7 @@ class RopLop_checker(unittest.TestCase):
 
     def check_rop_lop(self, y, out_shape):
         """
-        As check_mat_rop_lop, except the input is self.x witch is a
+        As check_mat_rop_lop, except the input is self.x which is a
         vector. The output is still a vector.
 
         """
@@ -158,8 +160,12 @@ class RopLop_checker(unittest.TestCase):
         v1 = rop_f(vx, vv)
         v2 = scan_f(vx, vv)
         assert numpy.allclose(v1, v2), ('ROP mismatch: %s %s' % (v1, v2))
-        self.check_nondiff_rop(theano.clone(y,
+        known_fail = False
+        try:
+            self.check_nondiff_rop(theano.clone(y,
                                 replace={self.x: break_op(self.x)}))
+        except AssertionError:
+            known_fail = True
 
         # TEST LOP
 
@@ -181,6 +187,11 @@ class RopLop_checker(unittest.TestCase):
         v2 = scan_f(vx, vv)
         assert numpy.allclose(v1, v2), ('LOP mismatch: %s %s' % (v1, v2))
 
+        if known_fail:
+            raise KnownFailureTest("Rop doesn't handle non-differentiable "
+                    "inputs correctly. Bug exposed by fixing Add.grad"
+                    " method.")
+
 
 class test_RopLop(RopLop_checker):
     def test_shape(self):
@@ -319,21 +330,21 @@ class test_RopLop(RopLop_checker):
         m_ = tensor.matrix('m_')
         v_ = tensor.vector('v_')
 
-        mval = self.rng.uniform(size=(3,7)).astype(theano.config.floatX)
+        mval = self.rng.uniform(size=(3, 7)).astype(theano.config.floatX)
         vval = self.rng.uniform(size=(7,)).astype(theano.config.floatX)
-        m_val = self.rng.uniform(size=(3,7)).astype(theano.config.floatX)
+        m_val = self.rng.uniform(size=(3, 7)).astype(theano.config.floatX)
         v_val = self.rng.uniform(size=(7,)).astype(theano.config.floatX)
 
-        rop_out1 = tensor.Rop([m, v, m+v], [m, v], [m_, v_])
+        rop_out1 = tensor.Rop([m, v, m + v], [m, v], [m_, v_])
         assert isinstance(rop_out1, list)
         assert len(rop_out1) == 3
-        rop_out2 = tensor.Rop((m, v, m+v), [m, v], [m_, v_])
+        rop_out2 = tensor.Rop((m, v, m + v), [m, v], [m_, v_])
         assert isinstance(rop_out2, tuple)
         assert len(rop_out2) == 3
-        lop_out1 = tensor.Lop([m, v, m+v], (m, v), [m_, v_])
+        lop_out1 = tensor.Lop([m, v, m + v], (m, v), [m_, v_])
         assert isinstance(lop_out1, tuple)
         assert len(lop_out1) == 2
-        lop_out2 = tensor.Lop((m, v, m+v), [m, v], [m_, v_])
+        lop_out2 = tensor.Lop((m, v, m + v), [m, v], [m_, v_])
         assert isinstance(lop_out2, list)
         assert len(lop_out2) == 2