Merge pull request #1489 from nouiz/doc_sparse

doc/install.txt

@@ -1006,6 +1006,14 @@ Theano is to download and execute this `Windows installer
 for Theano on AnacondaCE for Windows
 <https://github.com/Theano/Theano-wininstaller/raw/master/bin/theano_installer_latest.msi>`__.
 
+
+.. note::
+
+    It is possible that you need to logout/login or restart the
+    computer after installing AnacondaCE and before running Theano
+    installer. Otherwise, sometimes the Theano installer while trying
+    to find pip.
+
 .. note::
 
     This installer was tested on Windows 7, 64-bit edition, and AnacondaCE

doc/install_ubuntu.txt

@@ -53,6 +53,22 @@ Ubuntu 11.04:
  1) ``sudo apt-get install python-numpy python-scipy python-dev python-pip python-nose g++ git libatlas3gf-base libatlas-dev``
  2) ``sudo pip install Theano``
 
+.. note::
+
+    If you have error that contain "gfortran" in it, like this one:
+
+        ImportError: ('/home/Nick/.theano/compiledir_Linux-2.6.35-31-generic-x86_64-with-Ubuntu-10.10-maverick--2.6.6/tmpIhWJaI/0c99c52c82f7ddc775109a06ca04b360.so: undefined symbol: _gfortran_st_write_done'
+
+    The problem is probably that NumPy is linked with a different blas
+    then then one currently available (probably ATLAS). There is 2
+    possible fixes:
+
+    1) Uninstall ATLAS and install OpenBLAS.
+    2) Use the Theano flag "blas.ldflags=-lblas -lgfortran"
+
+    1) is better as OpenBLAS is faster then ATLAS and NumPy is
+    probably already linked with it. So you won't need any other
+    change in Theano files or Theano configuration.
 
 
 Test the newly installed packages

doc/library/sparse/index.txt

@@ -127,8 +127,10 @@ List of Implemented Operations
 - Construction of Sparses and their Properties
     - :class:`CSM <theano.sparse.basic.CSM>` and ``CSC``, ``CSR`` to construct a matrix.
       The grad implemented is regular.
-    - :class:`CSMProperties <theano.sparse.basic.CSMProperties>` to get the properties of a sparse matrix.
+    - :class:`CSMProperties <theano.sparse.basic.CSMProperties>` and ``csm_properties(x)``
+      to get the properties of a sparse matrix.
       The grad implemented is regular.
+    - csm_indices(x), csm_indptr(x), csm_data(x) and csm_shape(x) or x.shape.
     - :func:`sp_ones_like <theano.sparse.basic.sp_ones_like>`.
       The grad implemented is regular.
     - :func:`sp_zeros_like <theano.sparse.basic.sp_zeros_like>`.
@@ -259,6 +261,9 @@ List of Implemented Operations
     - :class:`Remove0 <theano.sparse.basic.Remove0>` and ``remove0``
     - :func:`clean <theano.sparse.basic.clean>` to resort indices and remove zeros
 
+- To help testing
+    - :func:`theano.sparse.tests.test_basic.sparse_random_inputs`
+
 ===================================================================
 :mod:`sparse` --  Sparse Op
 ===================================================================
@@ -271,3 +276,4 @@ List of Implemented Operations
 .. automodule:: theano.sparse.basic
     :members:
 
+.. autofunction:: theano.sparse.tests.test_basic.sparse_random_inputs

doc/tutorial/extending_theano.txt

@@ -401,7 +401,7 @@ efficiency over the basic solution that is asked here, the two operations would
 have to be jointly optimized explicitly in the code.)
 
 SciPy
------
+=====
 
 We can wrap SciPy functions in Theano. But SciPy is an optional dependency.
 Here is some code that allows the Op to be optional:
@@ -431,7 +431,7 @@ Here is some code that allows the Op to be optional:
             ...
 
 Random numbers in tests
------------------------
+=======================
 
 Making tests errors more reproducible is a good practice. To make your
 tests more reproducible, you need a way to get the same random
@@ -447,6 +447,89 @@ For more details see :ref:`random_value_in_tests`.
 
 :download:`Solution<extending_theano_solution_1.py>`
 
+Sparse
+======
+
+There is few differences if you want to make an op that use
+:ref:`sparse <tutsparse>` inputs or outputs. In particular, in the
+``make_node()`` function, you call
+``theano.sparse.as_sparse_variable(x)`` on sparse input variable
+instead of ``as_tensor_variable(x)``.
+
+Another difference is that you need to use SparseVariable and
+SparseType instead of TensorVariable and TensorType.
+
+Don't forget that we support only sparse matrix (so only 2 dimensions)
+and they don't support broadcast operation by default as scipy sparse
+matrix (but a few op do it when called manually). Also, we support 2
+formats for sparse type: ``csr`` and ``csr``. So in ``make_mode()``,
+you create outputs variables like this:
+
+.. code-block:: python
+
+    out_format = inputs[0].format  # or 'csr' or 'csc' if the output format is fixed
+    SparseType(dtype=inputs[0].dtype, format=out_format).make_variable()
+
+See the sparse :class:`theano.sparse.basic.Cast` op `code
+<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L753>`_
+for a good example for a sparse op with python code.
+
+.. note::
+
+   From the definition of CSR and CSC format, CSR column indices are
+   not necessarily sorted. Likewise for CSC row indices. Use
+   :class:`EnsureSortedIndices
+   <theano.sparse.basic.EnsureSortedIndices>` if your code don't
+   support it.
+
+   Also, there can be explicit zeros in your inputs. Use
+   :class:`Remove0 <theano.sparse.basic.Remove0>` or ``remove0`` to
+   make sure they aren't present in your input if you don't support
+   that.
+
+   To remove explicit zeros and make sure indices are sorted, use
+   :func:`clean <theano.sparse.basic.clean>`.
+
+Sparse Gradient
+---------------
+
+There is 2 types of :ref:`gradients <tutsparse_gradient>` : ``normal``
+gradient and ``structured`` gradient. Please document what your op
+implement in its docstring. It is important that the user know it and
+it is not always easy to infer from the code. Also make clear witch
+inputs/outputs are sparse and witch ones are dense.
+
+Sparse c code
+-------------
+
+Theano don't have a native c code interface for sparse matrix. The
+reason is simple, we use the scipy sparse matrix object and they don't
+have a c object. So we use a simple trick: a sparse matrix is made of
+4 fields that are vector: data, indices, indptr and shape. So to make
+an op with c code that have sparse variables as inputs, we make an op
+that take as input the needed fields of those sparse variables.
+
+You can extract the 4 fields with
+:func:`theano.sparse.basic.csm_properties`. You can use
+:func:`theano.sparse.basic.csm_data`,
+:func:`theano.sparse.basic.csm_indices`,
+:func:`theano.sparse.basic.csm_indptr` and
+:func:`theano.sparse.basic.csm_shape` to extract the individual
+fields.
+
+You can look at the `AddSD
+<https://github.com/Theano/Theano/blob/master/theano/sparse/basic.py#L1704>`_
+sparse op for an example with c code. It implement the addition of a
+sparse matrix with a dense matrix.
+
+Sparse Tests
+------------
+
+You can reuse the test system for tensor variable. To generate the
+needed sparse variable and data, you can use
+:func:`theano.sparse.tests.test_basic.sparse_random_inputs`. It take
+take many paramters including parameters for the format (csr or csc), the shape, the
+dtype, to have explicit 0 and to have unsorted indices.
 
 Final Note
 ==========

doc/tutorial/sparse.txt

@@ -174,6 +174,8 @@ provide a structured gradient. More explication below.
  [ 0.  0.  3.]
  [ 5.  0.  0.]]
 
+.. _tutsparse_gradient:
+
 Gradient
 --------
 

theano/compile/function.py

@@ -171,7 +171,8 @@ def function(inputs, outputs=None, mode=None, updates=None, givens=None,
             " got " + str(type(updates)) + ". Using "
             "a standard dictionary here results in "
             "non-deterministic behavior. You should use an OrderedDict"
-            " if you are using Python 2.7, or use a list of (shared, update)"
+            " if you are using Python 2.7 (theano.compat.python2x.OrderedDict"
+            " for older python), or use a list of (shared, update)"
             " pairs. Do not just convert your dictionary to this type before"
             " the call as the conversion will still be non-deterministic.",
             stacklevel=2)

theano/sparse/basic.py

@@ -511,23 +511,42 @@ class CSMProperties(gof.Op):
 
         data, indices, indptr, shape = csm_properties(csm)
         return [CSM(csm.format)(g[0], indices, indptr, shape)]
+
 # don't make this a function or it breaks some optimizations below
 csm_properties = CSMProperties()
+"""An CSMProperties object instance. It return the fields data,
+indices, indptr and shape of the sparse varible. Together they specify
+completly the the sparse variable when we know its format. Example::
+
+    the_data, the_indices, the_indptr, the_shape = csm_properties(a_sparse_var)
+"""
 
 
 def csm_data(csm):
+    """
+    return the data field of the sparse variable.
+    """
     return csm_properties(csm)[0]
 
 
 def csm_indices(csm):
+    """
+    return the indices field of the sparse variable.
+    """
     return csm_properties(csm)[1]
 
 
 def csm_indptr(csm):
+    """
+    return the indptr field of the sparse variable.
+    """
     return csm_properties(csm)[2]
 
 
 def csm_shape(csm):
+    """
+    return the shape field of the sparse variable.
+    """
     return csm_properties(csm)[3]
 
 
@@ -2338,8 +2357,7 @@ def vstack(blocks, format=None, dtype=None):
 
 
 class Remove0(gof.Op):
-    """Remove explicit zeros from a sparse matrix, and
-    resort indices.
+    """Remove explicit zeros from a sparse matrix.
 
     :param x: Sparse matrix.
 

theano/sparse/tests/test_basic.py

@@ -80,7 +80,8 @@ def random_lil(shape, dtype, nnz):
     return rval
 
 
-def sparse_random_inputs(format, shape, n=1, out_dtype=None, p=0.5, gap=None):
+def sparse_random_inputs(format, shape, n=1, out_dtype=None, p=0.5, gap=None,
+                         explicit_zero=False, unsorted_indices=False):
     """Return a tuple containing everything needed to
     perform a test.
 
@@ -97,9 +98,15 @@ def sparse_random_inputs(format, shape, n=1, out_dtype=None, p=0.5, gap=None):
                 max, when `gap` = (`a`, `b`) it provide a sample
                 from [a, b[. If `None` is used, it provide [0, 1]
                 for float dtypes and [0, 50[ for integer dtypes.
-
+    :param explicit_zero: When True, we add explicit zero in the
+                          returned sparse matrix
+    :param unsorted_indices: when True, we make sure there is
+                             unsorted indices in the returned
+                             sparse matrix.
     :return: (variable, data) where both `variable`
              and `data` are list.
+
+    :note: explicit_zero and unsorted_indices was added in Theano 0.6rc4
     """
 
     if out_dtype is None:
@@ -136,6 +143,19 @@ def sparse_random_inputs(format, shape, n=1, out_dtype=None, p=0.5, gap=None):
                 for k in range(n)]
     data = [getattr(scipy.sparse, format + '_matrix')(_rand(), dtype=out_dtype)
             for k in range(n)]
+    if unsorted_indices:
+        for idx in range(n):
+            d = data[idx]
+            d = d[range(d.shape[0])]
+            assert not d.has_sorted_indices
+            data[idx] = d
+    if explicit_zero:
+        for idx in range(n):
+            assert data[idx].nnz > 1, (
+                "can't make a sparse matrix with explicit 0")
+            d_idx = numpy.random.randint(data[idx].nnz)
+            data[idx].data[d_idx] = 0
+
     #numpy 1.5.0 with scipy 0.9.0 have scipy.sparse.XXX_matrix return
     #typenum 10(ulonglong) instead of 8(uint64) event if they are the same!
     #Theano don't like ulonglong type_num
@@ -1845,41 +1865,45 @@ class Remove0Tester(utt.InferShapeTester):
             ('csr', scipy.sparse.csr_matrix), ]
 
         for format, matrix_class in configs:
-            # real
-            origin = (numpy.arange(9) + 1).reshape((3, 3))
-            origin.astype(config.floatX)
-            mat = matrix_class(origin).astype(theano.config.floatX)
-
-            mat[0, 1] = mat[1, 0] = mat[2, 2] = 0
-
-            assert mat.size == 9
-
-            # symbolic
-            x = theano.sparse.SparseType(format=format, dtype=config.floatX)()
-            # the In thingy has to be there because theano has as rule not
-            # to optimize inputs
-            f = theano.function([theano.In(x, borrow=True, mutable=True)],
-                                Remove0()(x))
-
-            # assert optimization local_inplace_remove0 is applied in
-            # modes with optimization
-            if theano.config.mode not in ['FAST_COMPILE']:
-                # list of apply nodes in the optimized graph.
-                nodes = f.maker.fgraph.toposort()
-                # Check there isn't any Remove0 instance not inplace.
-                assert not any([isinstance(node.op, Remove0) and
-                                not node.op.inplace for node in nodes]), (
-                       'Inplace optimization should have been applied')
-                # Check there is at least one Remove0 inplace.
-                assert any([isinstance(node.op, Remove0) and node.op.inplace
-                            for node in nodes])
-            # checking
-            # makes sense to change its name
-            target = mat
-            result = f(mat)
-            mat.eliminate_zeros()
-            msg = 'Matrices sizes differ. Have zeros been removed ?'
-            assert result.size == target.size, msg
+            for zero, unsor in [(True, True), (True, False),
+                              (False, True), (False, False)]:
+                (x,), (mat,) = sparse_random_inputs(format, (6, 8),
+                                            out_dtype=config.floatX,
+                                            explicit_zero=zero,
+                                            unsorted_indices=unsor)
+                assert 0 in mat.data or not zero
+                assert not mat.has_sorted_indices or not unsor
+
+                # the In thingy has to be there because theano has as rule not
+                # to optimize inputs
+                f = theano.function([theano.In(x, borrow=True, mutable=True)],
+                                    Remove0()(x))
+
+                # assert optimization local_inplace_remove0 is applied in
+                # modes with optimization
+                if theano.config.mode not in ['FAST_COMPILE']:
+                    # list of apply nodes in the optimized graph.
+                    nodes = f.maker.fgraph.toposort()
+                    # Check there isn't any Remove0 instance not inplace.
+                    assert not any([isinstance(node.op, Remove0) and
+                                    not node.op.inplace for node in nodes]), (
+                           'Inplace optimization should have been applied')
+                    # Check there is at least one Remove0 inplace.
+                    assert any([isinstance(node.op, Remove0) and node.op.inplace
+                                for node in nodes])
+                # checking
+                # makes sense to change its name
+                target = mat
+                result = f(mat)
+                mat.eliminate_zeros()
+                msg = 'Matrices sizes differ. Have zeros been removed ?'
+                assert result.size == target.size, msg
+                if unsor:
+                    assert not result.has_sorted_indices
+                    assert not target.has_sorted_indices
+                else:
+                    assert result.has_sorted_indices
+                    assert target.has_sorted_indices
 
     def test_infer_shape(self):
         mat = (numpy.arange(12) + 1).reshape((4, 3))

theano/updates.py

@@ -32,7 +32,10 @@ class OrderedUpdates(OrderedDict):
             # Warn when using as input a non-ordered dictionary.
             warnings.warn('Initializing an `OrderedUpdates` from a '
                           'non-ordered dictionary with 2+ elements could '
-                          'make your code non-deterministic')
+                          'make your code non-deterministic. You can use '
+                          'an OrderedDict that is implemented at '
+                          'theano.compat.python2x.OrderedDict '
+                          'for python 2.4+.')
         super(OrderedUpdates, self).__init__(*key, **kwargs)
         for key in self:
             if not isinstance(key, SharedVariable):