advanced -> topics

doc/contents.txt

@@ -11,10 +11,9 @@ Contents
    LICENSE
    introduction
    install
-   numpy
    basic_tutorial/index
    advanced_tutorial/index
-   advanced/index
+   topics/index
    indexes/index
    glossary
    links

doc/introduction.txt

@@ -6,30 +6,32 @@ Introduction
 ============
 
 Theano is a Python library that allows you to define, optimize, and
-efficiently evaluate mathematical expressions involving multi-dimensional
-arrays. Using Theano, it is not uncommon to see speed improvements of
-ten-fold over using pure NumPy.
+efficiently evaluate mathematical expressions involving
+multi-dimensional arrays. Using Theano, for problems involving large
+amounts of data, it is possible to attain speeds that are only a few
+percentage points slower than hand-crafted C implementations.
 
 Theano melds some aspects of a computer algebra system (CAS) with
-aspects of an optimizing compiler. It can even transform some or
-all of the mathematical expression into C code and compile it into
-native machine instructions. This combination of CAS with optimizing
-compilation is particularly useful for computational fields in which
-complicated mathematical expressions are evaluated repeatedly and evaluation
-speed is critical.
+aspects of an optimizing compiler. It can even transform some or all
+of the mathematical expression into C code and compile it into native
+machine instructions. This combination of CAS with optimizing
+compilation is particularly useful for tasks in which complicated
+mathematical expressions are evaluated repeatedly and evaluation speed
+is critical.
 
 Theano supports a range of numerical types in multiple dimensions and
 a number of well-tested operations. It also allows you to compute the
-gradient of an expression with respect to another. Symbolic expressions
-may be compiled into functions, which work on the same data structures
-as numpy_, allowing for easy interoperability.
+gradient of an expression with respect to another. Symbolic
+expressions may be compiled into functions, which work on the same
+data structures as numpy_, allowing for easy interoperability.
 
-Theano's compiler applies many optimizations of varying complexity
-to these symbolic expressions. These optimizations include, but are
-not limited to:
+Theano's compiler applies many optimizations of varying complexity to
+these symbolic expressions. These optimizations include, but are not
+limited to:
 
 * constant folding
-* merging of similar subgraphs, to avoid calculating the same values more than once
+* merging of similar subgraphs, to avoid calculating the same values
+  more than once
 * arithmetic simplification (``x*y/x -> y``)
 * inserting efficient BLAS_ operations
 * using inplace operations wherever it is safe to do so.
@@ -37,20 +39,18 @@ not limited to:
 Theano defines several optimizations which improve the numerical
 stability of computations.
 
-Theano was written at the LISA_ lab to support the development
-of efficient machine learning algorithms while minimizing human time. We
-use it especially in gradient-based learning techniques.
-Theano is named after the `Greek mathematician`_, who may have
-been Pythagoras' wife.
-Theano is released under a BSD license (:ref:`link <license>`)
+Theano was written at the LISA_ lab to support the development of
+efficient machine learning algorithms while minimizing human time. We
+use it especially in gradient-based learning techniques.  Theano is
+named after the `Greek mathematician`_, who may have been Pythagoras'
+wife.  Theano is released under a BSD license (:ref:`link <license>`)
 
 
 Sneak peek
 ==========
 
-Here is an example of how to use Theano. It doesn't show
-off many of Theano's features, but it illustrates concretely what
-Theano is.
+Here is an example of how to use Theano. It doesn't show off many of
+Theano's features, but it illustrates concretely what Theano is.
 
 .. code-block:: python
 
@@ -73,8 +73,8 @@ Theano is.
 
 
 Theano is not a programming language in the normal sense because you
-write a program in Python that builds expressions for Theano. Still
-it is like a programming language in the sense that you have to
+write a program in Python that builds expressions for Theano. Still it
+is like a programming language in the sense that you have to
 
 - declare variables (``a,b``) and give their types
 
@@ -119,7 +119,6 @@ Theano is a sort of hybrid of the two which tries to make the best of
 both worlds.
 
 
-
 Getting started
 ===============
 
@@ -152,8 +151,8 @@ Questions, comments, praise, criticism as well as bug reports should
 be submitted to these mailing lists.
 
 We welcome all kinds of contributions. If you have any questions
-regarding how to extend Theano, please feel free to ask on the theano-dev_
-mailing list.
+regarding how to extend Theano, please feel free to ask on the
+theano-dev_ mailing list.
 
 
 

doc/advanced/debugmode.txt → doc/topics/debugmode.txt

@@ -41,6 +41,7 @@ In the example above, there is no way to guarantee that a future call to say,
 
 There following are DebugMode exceptions you might encounter:
 
+
 BadCLinkerOutput
 ----------------
 
@@ -116,6 +117,7 @@ performed, but the plan is that it will be.  (see ticket #320)
 
 For detailed documentation see :api:`FloatError`.
 
+
 InvalidValueError
 -----------------
 
@@ -126,6 +128,7 @@ Type.
 
 For detailed documentation see :api:`InvalidValueError`.
 
+
 DebugModeError
 --------------
 

doc/advanced/index.txt → doc/topics/index.txt

@@ -9,9 +9,7 @@ Advanced Topics
     :maxdepth: 2
    
     pipeline
-    unittest
     profilemode
-    debug_faq
     debugmode
     module_vs_op
     randomstreams

doc/advanced/unittest.txt → doc/topics/unittest.txt

 .. _unittest:
 
-===============
+============
 Unit Testing
-===============
+============
 
-Theano relies heavily on unit testing. Its importance cannot be stressed enough !
+Theano relies heavily on unit testing. Its importance cannot be
+stressed enough!
 
 Unit Testing revolves around the following principles:
 
-* ensuring correctness: making sure that your Op, Type or Optimization works in the way you intended it to work. It is important for this testing to be as thorough as possible: test not only the obvious cases, but more importantly the corner cases which are more likely to trigger bugs down the line.
-* test all possible failure paths. This means testing that your code fails in the appropriate manner, by raising the correct errors when in certain situations.
-* sanity check: making sure that everything still runs after you've done your modification. If your changes cause unit tests to start failing, it could be that you've changed an API on which other users rely on. It is therefore your responsibility to either a) provide the fix or b) inform the author of your changes and coordinate with that person to produce a fix. If this sounds like too much of a burden... then good ! APIs aren't meant to be changed on a whim !
-
-This page is in no way meant to replace tutorials on Python's unittest module, for this we refer the reader to the `official documentation <http://docs.python.org/library/unittest.html>`_.  We will however adress certain specificities about how unittests relate to theano.
+* ensuring correctness: making sure that your Op, Type or Optimization
+  works in the way you intended it to work. It is important for this
+  testing to be as thorough as possible: test not only the obvious
+  cases, but more importantly the corner cases which are more likely
+  to trigger bugs down the line.
+
+* test all possible failure paths. This means testing that your code
+  fails in the appropriate manner, by raising the correct errors when
+  in certain situations.
+
+* sanity check: making sure that everything still runs after you've
+  done your modification. If your changes cause unit tests to start
+  failing, it could be that you've changed an API on which other users
+  rely on. It is therefore your responsibility to either a) provide
+  the fix or b) inform the author of your changes and coordinate with
+  that person to produce a fix. If this sounds like too much of a
+  burden... then good! APIs aren't meant to be changed on a whim!
+
+This page is in no way meant to replace tutorials on Python's unittest
+module, for this we refer the reader to the `official documentation
+<http://docs.python.org/library/unittest.html>`_.  We will however
+adress certain specificities about how unittests relate to theano.
 
 Unittest Primer
 ===============
 
-A unittest is a subclass of ``unittest.TestCase``, with member functions with 
-names that start with the string ``test``.  For example:
+A unittest is a subclass of ``unittest.TestCase``, with member
+functions with names that start with the string ``test``.  For
+example:
 
 >>> class MyTestCase(unittest.TestCase):
 >>>    def test0(self):
@@ -34,15 +53,16 @@ names that start with the string ``test``.  For example:
 How to Run Unit Tests ?
 -----------------------
 
-Two options are avaiable. 
+Two options are available:
 
 Nosetests
 ~~~~~~~~~
 
-The easiest by far is to use ``nosetests`` which
-is a command line utility that recurses through a given directory, finds all
-unittests matching a specific criteria and executes them. By default, it will
-find & execute tests case in test*.py files whose method name starts with 'test'.
+The easiest by far is to use ``nosetests`` which is a command line
+utility that recurses through a given directory, finds all unittests
+matching a specific criteria and executes them. By default, it will
+find & execute tests case in test*.py files whose method name starts
+with 'test'.
 
 Running all unit tests
 
@@ -64,11 +84,12 @@ Running a specific unit test
 Using unittest module
 ~~~~~~~~~~~~~~~~~~~~~
 
-To launch tests cases from within python, you can also use the functionality
-offered by the ``unittest`` module. The simplest thing is to run all the tests in a file 
-using ``unittest.main()``. Python's built-in unittest module uses metaclasses 
-to know about all the ``unittest.TestCase`` classes you have created.  This 
-call will run them all, printing '.' for passed tests, and a stack trace for 
+To launch tests cases from within python, you can also use the
+functionality offered by the ``unittest`` module. The simplest thing
+is to run all the tests in a file using ``unittest.main()``. Python's
+built-in unittest module uses metaclasses to know about all the
+``unittest.TestCase`` classes you have created.  This call will run
+them all, printing '.' for passed tests, and a stack trace for
 exceptions. The standard footer code in theano's test files is:
 
 >>> if __name__ == '__main__':
@@ -92,13 +113,15 @@ To run just a single ``MyTestCase`` member test function called ``test0``:
 Folder Layout
 -------------
 
-"tests" directories are scattered throughout theano. Each tests subfolder is
-meant to contain the unittests which validate the .py files in the parent folder.
+"tests" directories are scattered throughout theano. Each tests
+subfolder is meant to contain the unittests which validate the .py
+files in the parent folder.
 
 Files containing unittests should be prefixed with the word "test".
 
-Optimally every python module should have a unittest file associated with it,
-as shown below. Unittests testing functionality of module <module>.py should therefore be stored in tests/test_<module>.py
+Optimally every python module should have a unittest file associated
+with it, as shown below. Unittests testing functionality of module
+<module>.py should therefore be stored in tests/test_<module>.py
 
 >>> Theano/theano/tensor/basic.py
 >>> Theano/theano/tensor/elemwise.py
@@ -112,20 +135,22 @@ How to Write a Unittest
 Test Cases and Methods
 ----------------------
 
-Unittests should be grouped "logically" into test cases, which are meant to
-group all unittests operating on the same element and/or concept. Test cases
-are implemented as Python classes which inherit from unittest.TestCase
+Unittests should be grouped "logically" into test cases, which are
+meant to group all unittests operating on the same element and/or
+concept. Test cases are implemented as Python classes which inherit
+from unittest.TestCase
 
-Test cases contain multiple test methods. These should be prefixed with the
-word "test". 
+Test cases contain multiple test methods. These should be prefixed
+with the word "test".
 
-Test methods should be as specific as possible and cover a particular aspect
-of the problem. For example, when testing the TensorDot Op, one test method
-could check for validity, while another could verify that the proper errors
-are raised when inputs have invalid dimensions.
+Test methods should be as specific as possible and cover a particular
+aspect of the problem. For example, when testing the TensorDot Op, one
+test method could check for validity, while another could verify that
+the proper errors are raised when inputs have invalid dimensions.
 
-Test method names should be as explicit as possible, so that users can see at
-first glance, what functionality is being tested and what tests need to be added.
+Test method names should be as explicit as possible, so that users can
+see at first glance, what functionality is being tested and what tests
+need to be added.
 
 Example:
 
@@ -136,10 +161,11 @@ Example:
 >>>     def test_invalid_dims(self):
 >>>         # do more stuff
 
-Test cases can define a special setUp method, which will get called before
-each test method is executed. This is a good place to put functionality which
-is shared amongst all test methods in the test case (i.e initializing data,
-parameters, seeding random number generators -- more on this later)
+Test cases can define a special setUp method, which will get called
+before each test method is executed. This is a good place to put
+functionality which is shared amongst all test methods in the test
+case (i.e initializing data, parameters, seeding random number
+generators -- more on this later)
 
 >>> class TestTensorDot(unittest.TestCase):
 >>>     def setUp(self):
@@ -147,15 +173,16 @@ parameters, seeding random number generators -- more on this later)
 >>>         self.avals = numpy.array([[1,5,3],[2,4,1]])
 >>>         self.bvals = numpy.array([[2,3,1,8],[4,2,1,1],[1,4,8,5]])
 
-Similarly, test cases can define a tearDown method, which will be implicitely
-called at the end of each test method.
+Similarly, test cases can define a tearDown method, which will be
+implicitely called at the end of each test method.
 
 
 Checking for correctness
 ------------------------
 
-When checking for correctness of mathematical expressions, the user should
-preferably compare theano's output to the equivalent numpy implementation. 
+When checking for correctness of mathematical expressions, the user
+should preferably compare theano's output to the equivalent numpy
+implementation.
 
 Example:
 
@@ -175,26 +202,34 @@ Avoid hard-coding variables, as in the following case:
 
 >>>         self.failUnless(numpy.all(f(self.avals,self.bvals)==numpy.array([[25,25,30,28],[21,18,14,25]])))
 
-This makes the test case less manageable and forces the user to update the
-variables each time the input is changed or possibly when the module being
-tested changes (after a bug fix for example). It also constrains the test case
-to specific input/output data pairs. The section on random values covers why this
-might not be such a good idea.
+This makes the test case less manageable and forces the user to update
+the variables each time the input is changed or possibly when the
+module being tested changes (after a bug fix for example). It also
+constrains the test case to specific input/output data pairs. The
+section on random values covers why this might not be such a good
+idea.
 
 Here is a list of useful functions, as defined by TestCase: 
 
-* checking the state of boolean variables: assert, failUnless, assertTrue, failIf, assertFalse
-* checking for (in)equality constraints: assertEqual, failUnlessEqual, assertNotEqual, failIfEqual
-* checking for (in)equality constraints up to a given precision (very useful in theano): assertAlmostEqual, failUnlessAlmostEqual, assertNotAlmostEqual, failIfAlmostEqual
+* checking the state of boolean variables: assert, failUnless,
+  assertTrue, failIf, assertFalse
+
+* checking for (in)equality constraints: assertEqual, failUnlessEqual,
+  assertNotEqual, failIfEqual
+
+* checking for (in)equality constraints up to a given precision (very
+  useful in theano): assertAlmostEqual, failUnlessAlmostEqual,
+  assertNotAlmostEqual, failIfAlmostEqual
 
 
 Checking for errors
 -------------------
 
-On top of verifying that your code provides the correct output, it is equally
-important to test that it fails in the appropriate manner, raising the
-appropriate exceptions, etc. Silent failures are deadly, as they can go unnoticed
-for a long time and a hard to detect "after-the-fact".
+On top of verifying that your code provides the correct output, it is
+equally important to test that it fails in the appropriate manner,
+raising the appropriate exceptions, etc. Silent failures are deadly,
+as they can go unnoticed for a long time and a hard to detect
+"after-the-fact".
 
 Example:
 
@@ -216,7 +251,8 @@ Useful functions, as defined by TestCase:
 Test Cases and Theano Modes
 ---------------------------
 
-When compiling theano functions or modules, a mode parameter can be given to specify which linker and optimizer to use.
+When compiling theano functions or modules, a mode parameter can be
+given to specify which linker and optimizer to use.
 
 Example:
 
@@ -224,9 +260,15 @@ Example:
 >>> m = theano.Module()
 >>> minstance = m.make(mode='DEBUG_MODE')
 
-Whenever possible, unit tests should omit this parameter. Leaving-out the mode will ensure that unit tests use the default mode (defined in compile.mode.default_mode). This default_mode is set to the THEANO_DEFAULT_MODE environment variable, if it is present. If not, it defaults to 'FAST_RUN'.
+Whenever possible, unit tests should omit this parameter. Leaving-out
+the mode will ensure that unit tests use the default mode (defined in
+compile.mode.default_mode). This default_mode is set to the
+THEANO_DEFAULT_MODE environment variable, if it is present. If not, it
+defaults to 'FAST_RUN'.
 
-This allows the user to easily switch the mode in which unittests are run. For example to run all tests in all modes from a BASH script, type this:
+This allows the user to easily switch the mode in which unittests are
+run. For example to run all tests in all modes from a BASH script,
+type this:
 
 .. code-block:: bash
 
@@ -237,14 +279,15 @@ This allows the user to easily switch the mode in which unittests are run. For e
 Using Random Values in Test Cases
 ---------------------------------
 
-numpy.random is often used in unit tests to initialize large data structures, 
-for use as inputs to the function or module being tested. When
-doing this, it is imperative that the random number generator be seeded at the
-be beginning of each unit test. This will ensure that unittest behaviour is
-consistent from one execution to another (i.e always pass or always fail).
+numpy.random is often used in unit tests to initialize large data
+structures, for use as inputs to the function or module being
+tested. When doing this, it is imperative that the random number
+generator be seeded at the be beginning of each unit test. This will
+ensure that unittest behaviour is consistent from one execution to
+another (i.e always pass or always fail).
 
-Instead of using numpy.random.seed to do this, we encourage users to do the
-following:
+Instead of using numpy.random.seed to do this, we encourage users to
+do the following:
 
 >>> from theano.tests import unittest_tools
 >>>
@@ -257,19 +300,24 @@ following:
 The behaviour of seed_rng is as follows:
 
 * If an explicit seed is given, it will be used for seending numpy's rng.
+
 * If not, it will try to get a seed from the THEANO_UNITTEST_SEED variable.
-* If THEANO_UNITTEST_SEED is set to "random", it will seed the rng. with None, which is equivalent to seeding with a random seed.
+
+* If THEANO_UNITTEST_SEED is set to "random", it will seed the
+  rng. with None, which is equivalent to seeding with a random seed.
+
 * If THEANO_UNITTEST_SEED is not defined, it will use a default seed of 666.
 
 
-The main advantage of using unittest_tools.seed_rng is that it allows us to
-change the seed used in the unitests, without having to manually edit all the
-files. For example, this allows the nightly build to run nosetests repeatedly,
-changing the seed on every run (hence achieving a higher confidence that the 
-variables are correct), while still making sure unittests are deterministic. 
+The main advantage of using unittest_tools.seed_rng is that it allows
+us to change the seed used in the unitests, without having to manually
+edit all the files. For example, this allows the nightly build to run
+nosetests repeatedly, changing the seed on every run (hence achieving
+a higher confidence that the variables are correct), while still
+making sure unittests are deterministic.
 
-Users who prefer their unittests to be random (when run on their local machine)
-can simply set THEANO_UNITTEST_SEED to 'random'.
+Users who prefer their unittests to be random (when run on their local
+machine) can simply set THEANO_UNITTEST_SEED to 'random'.
 
 Similarly, to provide a seed to numpy.random.RandomState, simply use:
 
@@ -277,23 +325,27 @@ Similarly, to provide a seed to numpy.random.RandomState, simply use:
 >>> # OR providing an explicit seed
 >>> rng = numpy.random.RandomState(unittest_tools.fetch_seed(1231)) #again not recommended
 
-Note that the ability to change the seed from one nosetest to another, is incompatible with the method of hard-coding the baseline variables (against which we compare the theano outputs). These must then be determined "algorithmically". Although this represents more work, the test suite will be better because of it.
+Note that the ability to change the seed from one nosetest to another,
+is incompatible with the method of hard-coding the baseline variables
+(against which we compare the theano outputs). These must then be
+determined "algorithmically". Although this represents more work, the
+test suite will be better because of it.
 
 
 Creating an Op UnitTest
 =======================
 
-A few tools have been developed to help automate the development of unitests
-for Theano Ops.
+A few tools have been developed to help automate the development of
+unitests for Theano Ops.
 
 
 Validating the Gradient
 -----------------------
 
 The ``verify_grad`` function can be used to validate that the ``grad``
-function of your Op is properly implemented. ``verify_grad`` is based on the
-Finite Difference Method where the derivative of function ``f`` at point ``x``
-is approximated as:
+function of your Op is properly implemented. ``verify_grad`` is based
+on the Finite Difference Method where the derivative of function ``f``
+at point ``x`` is approximated as:
 
 .. math::
 
@@ -302,8 +354,12 @@ is approximated as:
 ``verify_grad`` performs the following steps:
 
 * approximates the gradient numerically using the Finite Difference Method
-* calculate the gradient using the symbolic expression provided in the ``grad`` function
-* compares the two values. The tests passes if they are equal to within a certain tolerance.
+
+* calculate the gradient using the symbolic expression provided in the
+  ``grad`` function
+
+* compares the two values. The tests passes if they are equal to
+  within a certain tolerance.
 
 Here is the prototype for the verify_grad function.
 
@@ -315,12 +371,17 @@ the given tolerance.
 
 The parameters are as follows:
 
-* op: something that behaves like an Op instance with a single output (can be a python 
-  function combining multiple ops)
+* op: something that behaves like an Op instance with a single output
+  (can be a python function combining multiple ops)
+
 * pt: the list of numpy.ndarrays to use as inputs to the op
+
 * n_tests: number of times to run the test
+
 * rng: random number generator from which to draw random samples
+
 * eps: stepsize used in the Finite Difference Method
+
 * tol: relative tolerance used as threshold for gradient comparison
 
 Here is an example showing how to use verify_grad:
@@ -336,14 +397,15 @@ Here is an example showing how to use verify_grad:
 makeTester and makeBroadcastTester
 ==================================
 
-Most Op unittests perform the same function. All such tests must verify that
-the op generates the proper output, that the gradient is valid, that the Op
-fails in known/expected ways. Because so much of this is common, two helper
-functions exists to make your lives easier: ``makeTester`` and
-``makeBroadcastTester`` (defined in module ``theano.tensor.tests.test_basic``).
+Most Op unittests perform the same function. All such tests must
+verify that the op generates the proper output, that the gradient is
+valid, that the Op fails in known/expected ways. Because so much of
+this is common, two helper functions exists to make your lives easier:
+``makeTester`` and ``makeBroadcastTester`` (defined in module
+``theano.tensor.tests.test_basic``).
 
-Here is an example of ``makeTester`` generating testcases for the Dot product
-op:
+Here is an example of ``makeTester`` generating testcases for the Dot
+product op:
 
 >>> DotTester = makeTester(name = 'DotTester',
 >>>                        op = dot,
@@ -357,29 +419,34 @@ op:
 >>>                                           bad2 = (rand(5, 7), rand(8,3))),
 >>>                        grad = dict())
 
-In the above example, we provide a name and a reference to the op we want to
-test. We then provide in the ``expected`` field, a function which
-``makeTester`` can use to compute the correct values. The following five
-parameters are dictionaries which contain:
-
-* checks: dictionary of validation functions (dictionary key is a description
-  of what each function does). Each function accepts two parameters and
-  performs some sort of validation check on each op-input/op-output value pairs.
-  If the function returns False, an Exception is raised containing the
-  check's description.
-* good: contains valid input values, for which the output should match the 
-  expected output. Unittest will fail if this is not the case.
-* bad_build: invalid parameters which should generate an Exception when
-  attempting to build the graph (call to ``make_node`` should fail). 
-  Fails unless an Exception is raised.
-* bad_runtime: invalid parameters which should generate an Exception at
-  runtime, when trying to compute the actual output values (call to
+In the above example, we provide a name and a reference to the op we
+want to test. We then provide in the ``expected`` field, a function
+which ``makeTester`` can use to compute the correct values. The
+following five parameters are dictionaries which contain:
+
+* checks: dictionary of validation functions (dictionary key is a
+  description of what each function does). Each function accepts two
+  parameters and performs some sort of validation check on each
+  op-input/op-output value pairs.  If the function returns False, an
+  Exception is raised containing the check's description.
+
+* good: contains valid input values, for which the output should match
+  the expected output. Unittest will fail if this is not the case.
+
+* bad_build: invalid parameters which should generate an Exception
+  when attempting to build the graph (call to ``make_node`` should
+  fail).  Fails unless an Exception is raised.
+
+* bad_runtime: invalid parameters which should generate an Exception
+  at runtime, when trying to compute the actual output values (call to
   ``perform`` should fail). Fails unless an Exception is raised.
-* grad: dictionary containing input values which will be used in the call to
-  ``verify_grad``
+
+* grad: dictionary containing input values which will be used in the
+  call to ``verify_grad``
 
 
-``makeBroadcastTester`` is a wrapper function for makeTester. 
-If an ``inplace=True`` parameter is passed to it, it will take care of adding
-an entry to the ``checks`` dictionary. This check will ensure that inputs and
-outputs are equal, after the Op's perform function has been applied.
+``makeBroadcastTester`` is a wrapper function for makeTester.  If an
+``inplace=True`` parameter is passed to it, it will take care of
+adding an entry to the ``checks`` dictionary. This check will ensure
+that inputs and outputs are equal, after the Op's perform function has
+been applied.