Merged

doc/advanced/unittest.txt

@@ -14,9 +14,36 @@ Unit Testing revolves around the following principles:
 
 This page is in no way meant to replace tutorials on Python's unittest module, for this we refer the reader to the `official documentation <http://docs.python.org/library/unittest.html>`_.  We will however adress certain specificities about how unittests relate to theano.
 
+Unittest Primer
+===============
+
+A unittest is a subclass of ``unittest.TestCase``, with member functions with 
+names that start with the string ``test``.  For example:
+
+>>> class MyTestCase(unittest.TestCase):
+>>>    def test0(self):
+>>>        pass                        # test passes cleanly
+>>>    def test1(self):
+>>>        self.failUnless(2+2 == 5)   # raises an exception, causes test to fail
+>>>    def test2(self):
+>>>        assert 2+2 == 5             # causes error in test (basically a failure, but counted separately)
+>>>    def test2(self):
+>>>        assert 2+2 == 4             # this test has the same name as a previous one, so this is the one that runs.
+
+
 How to Run Unit Tests ?
 -----------------------
 
+Two options are avaiable. 
+
+Nosetests
+~~~~~~~~~
+
+The easiest by far is to use ``nosetests`` which
+is a command line utility that recurses through a given directory, finds all
+unittests matching a specific criteria and executes them. By default, it will
+find & execute tests case in test*.py files whose method name starts with 'test'.
+
 Running all unit tests
 
 >>> cd Theano/theano
@@ -34,6 +61,33 @@ Running a specific unit test
 
 >>> nosetests <filename>.py:<classname>.<method_name>
 
+Using unittest module
+~~~~~~~~~~~~~~~~~~~~~
+
+To launch tests cases from within python, you can also use the functionality
+offered by the ``unittest`` module. The simplest thing is to run all the tests in a file 
+using ``unittest.main()``. Python's built-in unittest module uses metaclasses 
+to know about all the ``unittest.TestCase`` classes you have created.  This 
+call will run them all, printing '.' for passed tests, and a stack trace for 
+exceptions. The standard footer code in theano's test files is:
+
+>>> if __name__ == '__main__':
+>>>    unittest.main()
+
+You can also choose to run a subset of the full test suite.
+
+To run all the tests in one or more ``TestCase`` subclasses:
+
+>>>    suite = unittest.TestLoader()
+>>>    suite = suite.loadTestsFromTestCase(MyTestCase0)
+>>>    suite = suite.loadTestsFromTestCase(MyTestCase1)
+>>>    ...
+>>>    unittest.TextTestRunner(verbosity=2).run(suite)
+
+To run just a single ``MyTestCase`` member test function called ``test0``:
+
+>>>   MyTestCase('test0').debug()
+
 
 Folder Layout
 -------------
@@ -198,13 +252,15 @@ following:
 >>>     def setUp(self):
 >>>         unittest_tools.seed_rng()
 >>>         # OR ... call with an explicit seed 
->>>         unittest_tools.seed_rng(234234)
+>>>         unittest_tools.seed_rng(234234) #use only if really necessary!
 
 The behaviour of seed_rng is as follows:
 
-* if the environment variable THEANO_UNITTEST_SEED is defined, it will be used to seed the random number generator (and override any seed provided by the user)
-* if THEANO_UNITTEST_SEED is not defined, the user-supplied seed will be used to seed the rng
-* if THEANO_UNITTEST_SEED is not defined and no seed is given, the rng will be seeded with a random seed.
+* If an explicit seed is given, it will be used for seending numpy's rng.
+* If not, it will try to get a seed from the THEANO_UNITTEST_SEED variable.
+* If THEANO_UNITTEST_SEED is set to "random", it will seed the rng. with None, which is equivalent to seeding with a random seed.
+* If THEANO_UNITTEST_SEED is not defined, it will use a default seed of 666.
+
 
 The main advantage of using unittest_tools.seed_rng is that it allows us to
 change the seed used in the unitests, without having to manually edit all the
@@ -213,13 +269,13 @@ changing the seed on every run (hence achieving a higher confidence that the
 variables are correct), while still making sure unittests are deterministic. 
 
 Users who prefer their unittests to be random (when run on their local machine)
-can simply undefine THEANO_UNITTEST_SEED.
+can simply set THEANO_UNITTEST_SEED to 'random'.
 
 Similarly, to provide a seed to numpy.random.RandomState, simply use:
 
 >>> rng = numpy.random.RandomState(unittest_tools.fetch_seed())
 >>> # OR providing an explicit seed
->>> rng = numpy.random.RandomState(unittest_tools.fetch_seed(1231))
+>>> rng = numpy.random.RandomState(unittest_tools.fetch_seed(1231)) #again not recommended
 
 Note that the ability to change the seed from one nosetest to another, is incompatible with the method of hard-coding the baseline variables (against which we compare the theano outputs). These must then be determined "algorithmically". Although this represents more work, the test suite will be better because of it.
 
@@ -249,14 +305,81 @@ is approximated as:
 * calculate the gradient using the symbolic expression provided in the ``grad`` function
 * compares the two values. The tests passes if they are equal to within a certain tolerance.
 
->>> def verify_grad(testcase, op, pt, n_tests=1, rng=numpy.random, eps=1.0e-7, tol=0.0001,
->>>                mode=compile.Mode(optimizer=None, linker='c&py')):
->>>    """
->>>    Raises an Exception if the analytic gradient and numerical gradient exceeds a certain tolerance.
->>>    :param testcase: (obsolete) a reference to the unittest object calling 
->>>    :param pt: the list of numpy.ndarrays to use as inputs to the op
->>>    :param op: something that behaves like an Op instance with a single output
->>>     (can be a python function combining multiple ops)
->>>    :param testcase: the thing to call `fail` on if things go awry.
-
-
+Here is the prototype for the verify_grad function.
+
+>>> def verify_grad(op, pt, n_tests=2, rng=None, eps=1.0e-7, tol=0.0001):
+
+``verify_grad`` raises an Exception if the difference between the analytic gradient and
+numerical gradient (computed through the Finite Difference Method) exceeds
+the given tolerance.
+
+The parameters are as follows:
+
+* op: something that behaves like an Op instance with a single output (can be a python 
+  function combining multiple ops)
+* pt: the list of numpy.ndarrays to use as inputs to the op
+* n_tests: number of times to run the test
+* rng: random number generator from which to draw random samples
+* eps: stepsize used in the Finite Difference Method
+* tol: relative tolerance used as threshold for gradient comparison
+
+Here is an example showing how to use verify_grad:
+
+>>> def test_flatten_outdimNone():
+>>>    a = dmatrix()
+>>>    # ...
+>>>    a_val = numpy.asarray([[0,1,2],[3,4,5]], dtype='float64')
+>>>    # ...
+>>>    tensor.verify_grad(Flatten(), [a_val])
+
+
+makeTester and makeBroadcastTester
+==================================
+
+Most Op unittests perform the same function. All such tests must verify that
+the op generates the proper output, that the gradient is valid, that the Op
+fails in known/expected ways. Because so much of this is common, two helper
+functions exists to make your lives easier: ``makeTester`` and
+``makeBroadcastTester`` (defined in module ``theano.tensor.tests.test_basic``).
+
+Here is an example of ``makeTester`` generating testcases for the Dot product
+op:
+
+>>> DotTester = makeTester(name = 'DotTester',
+>>>                        op = dot,
+>>>                        expected = lambda x, y: numpy.dot(x, y),
+>>>                        checks = {},
+>>>                        good = dict(correct1 = (rand(5, 7), rand(7, 5)),
+>>>                                    correct2 = (rand(5, 7), rand(7, 9)),
+>>>                                    correct3 = (rand(5, 7), rand(7))),
+>>>                        bad_build = dict(),
+>>>                        bad_runtime = dict(bad1 = (rand(5, 7), rand(5, 7)),
+>>>                                           bad2 = (rand(5, 7), rand(8,3))),
+>>>                        grad = dict())
+
+In the above example, we provide a name and a reference to the op we want to
+test. We then provide in the ``expected`` field, a function which
+``makeTester`` can use to compute the correct values. The following five
+parameters are dictionaries which contain:
+
+* checks: dictionary of validation functions (dictionary key is a description
+  of what each function does). Each function accepts two parameters and
+  performs some sort of validation check on each op-input/op-output value pairs.
+  If the function returns False, an Exception is raised containing the
+  check's description.
+* good: contains valid input values, for which the output should match the 
+  expected output. Unittest will fail if this is not the case.
+* bad_build: invalid parameters which should generate an Exception when
+  attempting to build the graph (call to ``make_node`` should fail). 
+  Fails unless an Exception is raised.
+* bad_runtime: invalid parameters which should generate an Exception at
+  runtime, when trying to compute the actual output values (call to
+  ``perform`` should fail). Fails unless an Exception is raised.
+* grad: dictionary containing input values which will be used in the call to
+  ``verify_grad``
+
+
+``makeBroadcastTester`` is a wrapper function for makeTester. 
+If an ``inplace=True`` parameter is passed to it, it will take care of adding
+an entry to the ``checks`` dictionary. This check will ensure that inputs and
+outputs are equal, after the Op's perform function has been applied.

doc/install.txt

@@ -155,8 +155,9 @@ automatic code generation, but that way is much, much slower.
 
 - `THEANO_UNITTEST_SEED`:
     An integer value specifying which seed should be used when
-    running unit tests.
-    Setting this value will make the unit tests deterministic.
+    running unit tests. If this variable is not defined, the seed will default
+    to 666. Users wishing for non-deterministic behaviour can set this
+    variable to 'random'.
 
 
 Mac

theano/tests/unittest_tools.py

@@ -4,7 +4,21 @@ import numpy
 import os, sys
 
 def fetch_seed(pseed=None):
-    seed = os.getenv("THEANO_UNITTEST_SEED", pseed)
+    """
+    Returns the seed to use for running the unit tests.
+    If an explicit seed is given, it will be used for seending numpy's rng.
+    If not, it will try to get a seed from the THEANO_UNITTEST_SEED variable.
+    If THEANO_UNITTEST_SEED is set to "random", it will seed the rng. with None,
+    which is equivalent to seeding with a random seed.
+    If THEANO_UNITTEST_SEED is not defined, it will use a default seed of 666.
+
+    Useful for seeding RandomState objects.
+    >>> rng = numpy.random.RandomState(unittest_tools.fetch_seed())
+    """
+     
+    seed = pseed or os.getenv("THEANO_UNITTEST_SEED", 666)
+    seed = None if seed=='random' else seed
+
     try:
         seed = int(seed) if seed else None
     except ValueError:
@@ -15,6 +29,10 @@ def fetch_seed(pseed=None):
     return seed
 
 def seed_rng(pseed=None):
+    """
+    Seeds numpy's random number generator with the value returned by fetch_seed.
+    Usage: unittest_tools.seed_rng()
+    """
 
     seed = fetch_seed(pseed)
     if pseed and pseed!=seed: