merge

doc/advanced/pipeline.txt

@@ -97,6 +97,24 @@ output containers will be cleared (if they stay there, as would be the
 case if ``borrow`` was True, the thunk would be allowed to reuse (or
 "recycle") the storage).
 
+.. note::
+    Compiled libraries are stored within a specific compilation directory,
+    which by default is set to ``$HOME/.theano/compiledir_xxx``, where
+    ``xxx`` identifies the platform. It may be manually set to a different
+    location either by setting the ``THEANO_COMPILEDIR`` environment variable,
+    or by calling ``theano.gof.compiledir.set_compiledir(..)`` within your
+    Python script.
+
+    The compile cache is based upon the C++ code of the graph to be compiled.
+    So, if you change compilation environment variables, such as
+    ``THEANO_BLAS_LDFLAGS``, you will need to manually remove your compile cache,
+    using ``Theano/bin/theano-compiledir clear``
+
+    Theano also implements a lock mechanism that prevents
+    multiple compilations within the same compilation directory (to avoid
+    crashes with paralell execution of some scripts). This mechanism is
+    currently disabled by default, but can be enabled using the function
+    ``theano.gof.compilelock.set_lock_status(..)``.
 
 Step 4 - Wrap the thunk in a pretty package
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/advanced/unittest.txt

@@ -14,9 +14,36 @@ Unit Testing revolves around the following principles:
 
 This page is in no way meant to replace tutorials on Python's unittest module, for this we refer the reader to the `official documentation <http://docs.python.org/library/unittest.html>`_.  We will however adress certain specificities about how unittests relate to theano.
 
+Unittest Primer
+===============
+
+A unittest is a subclass of ``unittest.TestCase``, with member functions with 
+names that start with the string ``test``.  For example:
+
+>>> class MyTestCase(unittest.TestCase):
+>>>    def test0(self):
+>>>        pass                        # test passes cleanly
+>>>    def test1(self):
+>>>        self.failUnless(2+2 == 5)   # raises an exception, causes test to fail
+>>>    def test2(self):
+>>>        assert 2+2 == 5             # causes error in test (basically a failure, but counted separately)
+>>>    def test2(self):
+>>>        assert 2+2 == 4             # this test has the same name as a previous one, so this is the one that runs.
+
+
 How to Run Unit Tests ?
 -----------------------
 
+Two options are avaiable. 
+
+Nosetests
+~~~~~~~~~
+
+The easiest by far is to use ``nosetests`` which
+is a command line utility that recurses through a given directory, finds all
+unittests matching a specific criteria and executes them. By default, it will
+find & execute tests case in test*.py files whose method name starts with 'test'.
+
 Running all unit tests
 
 >>> cd Theano/theano
@@ -34,6 +61,33 @@ Running a specific unit test
 
 >>> nosetests <filename>.py:<classname>.<method_name>
 
+Using unittest module
+~~~~~~~~~~~~~~~~~~~~~
+
+To launch tests cases from within python, you can also use the functionality
+offered by the ``unittest`` module. The simplest thing is to run all the tests in a file 
+using ``unittest.main()``. Python's built-in unittest module uses metaclasses 
+to know about all the ``unittest.TestCase`` classes you have created.  This 
+call will run them all, printing '.' for passed tests, and a stack trace for 
+exceptions. The standard footer code in theano's test files is:
+
+>>> if __name__ == '__main__':
+>>>    unittest.main()
+
+You can also choose to run a subset of the full test suite.
+
+To run all the tests in one or more ``TestCase`` subclasses:
+
+>>>    suite = unittest.TestLoader()
+>>>    suite = suite.loadTestsFromTestCase(MyTestCase0)
+>>>    suite = suite.loadTestsFromTestCase(MyTestCase1)
+>>>    ...
+>>>    unittest.TextTestRunner(verbosity=2).run(suite)
+
+To run just a single ``MyTestCase`` member test function called ``test0``:
+
+>>>   MyTestCase('test0').debug()
+
 
 Folder Layout
 -------------
@@ -198,13 +252,15 @@ following:
 >>>     def setUp(self):
 >>>         unittest_tools.seed_rng()
 >>>         # OR ... call with an explicit seed 
->>>         unittest_tools.seed_rng(234234)
+>>>         unittest_tools.seed_rng(234234) #use only if really necessary!
 
 The behaviour of seed_rng is as follows:
 
-* if the environment variable THEANO_UNITTEST_SEED is defined, it will be used to seed the random number generator (and override any seed provided by the user)
-* if THEANO_UNITTEST_SEED is not defined, the user-supplied seed will be used to seed the rng
-* if THEANO_UNITTEST_SEED is not defined and no seed is given, the rng will be seeded with a random seed.
+* If an explicit seed is given, it will be used for seending numpy's rng.
+* If not, it will try to get a seed from the THEANO_UNITTEST_SEED variable.
+* If THEANO_UNITTEST_SEED is set to "random", it will seed the rng. with None, which is equivalent to seeding with a random seed.
+* If THEANO_UNITTEST_SEED is not defined, it will use a default seed of 666.
+
 
 The main advantage of using unittest_tools.seed_rng is that it allows us to
 change the seed used in the unitests, without having to manually edit all the
@@ -213,13 +269,13 @@ changing the seed on every run (hence achieving a higher confidence that the
 variables are correct), while still making sure unittests are deterministic. 
 
 Users who prefer their unittests to be random (when run on their local machine)
-can simply undefine THEANO_UNITTEST_SEED.
+can simply set THEANO_UNITTEST_SEED to 'random'.
 
 Similarly, to provide a seed to numpy.random.RandomState, simply use:
 
 >>> rng = numpy.random.RandomState(unittest_tools.fetch_seed())
 >>> # OR providing an explicit seed
->>> rng = numpy.random.RandomState(unittest_tools.fetch_seed(1231))
+>>> rng = numpy.random.RandomState(unittest_tools.fetch_seed(1231)) #again not recommended
 
 Note that the ability to change the seed from one nosetest to another, is incompatible with the method of hard-coding the baseline variables (against which we compare the theano outputs). These must then be determined "algorithmically". Although this represents more work, the test suite will be better because of it.
 
@@ -249,14 +305,81 @@ is approximated as:
 * calculate the gradient using the symbolic expression provided in the ``grad`` function
 * compares the two values. The tests passes if they are equal to within a certain tolerance.
 
->>> def verify_grad(testcase, op, pt, n_tests=1, rng=numpy.random, eps=1.0e-7, tol=0.0001,
->>>                mode=compile.Mode(optimizer=None, linker='c&py')):
->>>    """
->>>    Raises an Exception if the analytic gradient and numerical gradient exceeds a certain tolerance.
->>>    :param testcase: (obsolete) a reference to the unittest object calling 
->>>    :param pt: the list of numpy.ndarrays to use as inputs to the op
->>>    :param op: something that behaves like an Op instance with a single output
->>>     (can be a python function combining multiple ops)
->>>    :param testcase: the thing to call `fail` on if things go awry.
-
-
+Here is the prototype for the verify_grad function.
+
+>>> def verify_grad(op, pt, n_tests=2, rng=None, eps=1.0e-7, tol=0.0001):
+
+``verify_grad`` raises an Exception if the difference between the analytic gradient and
+numerical gradient (computed through the Finite Difference Method) exceeds
+the given tolerance.
+
+The parameters are as follows:
+
+* op: something that behaves like an Op instance with a single output (can be a python 
+  function combining multiple ops)
+* pt: the list of numpy.ndarrays to use as inputs to the op
+* n_tests: number of times to run the test
+* rng: random number generator from which to draw random samples
+* eps: stepsize used in the Finite Difference Method
+* tol: relative tolerance used as threshold for gradient comparison
+
+Here is an example showing how to use verify_grad:
+
+>>> def test_flatten_outdimNone():
+>>>    a = dmatrix()
+>>>    # ...
+>>>    a_val = numpy.asarray([[0,1,2],[3,4,5]], dtype='float64')
+>>>    # ...
+>>>    tensor.verify_grad(Flatten(), [a_val])
+
+
+makeTester and makeBroadcastTester
+==================================
+
+Most Op unittests perform the same function. All such tests must verify that
+the op generates the proper output, that the gradient is valid, that the Op
+fails in known/expected ways. Because so much of this is common, two helper
+functions exists to make your lives easier: ``makeTester`` and
+``makeBroadcastTester`` (defined in module ``theano.tensor.tests.test_basic``).
+
+Here is an example of ``makeTester`` generating testcases for the Dot product
+op:
+
+>>> DotTester = makeTester(name = 'DotTester',
+>>>                        op = dot,
+>>>                        expected = lambda x, y: numpy.dot(x, y),
+>>>                        checks = {},
+>>>                        good = dict(correct1 = (rand(5, 7), rand(7, 5)),
+>>>                                    correct2 = (rand(5, 7), rand(7, 9)),
+>>>                                    correct3 = (rand(5, 7), rand(7))),
+>>>                        bad_build = dict(),
+>>>                        bad_runtime = dict(bad1 = (rand(5, 7), rand(5, 7)),
+>>>                                           bad2 = (rand(5, 7), rand(8,3))),
+>>>                        grad = dict())
+
+In the above example, we provide a name and a reference to the op we want to
+test. We then provide in the ``expected`` field, a function which
+``makeTester`` can use to compute the correct values. The following five
+parameters are dictionaries which contain:
+
+* checks: dictionary of validation functions (dictionary key is a description
+  of what each function does). Each function accepts two parameters and
+  performs some sort of validation check on each op-input/op-output value pairs.
+  If the function returns False, an Exception is raised containing the
+  check's description.
+* good: contains valid input values, for which the output should match the 
+  expected output. Unittest will fail if this is not the case.
+* bad_build: invalid parameters which should generate an Exception when
+  attempting to build the graph (call to ``make_node`` should fail). 
+  Fails unless an Exception is raised.
+* bad_runtime: invalid parameters which should generate an Exception at
+  runtime, when trying to compute the actual output values (call to
+  ``perform`` should fail). Fails unless an Exception is raised.
+* grad: dictionary containing input values which will be used in the call to
+  ``verify_grad``
+
+
+``makeBroadcastTester`` is a wrapper function for makeTester. 
+If an ``inplace=True`` parameter is passed to it, it will take care of adding
+an entry to the ``checks`` dictionary. This check will ensure that inputs and
+outputs are equal, after the Op's perform function has been applied.

doc/install.txt

@@ -155,8 +155,9 @@ automatic code generation, but that way is much, much slower.
 
 - `THEANO_UNITTEST_SEED`:
     An integer value specifying which seed should be used when
-    running unit tests.
-    Setting this value will make the unit tests deterministic.
+    running unit tests. If this variable is not defined, the seed will default
+    to 666. Users wishing for non-deterministic behaviour can set this
+    variable to 'random'.
 
 
 Mac

doc/links.txt

@@ -13,7 +13,7 @@ Theano requirements
 
 - mercurial_: A distributed revision control system (RCS).
 - nosetests_: A system for unit tests.
-- numpy_: A library for efficient numerical computing.s
+- numpy_: A library for efficient numerical computing.
 - python_: The programming language Theano is for.
 - scipy_: A library for scientific computing.
 

doc/sandbox/sandbox.txt

@@ -153,18 +153,6 @@ export THEANO_BLAS_LDFLAGS='-lmkl -liomp5 -fopenmp'
 
 export OMP_NUM_THREADS=2
 
-=======================================================
-Cache
-=======================================================
-
-The compile cache is written to ``THEANO_COMPILEDIR``.  If this environment
-variable is not present, the compile cache defaults to ``$HOME/.theano``.
-The compile cache is based upon the C++ code of the graph to be compiled.
-So, if you change compilation environment variables, such as
-``THEANO_BLAS_LDFLAGS``, you will need to manually remove your compile cache.
-
-``Theano/bin/theano-compiledir clear``
-
 =======================================================
 Type checking
 =======================================================

theano/gof/cc.py

@@ -2,7 +2,6 @@
 Defines Linkers that deal with C implementations.
 """
 
-
 # Python imports
 from copy import copy
 import md5
@@ -20,6 +19,7 @@ import link
 import utils
 
 from compiledir import *
+from compilelock import get_lock, release_lock
 
 class CodeBlock:
     """WRITEME
@@ -301,7 +301,6 @@ def struct_variable_codeblocks(variable, policies, id, symbol_table, sub):
 
     return struct_builder, block
 
-
 class CLinker(link.Linker):
     """WRITEME
 
@@ -616,8 +615,17 @@ class CLinker(link.Linker):
           f()
           first_output = ostor[0].data
         """
+        # Note: acquiring the lock here may not be necessary. However, it is
+        # cheap enough that it should not matter.
+        get_lock()
+        try:
             cthunk, in_storage, out_storage, error_storage = self.__compile__(input_storage, output_storage)
-        return _execute(cthunk, self.init_tasks, self.tasks, error_storage), in_storage, out_storage
+            res = _execute(cthunk, self.init_tasks, self.tasks, error_storage), in_storage, out_storage
+        except:
+            release_lock()
+            raise
+        release_lock()
+        return res
 
     def cthunk_factory(self, error_storage, in_storage, out_storage):
         """WRITEME
@@ -632,9 +640,14 @@ class CLinker(link.Linker):
         type, value and traceback of the exception in error_storage.
         """
 
+        get_lock()
+
+        try:
+
             # check if we already compiled this
             if not getattr(self, 'instantiate', False):
     
+    
                 self.code_gen()
                 module_name = self.hash
     
@@ -708,11 +721,14 @@ class CLinker(link.Linker):
                     instantiate.customize.add_library(lib)
     
                 mod.add_function(instantiate)
-            #mod.compile(location = compile_dir())
+    
                 mod.compile(location = get_compiledir())
+    
                 module = __import__("%s" % (module_name), {}, {}, [module_name])
     
                 self.instantiate = module.instantiate
+                
+    
             else:
                 # Eliminate duplicate inputs and outputs from the storage that we will pass to instantiate
                 out_storage = [x for i, x in enumerate(out_storage) if (i+len(in_storage)) not in self.dupidx]
@@ -724,6 +740,12 @@ class CLinker(link.Linker):
             ret = module.instantiate(error_storage, *(in_storage + out_storage + orphd))
             #win pdb add 3 ref count, so we disable it by default.
             #assert sys.getrefcount(ret) == 2 # refcount leak check
+
+        except:
+            release_lock()
+            raise
+        release_lock()
+
         return ret
 
 
@@ -793,6 +815,10 @@ class OpWiseCLinker(link.LocalLinker):
         return self
 
     def make_all(self, profiler = None, input_storage = None, output_storage = None):
+
+        # Acquire lock on compilation directory.
+        get_lock()
+
         env = self.env
         order = env.toposort()
         no_recycling = self.no_recycling
@@ -869,6 +895,9 @@ class OpWiseCLinker(link.LocalLinker):
 
         f.allow_gc = self.allow_gc
 
+        # Release lock on compilation directory.
+        release_lock()
+
         return f, [link.Container(input, storage) for input, storage in zip(env.inputs, input_storage)], \
             [link.Container(output, storage, True) for output, storage in zip(env.outputs, output_storage)], \
             thunks, order
@@ -960,8 +989,3 @@ class DualLinker(link.Linker):
 
         return f, i1, o1
 
-
-
-
-
-

theano/gof/compiledir.py

@@ -32,7 +32,13 @@ def set_compiledir(path=None):
             path = os.path.join(os.getenv('HOME'), '.theano', 'compiledir_'+platform_id)
 
     if not os.access(path, os.R_OK | os.W_OK):
+        try:
             os.makedirs(path, 0770) #read-write-execute for this user only
+        except OSError, e:
+            # Maybe another parallel execution of theano was trying to create
+            # the same directory at the same time.
+            if e.errno != EEXIST:
+                raise
 
     # PROBLEM: sometimes the first approach based on os.system('touch')
     # returned -1 for an unknown reason; the alternate approach here worked

theano/gof/compilelock.py

+# Locking mechanism.
+
+import compiledir
+import os, random, time
+
+def get_lock():
+    """
+    Obtain lock on compilation directory.
+    """
+    if not hasattr(get_lock, 'n_lock'):
+        # Initialization.
+        get_lock.n_lock = 0
+        get_lock.lock_dir = os.path.join(compiledir.get_compiledir(), 'lock_dir')
+        if not hasattr(get_lock, 'lock_is_enabled'):
+            # Currently lock is disabled by default, due to the annoying
+            # persistence of lock files when compilation is being interrupted.
+            get_lock.lock_is_enabled = False
+    # Only really try to acquire the lock if we do not have it already.
+    if get_lock.lock_is_enabled and get_lock.n_lock == 0:
+        lock(get_lock.lock_dir, timeout = 60, verbosity = 10)
+    get_lock.n_lock += 1
+
+def release_lock():
+    """
+    Release lock on compilation directory.
+    """
+    get_lock.n_lock -= 1
+    assert get_lock.n_lock >= 0
+    # Only really release lock once all lock requests have ended.
+    if get_lock.lock_is_enabled and get_lock.n_lock == 0:
+        unlock(get_lock.lock_dir)
+
+def set_lock_status(use_lock):
+    """
+    Enable or disable the lock on the compilation directory (which is enabled
+    by default). Disabling may make compilation slightly faster (but is not
+    recommended for parallel execution).
+
+    @param use_lock: whether to use the compilation lock or not
+    @type  use_lock: bool
+    """
+    get_lock.lock_is_enabled = use_lock
+
+def lock(tmp_dir, timeout = 60, min_wait = 5, max_wait = 10, verbosity = 0):
+    """
+    Obtain lock access by creating a given temporary directory (whose base will
+    be created if needed, but will not be deleted after the lock is removed).
+    If access is refused by the same lock owner during more than 'timeout'
+    seconds, then the current lock is overridden. If timeout is None, then no
+    timeout is performed.
+    The lock is performed by creating a 'lock' file in 'tmp_dir' that contains
+    a unique id identifying the owner of the lock (the process id, followed by
+    a random string).
+    When there is already a lock, the process sleeps for a random amount of
+    time between min_wait and max_wait seconds before trying again.
+    If 'verbosity' is >= 1, then a message will be displayed when we need to
+    wait for the lock. If it is set to a value >1, then this message will be
+    displayed each time we re-check for the presence of the lock. Otherwise it
+    is displayed only when we notice the lock's owner has changed.
+
+    @param tmp_dir: lock directory that will be created when acquiring the lock
+    @type  tmp_dir: string
+
+    @param timeout: time (in seconds) to wait before replacing an existing lock
+    @type  timeout: int
+
+    @param min_wait: minimum time (in seconds) to wait before trying again to
+                     get the lock
+    @type  min_wait: int
+
+    @param max_wait: maximum time (in seconds) to wait before trying again to
+                     get the lock
+    @type  max_wait: int
+
+    @param verbosity: amount of feedback displayed to screen
+    @type  verbosity: int
+    """
+    # Create base of lock directory if required.
+    base_lock = os.path.basename(tmp_dir)
+    if not os.path.isdir(base_lock):
+        try:
+            os.makedirs(base_lock)
+        except:
+            # Someone else was probably trying to create it at the same time.
+            # We wait two seconds just to make sure the following assert does
+            # not fail on some NFS systems.
+            os.sleep(2)
+    assert os.path.isdir(base_lock)
+
+    # Variable initialization.
+    lock_file = os.path.join(tmp_dir, 'lock')
+    random.seed()
+    unique_id = '%s_%s' % (os.getpid(),
+            ''.join([str(random.randint(0,9)) for i in range(10)]))
+    no_display = (verbosity == 0)
+
+    # Acquire lock.
+    while True:
+        try:
+            last_owner = 'no_owner'
+            time_start = time.time()
+            while os.path.isdir(tmp_dir):
+                try:
+                    read_owner = open(lock_file).readlines()[0].strip()
+                except:
+                    read_owner = 'failure'
+                if last_owner == read_owner:
+                    if timeout is not None and time.time() - time_start >= timeout:
+                        # Timeout exceeded.
+                        break
+                else:
+                    last_owner = read_owner
+                    time_start = time.time()
+                    no_display = (verbosity == 0)
+                if not no_display:
+                    print 'Waiting for existing lock by %s (I am %s)' % (
+                            read_owner, unique_id)
+                    if verbosity <= 1:
+                        no_display = True
+                time.sleep(random.uniform(min_wait, max_wait))
+    
+            try:
+                os.mkdir(tmp_dir)
+            except:
+                # Error while creating the directory: someone else must have tried
+                # at the exact same time.
+                continue
+            # Safety check: the directory should be here.
+            assert os.path.isdir(tmp_dir)
+    
+            # Write own id into lock file.
+            lock_write = open(lock_file, 'w')
+            lock_write.write(unique_id + '\n')
+            lock_write.close()
+    
+            # Verify we are really the lock owner (this should not be needed,
+            # but better be safe than sorry).
+            owner = open(lock_file).readlines()[0].strip()
+            if owner != unique_id:
+                # Too bad, try again.
+                continue
+            else:
+                # We got the lock, hoorray!
+                return
+
+        except:
+            # If something wrong happened, we try again.
+            raise
+            continue
+    
+def unlock(tmp_dir):
+    """
+    Remove current lock.
+    This function assumes we have obtained the lock using lock(tmp_dir, ...),
+    so it does not check we are the lock owner.
+
+    @param tmp_dir: lock directory that will be removed when releasing the lock
+    @type  tmp_dir: string
+    """
+    lock_file = os.path.join(tmp_dir, 'lock')
+    if os.path.exists(lock_file):
+        os.remove(lock_file)
+    if os.path.exists(tmp_dir):
+        os.rmdir(tmp_dir)

theano/gof/cutils.py

 
 from compiledir import *
+from compilelock import get_lock, release_lock
 import sys
 
 
@@ -30,8 +31,14 @@ except ImportError:
     
     cthunk = object()
     mod = weave.ext_tools.ext_module('cutils_ext')
-    fun =weave.ext_tools.ext_function('run_cthunk', single_runner, ['cthunk'])
+    fun = weave.ext_tools.ext_function('run_cthunk', single_runner, ['cthunk'])
     fun.customize.add_extra_compile_arg('--permissive')
     mod.add_function(fun)
+    get_lock()
+    try:
         mod.compile(location = get_compiledir())
+    except:
+        release_lock()
+        raise
+    release_lock()
     from cutils_ext import *

theano/tests/unittest_tools.py

@@ -4,7 +4,21 @@ import numpy
 import os, sys
 
 def fetch_seed(pseed=None):
-    seed = os.getenv("THEANO_UNITTEST_SEED", pseed)
+    """
+    Returns the seed to use for running the unit tests.
+    If an explicit seed is given, it will be used for seending numpy's rng.
+    If not, it will try to get a seed from the THEANO_UNITTEST_SEED variable.
+    If THEANO_UNITTEST_SEED is set to "random", it will seed the rng. with None,
+    which is equivalent to seeding with a random seed.
+    If THEANO_UNITTEST_SEED is not defined, it will use a default seed of 666.
+
+    Useful for seeding RandomState objects.
+    >>> rng = numpy.random.RandomState(unittest_tools.fetch_seed())
+    """
+     
+    seed = pseed or os.getenv("THEANO_UNITTEST_SEED", 666)
+    seed = None if seed=='random' else seed
+
     try:
         seed = int(seed) if seed else None
     except ValueError:
@@ -15,6 +29,10 @@ def fetch_seed(pseed=None):
     return seed
 
 def seed_rng(pseed=None):
+    """
+    Seeds numpy's random number generator with the value returned by fetch_seed.
+    Usage: unittest_tools.seed_rng()
+    """
 
     seed = fetch_seed(pseed)
     if pseed and pseed!=seed: