Basically, this file contains stuff that should be documented, but is not.

Feel free to contribute things that you want documented, as well as to add
or correct documentation.


======================================
How do you define the grad function?
======================================

Let's talk about defining the `grad()` function in an Op, using an
illustrative example.

In Poisson regression (Ranzato and Szummer, 2008), the target *t* is
integer data, which we predict using a continuous output *o*.
In the negative log likelihood of the Poisson regressor, there is a term:

.. math::

    \log(t!)

Let's say we write a logfactorial Op. We then compute the gradient

You should define gradient, even if it is undefined.
[give log factorial example]

If an Op does not define ``grad``, but this Op does not appear in the path when
you compute the gradient, then there is no problem.

If an Op does not define ``grad``, and this Op *does* appear in the path when
you compute the gradient, **WRITEME**.

Gradients for a particular variable can be one of four kinds:
1) forgot to implement it

You will get an exception of the following form.

theano.gof.utils.MethodNotDefined: ('grad', <class 'pylearn.algorithms.sandbox.cost.LogFactorial'>, 'LogFactorial')

2) a symbolic variable
3) None / zero
4) undefined mathematically

currently, there is no way for a grad() method to distinguish between cases 3
and 4
but the distinction is important because graphs with type-3 gradients are ok
to run, whereas graphs with type-4 gradients are not.
so I suggested that Joseph return a type-4 gradient by defining an Op with no
perform method.
the idea would be that this would suit the graph-construction phase, but would
prevent linking.
how does that sound to you?

**This documentation is useful when we show users how to write Ops.**

======================================
What is staticmethod, st_impl?
======================================

``st_impl`` is an optional method in an Op.
``@staticmethod`` is a Python decorator for a class method that does not
implicitly take the class instance as a first argument. Hence, st_impl
can be used for Op implementations when no information from the Op
instance is needed. This can be useful for testing an implementation.
See :api:`XlogX` for an example.

**This documentation is useful when we show users how to write Ops.
Olivier says this behavior should be discouraged but I feel that st_impl
should be encouraged where possible.**

============================================================
how do we write scalar ops and upgrade them to tensor ops?
============================================================

**Olivier says that :api:`XlogX` gives a good example. In fact, I would
like to beef xlogx up into our running example for demonstrating how to
write an Op:**

.. code-block:: python

    class XlogX(scalar.UnaryScalarOp):
        """
        Compute X * log(X), with special case 0 log(0) = 0.
        """
        @staticmethod
        def st_impl(x):
            if x == 0.0:
                return 0.0
            return x * numpy.log(x)
        def impl(self, x):
            return XlogX.st_impl(x)
        def grad(self, inp, grads):
            x, = inp
            gz, = grads
            return [gz * (1 + scalar.log(x))]
        def c_code(self, node, name, inp, out, sub):
            x, = inp
            z, = out
            if node.inputs[0].type in [scalar.float32, scalar.float64]:
                return """%(z)s =
                    %(x)s == 0.0
                    ? 0.0
                    : %(x)s * log(%(x)s);""" % locals()
            raise NotImplementedError('only floatingpoint is implemented')
    scalar_xlogx  = XlogX(scalar.upgrade_to_float, name='scalar_xlogx')
    xlogx = tensor.Elemwise(scalar_xlogx, name='xlogx')

**It is also necessary to talk about UnaryScalarOp vs. BinaryOp.**

UnaryScalarOp is the same as scalar.ScalarOp with member variable nin=1.
**give an example of this**

=======================================================
Documentation on how to write tests
=======================================================

Guillaume can you make sure to hit these points:

    * What are canonical examples of tests?

        * What are the different test patterns?

    * nnet.py:

        * What is going on with test1, test2, test3, test4?

    * What is the right eq function to use?

        * There are a lot of tests that define their own epsilon, but this should be standardized.  e.g. in test_elemwise.py ``self.assertTrue((numpy.abs(f(xv) - zv) < 1e-10).all())``

    * If the expected variable of a test is that an Exception is thrown, how do we correctly detect and handle that?

        nosetests has ``assertRaises``

    * Convention is that all test files must start with ``test_``, not
        ``_test_``, so rename all that use the old convention?


=======================================================
How to use the PrintOp
=======================================================

** This is also useful in the How to write an Op tutorial. **

=======================================================
Modules
=======================================================

* What is the correct way to tie weights?

=======================================================
Mammouth
=======================================================

**This is internal documentation. Guillaume can you make sure to hit these points:**

export THEANO_BLAS_LDFLAGS='-lmkl -liomp5 -fopenmp'

**Do we want the following:**

export OMP_NUM_THREADS=2

=======================================================
Type checking
=======================================================

    * Are there functions for doing type checking?
        like dtype of this matrix is an int-type (not just int32
        or int64)
        "if isinstance(item, int):" is the preferred way to do it in
        python now, so mimic this
        If the type is wrong, what exception should be raised?

======================================
More simple numpy stuff
======================================

    * If we have a matrix with only one row, how do we convert it to a vector?
        ``x.reshape(x.size)``
        You can also use ``resize`` but there is not reason to ''resize''
    * How do you convert the type of a numpy array?
        ``theano._asarray(x, dtype = 'int32')``
        Note that using ``numpy.asarray`` is potentially dangerous, due to
        a problem in numpy where the type may not be properly set (see
        numpy's Track ticket #870).


=========================================
How to reuse (overwrite) a storage tensor
=========================================

``theano.compile.io.Out(gw1, borrow = True)`` for that value in
``compile.function``
