
.. _basictutexamples:

=============
More examples
=============


Logistic function
=================

Here's another straightforward example, though a bit more elaborate
than adding two numbers together. Let's say that you want to compute
the logistic curve, which is given by:

.. math::

   s(x) = \frac{1}{1 + e^{-x}}

.. figure:: logistic.png

    A plot of the logistic function, with x on the x-axis and s(x) on the
    y-axis.

You want to compute the function :ref:`elementwise <libdoc_tensor_elementwise>` on matrices of
doubles, which means that you want to apply this function to each
individual element of the matrix.

Well, what you do is this:

>>> x = T.dmatrix('x')
>>> s = 1 / (1 + T.exp(-x))
>>> logistic = function([x], s)
>>> logistic([[0, 1], [-1, -2]])
array([[ 0.5       ,  0.73105858],
       [ 0.26894142,  0.11920292]])

The reason logistic is performed elementwise is because all of its
operations---division, addition, exponentiation, and division---are
themselves elementwise operations.

It is also the case that:

.. math::

    s(x) = \frac{1}{1 + e^{-x}} = \frac{1 + \tanh(x/2)}{2}

We can verify that this alternate form produces the same values:

>>> s2 = (1 + T.tanh(x / 2)) / 2
>>> logistic2 = function([x], s2)
>>> logistic2([[0, 1], [-1, -2]])
array([[ 0.5       ,  0.73105858],
       [ 0.26894142,  0.11920292]])


Computing more than one thing at the same time
==============================================

Theano supports functions with multiple outputs. For example, we can
compute the :ref:`elementwise <libdoc_tensor_elementwise>` difference, absolute difference, and
squared difference between two matrices ``a`` and ``b`` at the same time:

>>> a, b = T.dmatrices('a', 'b')
>>> diff = a - b
>>> abs_diff = abs(diff)
>>> diff_squared = diff**2
>>> f = function([a, b], [diff, abs_diff, diff_squared])

.. note:: 
   `dmatrices` produces as many outputs as names that you provide.  It's a
   shortcut for allocating symbolic variables that we will often use in the
   tutorials.

When we use the function, it will return the three variables (the printing
was reformatted for readability):

>>> f([[1, 1], [1, 1]], [[0, 1], [2, 3]])
[array([[ 1.,  0.],
        [-1., -2.]]),
 array([[ 1.,  0.],
        [ 1.,  2.]]),
 array([[ 1.,  0.],
        [ 1.,  4.]])]


Computing gradients
===================

Now let's use Theano for a slightly more sophisticated task: create a
function which computes the derivative of some expression ``y`` with
respect to its parameter ``x``. For instance, we can compute the
gradient of :math:`x^2` with respect to :math:`x`. Note that:
:math:`d(x^2)/dx = 2 \cdot x`.

Here is code to compute this gradient:

>>> x = T.dscalar('x')
>>> y = x**2
>>> gy = T.grad(y, x)
>>> pp(gy)  # print out the gradient prior to optimization
'((fill((x ** 2), 1.0) * 2) * (x ** (2 - 1)))'
>>> f = function([x], gy)
>>> f(4)
array(8.0)
>>> f(94.2)
array(188.40000000000001)

In the example above, we can see from ``pp(gy)`` that we are computing
the correct symbolic gradient.
``fill((x ** 2), 1.0)`` means to make a matrix of the same shape as
``x ** 2`` and fill it with 1.0.

.. note::
    The optimizer simplifies the symbolic gradient expression.  You can see
    this by digging inside the internal properties of the compiled function.

    .. code-block:: python

        pp(f.maker.env.outputs[0])
        '(2.0 * x)'

    After optimization there is only one Apply node left in the graph, which
    doubles the input.

We can also compute the gradient of complex expressions such as the
logistic function defined above. It turns out that the derivative of the
logistic is: :math:`ds(x)/dx = s(x) \cdot (1 - s(x))`.

.. figure:: dlogistic.png

    A plot of the gradient of the logistic function, with x on the x-axis
    and :math:`ds(x)/dx` on the y-axis.

>>> x = T.dmatrix('x')
>>> s = 1 / (1 + T.exp(-x))
>>> gs = T.grad(s, x)
>>> dlogistic = function([x], gs)
>>> dlogistic([[0, 1], [-1, -2]])
array([[ 0.25      ,  0.19661193],
       [ 0.19661193,  0.10499359]])


The resulting function computes the gradient of its first argument
with respect to the second. In this way, Theano can be used for
`automatic differentiation <http://en.wikipedia.org/wiki/Automatic_differentiation>`_.

.. note::
   
   The second argument of ``T.grad`` can be a list, in which case the
   output is also a list. The order in both list is important, element
   *i* of the output list is the gradient of the first argument of
   ``T.grad`` with respect to the *i*-th element of the list given as second argument.
   The first arguement of ``T.grad`` has to be a scalar (a tensor
   of size 1). For more information on the semantics of the arguments of 
   ``T.grad`` and details about the implementation, see :ref:`this <libdoc_gradient>`.


Setting a default value for an argument
=======================================

Let's say you want to define a function that adds two numbers, except
that if you only provide one number, the other input is assumed to be
one. You can do it like this:

>>> x, y = T.dscalars('x', 'y')
>>> z = x + y
>>> f = function([x, Param(y, default=1)], z)
>>> f(33)
array(34.0)
>>> f(33, 2)
array(35.0)

This makes use of the :ref:`Param <function_inputs>` class which allows
you to specify properties of your function's parameters with greater detail. Here we
give a default value of 1 for ``y`` by creating a ``Param`` instance with
its ``default`` field set to 1.

Inputs with default values must follow inputs without default
values (like python's functions).  There can be multiple inputs with default values. These parameters can
be set positionally or by name, as in standard Python:

>>> x, y, w = T.dscalars('x', 'y', 'w')
>>> z = (x + y) * w
>>> f = function([x, Param(y, default=1), Param(w, default=2, name='w_by_name')], z)
>>> f(33)
array(68.0)
>>> f(33, 2)
array(70.0)
>>> f(33, 0, 1)
array(33.0)
>>> f(33, w_by_name=1)
array(34.0)
>>> f(33, w_by_name=1, y=0)
array(33.0)

.. note::
   ``Param`` does not know the name of the local variables ``y`` and ``w``
   that are passed as arguments.  The symbolic variable objects have name
   attributes (set by ``dscalars`` in the example above) and *these* are the
   names of the keyword parameters in the functions that we build.  This is
   the mechanism at work in ``Param(y, default=1)``.  In the case of ``Param(w,
   default=2, name='w_by_name')``, we override the symbolic variable's name
   attribute with a name to be used for this function.


.. _functionstateexample:

Using shared variables
======================

It is also possible to make a function with an internal state. For
example, let's say we want to make an accumulator: at the beginning,
the state is initialized to zero. Then, on each function call, the state
is incremented by the function's argument.

First let's define the ``accumulator`` function. It adds its argument to the
internal state, and returns the old state value.

>>> state = shared(0)
>>> inc = T.iscalar('inc')
>>> accumulator = function([inc], state, updates=[(state, state+inc)])

This code introduces a few new concepts.  The ``shared`` function constructs
so-called :term:`shared variables`.  These are hybrid symbolic and non-symbolic
variables.  Shared variables can be used in symbolic expressions just like
the objects returned by ``dmatrices(...)`` but they also have a ``.value``
property that defines the value taken by this symbolic variable in *all* the
functions that use it.  It is called a *shared* variable because its value is
shared between many functions.  We'll come back to this soon.

The other new thing in this code is the ``updates`` parameter of function.
The updates is a list of pairs of the form (shared-variable, new expression).
It can also be a dictionary whose keys are shared-variables and values are
the new expressions.  Either way, it means "whenever this function runs, it
will replace the ``.value`` of each shared variable with the result of the
corresponding expression".  Above, our accumulator replaces the ``state``'s value with the sum
of the state and the increment amount.

Anyway, let's try it out! 

>>> state.value
array(0)
>>> accumulator(1)
array(0)
>>> state.value
array(1)
>>> accumulator(300)
array(1)
>>> state.value
array(301)

It is possible to reset the state. Just assign to the ``.value`` property:

>>> state.value = -1
>>> accumulator(3)
array(-1)
>>> state.value
array(2)

As we mentioned above, you can define more than one function to use the same
shared variable.  These functions can both update the value.

>>> decrementor = function([inc], state, updates=[(state, state-inc)])
>>> decrementor(2)
array(2)
>>> state.value
array(0)

You might be wondering why the updates mechanism exists.  You can always
achieve a similar thing by returning the new expressions, and working with
them in numpy as usual.  The updates mechanism can be a syntactic convenience,
but it is mainly there for efficiency.  Updates to shared variables can
sometimes be done more quickly using in-place algorithms (e.g. low-rank matrix
updates).  Also, theano has more control over where and how shared variables are
allocated, which is one of the important elements of getting good performance
on the GPU.

It may happen that you expressed some formula using a shared variable, but 
you do *not* want to use its value. In this case, you can use the
``givens`` parameter of ``function`` which replaces a particular node in a graph
for the purpose of one particular function.

>>> fn_of_state = state * 2 + inc
>>> foo = lscalar()    # the type (lscalar) must match the shared variable we
>>>                    # are replacing with the ``givens`` list
>>> skip_shared = function([inc, foo], fn_of_state,
        givens=[(state, foo)])
>>> skip_shared(1, 3)  # we're using 3 for the state, not state.value
array(7)
>>> state.value        # old state still there, but we didn't use it
array(0)  

The givens parameter can be used to replace any symbolic variable, not just a
shared variable. You can replace constants, and expressions, in general.  Be
careful though, not to allow the expressions introduced by a givens
substitution to be co-dependent, the order of substitution is not defined, so
the substitutions have to work in any order.

.. _using_random_numbers:

Using Random Numbers
====================

Because in Theano you first express everything symbolically and
afterwards compile this expression to get functions, 
using pseudo-random numbers is not as straightforward as it is in 
numpy, though also not too complicated. 

The way to think about putting randomness into Theano's computations is 
to put random variables in your graph. Theano will allocate a numpy 
RandomStream object (a random number generator) for each such 
variable, and draw from it as necessary. I'll call this sort of 
sequence of random numbers a *random stream*. *Random streams* are at 
their core shared variables, so the observations on shared variables
hold here as well.

Brief example
-------------

Here's a brief example.  The setup code is:

.. code-block:: python

    from theano.tensor.shared_randomstreams import RandomStreams
    srng = RandomStreams(seed=234)
    rv_u = srng.uniform((2,2))
    rv_n = srng.normal((2,2))
    f = function([], rv_u)
    g = function([], rv_n, no_default_updates=True)    #Not updating rv_n.rng
    nearly_zeros = function([], rv_u + rv_u - 2 * rv_u)

Here, 'rv_u' represents a random stream of 2x2 matrices of draws from a uniform
distribution.  Likewise,  'rv_n' represenents a random stream of 2x2 matrices of
draws from a normal distribution.  The distributions that are implemented are
defined in :class:`RandomStreams`.

Now let's use these things.  If we call f(), we get random uniform numbers.
The internal state of the random number generator is automatically updated,
so we get different random numbers every time.

>>> f_val0 = f()
>>> f_val1 = f()  #different numbers from f_val0

When we add the extra argument ``no_default_updates=True`` to
``function`` (as in ``g``), then the random number generator state is
not affected by calling the returned function.  So for example, calling
``g`` multiple times will return the same numbers.

>>> g_val0 = g()  # different numbers from f_val0 and f_val1
>>> g_val0 = g()  # same numbers as g_val0 !!!

An important remark is that a random variable is drawn at most once during any
single function execution.  So the ``nearly_zeros`` function is guaranteed to
return approximately 0 (except for rounding error) even though the ``rv_u``
random variable appears three times in the output expression.

>>> nearly_zeros = function([], rv_u + rv_u - 2 * rv_u)

Seedings Streams
----------------

Random variables can be seeded individually or collectively.

You can seed just one random variable by seeding or assigning to the
``.rng.value`` attribute.

>>> rv_u.rng.value.seed(89234)  # seeds the generator for rv_u

You can also seed *all* of the random variables allocated by a :class:`RandomStreams`
object by that object's ``seed`` method.  This seed will be used to seed a
temporary random number generator, that will in turn generate seeds for each
of the random variables.

>>> srng.seed(902340)  # seeds rv_u and rv_n with different seeds each

Sharing Streams between Functions
---------------------------------

As usual for shared variables, the random number generators used for random
variables are common between functions.  So our ``nearly_zeros`` function will
update the state of the generators used in function ``f`` above.

For example:

>>> state_after_v0 = rv_u.rng.value.get_state()
>>> nearly_zeros()       # this affects rv_u's generator
>>> v1 = f()
>>> rv_u.rng.value.set_state(state_after_v0)
>>> v2 = f()             # v2 != v1



