.. _scan_internals:

Internal documentation of the scan op
=====================================

Top-level description of scan
-----------------------------

The `scan` operation is meant to be able to describe symbolically loops,
recurrent relations or dynamical systems. In general, we will say that the
scan op implements system of equations of the following form:

.. math::

    \mathbf{x}_1(t) = f_{\mathbf{x}_1}
        (\mathbf{u}_1(t), \mathbf{u}_1(t-1), \ldots, \mathbf{u}_1(t-l_1), 
         \mathbf{u}_2(t), \ldots, \mathbf{u}_2(t-l_2),
         \ldots,
         \mathbf{u}_M(t), \ldots, \mathbf{u}_M(t - l_M),
         \mathbf{x}_1(t-1), \ldots, \mathbf{x}_1(t-k_1),
         \ldots,
         \mathbf{x}_N(t-1), \ldots, \mathbf{x}_N(t-k_N), 
         \mathbf{w}_1, \ldots, \mathbf{w}_Q)

    \vdots

    \mathbf{x}_N(t) = f_{\mathbf{x}_N}
        (\mathbf{u}_1(t), \mathbf{u}_1(t-1), \ldots, \mathbf{u}_1(t-l_1), 
         \mathbf{u}_2(t), \ldots, \mathbf{u}_2(t-l_2),
         \ldots,
         \mathbf{u}_M(t), \ldots, \mathbf{u}_M(t - l_M),
         \mathbf{x}_1(t-1), \ldots, \mathbf{x}_1(t-k_1),
         \ldots,
         \mathbf{x}_N(t-1), \ldots, \mathbf{x}_N(t-k_N), 
         \mathbf{w}_1, \ldots, \mathbf{w}_Q)
    
    \mathbf{y}_1(t) = f_{\mathbf{y}_1}
        (\mathbf{u}_1(t), \mathbf{u}_1(t-1), \ldots, \mathbf{u}_1(t-l_1), 
         \mathbf{u}_2(t), \ldots, \mathbf{u}_2(t-l_2),
         \ldots,
         \mathbf{u}_M(t), \ldots, \mathbf{u}_M(t - l_M),
         \mathbf{x}_1(t-1), \ldots, \mathbf{x}_1(t-k_1),
         \ldots,
         \mathbf{x}_N(t-1), \ldots, \mathbf{x}_N(t-k_N), 
         \mathbf{w}_1, \ldots, \mathbf{w}_Q)

      \vdots

    \mathbf{y}_M(t) = f_{\mathbf{y}_M}
        (\mathbf{u}_1(t), \mathbf{u}_1(t-1), \ldots, \mathbf{u}_1(t-l_1), 
         \mathbf{u}_2(t), \ldots, \mathbf{u}_2(t-l_2),
         \ldots,
         \mathbf{u}_M(t), \ldots, \mathbf{u}_M(t - l_M),
         \mathbf{x}_1(t-1), \ldots, \mathbf{x}_1(t-k_1),
         \ldots,
         \mathbf{x}_N(t-1), \ldots, \mathbf{x}_N(t-k_N), 
         \mathbf{w}_1, \ldots, \mathbf{w}_Q)

The equations describe a system evolving in time, where :math:`t` represents the
current step. The system is described by inputs, states, outputs and
weights. 

The inputs, denoted by :math:`\mathbf{u}` are time-varying quantities, 
hence indexed by :math:`t`. They however only influence the system, but are
not influenced by the system. 

The states :math:`\mathbf{x}` are time-varying quantities, whose value at
time :math:`t` depends on its (or other state) previous values as well as
the inputs and weights. Note that the first few values of the states are
always provided, otherwise we could not imploy the recurrent equation to
generate these sequence of values without a starting point. 

The outputs, :math:`\mathbf{y}` are outputs of the system, i.e. values that
depend on the previous values of the states and inputs. The difference
between outputs and states is that outputs do not feed back into the system. 

The weights :math:`\mathbf{w}` are fixed quantities that are re-used at
every time step of the evolution of the system. 

Each of the equations above are implemented by the **inner function** of scan. You 
can think of the **inner function** as a theano function that gets executed
at each step to get the new values. This **inner function** should not be
confused with the **constructive function**, which is what the user gives to
the scan function. The **constructive function** is used to construct the
computational graph that is afterwards compiled into the **inner function**.


Naming conventions
------------------

* ``x`` will stand for a state :math:`\mathbf{x}`, while ``xs`` will represent
  the list of all states
* ``y`` will stand for an output :math:`\mathbf{y}`, while ``ys`` will
  represent the list of all outputs
* ``xy`` will stand for either a state or an output, while ``xys`` will be
  the list of all states and outputs
* ``u`` will be an input, wile ``us`` will be the list of all inputs
* ``w`` will stand for a weight tensor, while ``ws`` for the list of all
  weight tensors
* ``z`` will stand for states that are not numeric in nature. More
  specifically *random states*. ``zs`` is the list of all such states.
* ``t`` is the time index (the current step in the evolution of the system).
* ``T`` is the total number of steps in the evolution of the system.
* ``N`` will be the number of states, ``n`` will be the index that loops
  over the number of states
* ``M`` will be the number of inputs and ``m`` the dedicated index for
  inputs
* ``Q`` the number of weights
* the suffix ``_slices`` added to either ``x`` or ``u`` will mean the list of
  variables representing slices of states or inputs. These are the arguments
  given to the constructive function of scan (see above). 
* the suffix ``_inner`` added to ``x``, ``y``, ``xy``, ``u``, ``w`` or ``z``
  will mean the variables representing the state/output/input/weights in the
  inner function
* the suffix ``_outer`` added to ``x``, ``y``, ``xy``, ``u``, ``w`` or ``z``
  will mean the variables representing the state/output/input/weights in the
  main computational graph (the one containing the scan op).
* the suffx ``_results`` added to ``x``, ``y`` or ``xy`` are the list of
  expressions representing the next value of the state/output (i.e. the
  results after executing the inner function of scan)

Files 
-----

The implementation of scan is spread over several files. The different
files, and section of the code they deal with, are :

* ``scan.py`` implements the ``scan`` function. The ``scan`` function
  arranges the arguments of scan correctly, constructs the scan op and
  afterwards calls the constructed scan op on the arguments. This function
  takes care of figuring out missing inputs and shared variables. 

* ``scan_op.py`` implements the ``scanOp`` class. The ``scanOp`` respects
  the ``Op`` interface, and contains most of the logic of the scan operator.

* ``scan_utils.py`` contains several helpful functions used through out the
  other files that are specific of the scan operator.

* ``scan_views.py`` contains different views of the scan op that have
  simpler and easier signatures to be used in specific cases.

* ``scan_opt.py`` contains the list of all optimizations for the scan
  operator.

* ``cscan.pyx`` contains a cython implementation for the scan operator.


