
.. _omlw2014_Introduction:


************
Introduction
************

Python in one slide
-------------------

* General-purpose high-level **OO interpreted language**
 
* Emphasizes **code readability**
 
* Comprehensive standard library
 
* Dynamic type and memory management

* Built-in types: int, float, str, list, dict, tuple, object

* Slow execution

* Popular in **web-dev** and **scientific communities**


NumPy in one slide
------------------

* Python floats are full-fledged objects on the heap

 * Not suitable for high-performance computing!

* NumPy provides a N-dimensional numeric array in Python

 * Perfect for high-performance computing.
 * Slice are return view (no copy)

* NumPy provides

 * elementwise computations

 * linear algebra, Fourier transforms

 * pseudorandom numbers from many distributions

* SciPy provides lots more, including

 * more linear algebra

 * solvers and optimization algorithms

 * matlab-compatible I/O

 * I/O and signal processing for images and audio

.. code-block:: python

    ##############################
    # Properties of NumPy arrays
    # that you really need to know
    ##############################

    import numpy as np          # import can rename
    a = np.random.rand(3, 4, 5) # random generators
    a32 = a.astype('float32')   # arrays are strongly typed

    a.ndim                      # int: 3
    a.shape                     # tuple: (3, 4, 5)
    a.size                      # int: 60
    a.dtype                     # np.dtype object: 'float64'
    a32.dtype                   # np.dtype object: 'float32'

    assert a[1, 1, 1] != 10     # a[1, 1, 1] is a view
    a[1, 1, 1] = 10             # So affectation to it change the
    assert a[1, 1, 1] == 10     # original array


Arrays can be combined with numeric operators, standard mathematical
functions. NumPy has great `documentation <http://docs.scipy.org/doc/numpy/reference/>`_.

What's missing?
---------------

* Non-lazy evaluation (required by Python) hurts performance

* NumPy is bound to the CPU

* NumPy lacks symbolic or automatic differentiation

Quick look at a small examples:

.. code-block:: python

    #########################
    # Theano for Training a
    # Neural Network on MNIST
    #########################

    import numpy as np

    import theano
    import theano.tensor as tensor

    x = np.load('data_x.npy')
    y = np.load('data_y.npy')

    # symbol declarations
    sx = tensor.matrix()
    sy = tensor.matrix()
    w = theano.shared(np.random.normal(avg=0, std=.1,
                                       size=(784, 500)))
    b = theano.shared(np.zeros(500))
    v = theano.shared(np.zeros((500, 10)))
    c = theano.shared(np.zeros(10))

    # symbolic expression-building
    hid = tensor.tanh(tensor.dot(sx, w) + b)
    out = tensor.tanh(tensor.dot(hid, v) + c)
    err = 0.5 * tensor.sum(out - sy) ** 2
    gw, gb, gv, gc = tensor.grad(err, [w, b, v, c])

    # compile a fast training function
    train = theano.function([sx, sy], err,
        updates={
            w: w - lr * gw,
            b: b - lr * gb,
            v: v - lr * gv,
            c: c - lr * gc})

    # now do the computations
    batchsize = 100
    for i in xrange(1000):
        x_i = x[i * batchsize: (i + 1) * batchsize]
        y_i = y[i * batchsize: (i + 1) * batchsize]
        err_i = train(x_i, y_i)

    
Theano in one slide
-------------------

* High-level domain-specific language tailored to numeric computation

* Compiles most common expressions to C for CPU and GPU.

* Limited expressivity means lots of opportunities for expression-level optimizations

 * No function call -> global optimization

 * Strongly typed -> compiles to machine instructions

 * Array oriented -> easy parallelism

 * Support for looping and branching in expressions

* Expression substitution optimizations automatically draw
  on many backend technologies for best performance.

 * BLAS, SciPy, Cython, CUDA

 * Slower fallbacks always available

* Automatic differentiation and R op

* Sparse matrices


Project status
--------------

* Mature: theano has been developed and used since January 2008 (6.5 yrs old)

* Driven over 100 research papers

* Good user documentation

* Active mailing list with participants from outside our lab

* Core technology for a few Silicon-Valley startup

* Many contributors (some from outside our lab)

* Used to teach many university classes

* Used for research at Google and Yahoo. (TODO, should we remove? I think so)


Pylearn2 in one slide
---------------------

TODO

Other global information
------------------------

Theano have small basic operation, not layers as base operation:

* Easy reuse
* Don't need to reimplement the grad for each variation of layers

This could cause slowness (more small operation), but the optimizer fix that.

Pylearn2 wrap the small operations into layers like other
projects:

* There is no overhead to this extra layer, due to the
  compilation of the function by Theano.


Why scripting for GPUs?
-----------------------

They *Complement each other*:

* GPUs are everything that scripting/high level languages are not

 * Highly parallel

 * Very architecture-sensitive

 * Built for maximum FP/memory throughput

 * So hard to program that meta-programming is easier.

* CPU: largely restricted to control

 * Optimized for sequential code and low latency (rather than high throughput)

 * Tasks (1000/sec)

 * Scripting fast enough

Best of both: scripted CPU invokes JIT-compiled kernels on GPU.
