
.. _introduction:


************
Introduction
************

Background Questionaire
-----------------------

 * Who has used Theano before?

   * What did you do with it?

 * Who has used Python? numpy? scipy? matplotlib?

 * Who has used iPython?

   * Who has used it as a distributed computing engine?

 * Who has done C/C++ programming?

 * Who has organized computation around a particular physical memory layout?

 * Who has used a multidimensional array of >2 dimensions?

 * Who has written a Python module in C before?

   * Who has written a program to *generate* Python modules in C?

 * Who has used a templating engine?

 * Who has programmed a GPU before?

   * Using OpenGL / shaders ?

   * Using CUDA (runtime? / driver?)

   * Using PyCUDA ?

   * Using OpenCL / PyOpenCL ?

   * Other?

 * Who has used Cython?


Python in one slide
-------------------

Features:

- General-purpose high-level OO interpreted language
- Emphasizes code readability
- Comprehensive standard library
- Dynamic type and memory management

Language things:

- Indentation for block delimiters
- Dictionary ``d={'var1':'value1', 'var2':42, ...}``
- List comprehension: ``[i+3 for i in range(10)]``

.. code-block: python

    a = {'a': 5, 'b': None}   # dictionary of two elements
    b = [1,2,3]               # list of three int literals

    def foo(b, c=3):          # function w default param c
        return a + b + c      # note scoping, indentation


Numpy in one slide
------------------

- Numpy provides a N-dimensional numeric array in Python
- Well known basis for scientific computing
- provides:
  - elementwise computations
  - linear algebra, Fourier transforms
  - pseudorandom numbers from many distributions

- Base scientific computing package in Python on the CPU
- A powerful N-dimensional array object

  - ndarray.{ndim, shape, size, dtype, itemsize, stride}
  
- Sophisticated broadcasting functions
    
  - ``numpy.random.rand(4,5) * numpy.random.rand(1,5)`` -> mat(4,5)
  - ``numpy.random.rand(4,5) * numpy.random.rand(4,1)`` -> mat(4,5)
  - ``numpy.random.rand(4,5) * numpy.random.rand(5)`` -> mat(4,5)

- Tools for integrating C/C++ and Fortran code
- Linear algebra, Fourier transform and pseudorandom number generation




What's missing?
---------------


.. :

    Theano tries to be the **holy grail** in computing: *easy to code* and *it fast to execute* !

    It works only on mathematical expressions, so you won't have:

      - Function call inside a theano function
      - Structure, enum
      - Dynamic type (Theano is Fully typed)

    Unfortunately it doesn't do coffee... yet.

    .. image:: pics/Caffeine_Machine_no_background_red.png


Theano status
-------------

Why you can rely on Theano:

 - Theano has been developed and used since January 2008 (3.5 yrs old)
 - Core technology for a funded Silicon-Valley startup
 - Driven over 40 research papers in the last few years
 - Good user documentation
 - Active mailing list with participants from outside our lab
 - Many contributors (some from outside our lab)
 - Used to teach IFT6266 for two years
 - Used by everyone in our lab (~ 30 people)
 - Deep Learning Tutorials
 - Unofficial RPMs for Mandriva
 - Downloads (June 8 2011, since last January): Pypi 780, MLOSS: 483, Assembla (`bleeding edge` repository): unknown

Why Theano is better ?
----------------------

Executing the code is faster because Theano:
  - Rearranges high-level expressions
  - Produces customized low-level code
  - Uses a variety of backend technologies (GPU,...)

Writing the code is faster because:
  - High-level language allows to **concentrate on the algorithm**
  - Theano does **automatic optimization**

    - No need to manually optimize for each algorithm you want to test
  - Theano does **automatic efficient symbolic differentiation**
    
    - No need to manually differentiate your functions (tedious & error-prone for complicated expressions!)

Why scripting for GPUs ?
------------------------

**GPUs?**

- Faster, cheaper, more efficient power usage
- How much faster? I have seen numbers from 100x slower to 1000x faster.

  - It depends on the algorithms
  - How the benchmark is done
      
    - Quality of implementation
    - How much time was spent optimizing CPU vs GPU code

  - In Theory:

    - Intel Core i7 980 XE (107Gf/s float64) 6 cores
    - NVIDIA C2050 (515 Gf/s float64, 1Tf/s float32) 480 cores
    - NVIDIA GTX580 (1.5Tf/s float32) 512 cores
  
  - Theano goes up to 100x faster on th GPU because we don't use multiple core on CPU
    
    - Theano can be linked with multi-core capable BLAS (GEMM and GEMV)
  - If you see 1000x, it probably means the benchmark is not fair

**Scripting for GPUs?**

They *Complement each other*:

- GPUs are everything that scripting/high level languages are not

  - Highly parallel
  - Very architecture-sensitive
  - Built for maximum FP/memory throughput
- CPU: largely restricted to control

  - Optimized for sequential code and low latency (rather than high throughput)
  - Tasks (1000/sec)
  - Scripting fast enough

Theano vs PyCUDA vs PyOpenCL vs CUDA
------------------------------------

- Theano

  - Mathematical expression compiler
  - Generates costum C and CUDA code
  - Uses Python code when performance is not critical

- CUDA

  - C extension by NVIDA that allow to code and use GPU

- PyCUDA (Python + CUDA)

  - Python interface to CUDA
  - Memory management of GPU objects
  - Compilation of code for the low-level driver

- PyOpenCL (Python + OpenCL)
  - PyCUDA for OpenCL

