
.. _omlw2014_index:

======================================================
Theano, Pylearn2, libgpuarray Presentation @ OMLW 2014
======================================================

August 22, 2014, New York University, US.

By Frédéric Bastien and Bart van Merriënboer. University of Montréal, Canada.


Theano, Pylearn2 and libgpuarray software stack for machine learning.


It complements the Python numeric/scientific software stack (e.g. NumPy, SciPy,
scikits, matplotlib, PIL.)

Theano
======

Theano is a software for evaluating and manipulating complicated array
expressions.

What does it do?

 * aggressive expression optimizations,

 * automatic GPU use,

 * automatic symbolic differentiation, Jacobian, Hession computation
   and R/L op (for hessian free).

Design and feature set has been driven by machine learning research
at the University of
Montreal (groups of Yoshua Bengio, Pascal Vincent, Aaron Courville and Roland Memisevic)
The result is a very good library for doing research in deep
learning and neural network training, and a flexible framework for
many other models and algorithms in machine learning more generally.

It has proven to be useful for implementing:

 - linear and nonlinear neural network classifiers

   - including Maxout, Dropout

 - convolutional models

 - Energy models: RBM, DBN, GRBM, ssRBM, AIS

 - Auto-encoders: DAE, CAE

 - GP regression

 - sparse coding

 - recurrent neural networks, echo state, (HMM?) TODO

 - online and batch learning and optimization

 - Even SVM!

As people's needs change this list will grow, but Theano is built
around vector, matrix, and tensor expressions. It also support sparse matrix.


Pylearn2
========

Pylearn2 is undergoing rapid development. Don’t expect a clean
road without bumps! It is made for machine learning
practitioner/researcher first.

Pylearn2 is a machine learning library. Most of its functionality is
built on top of Theano. This means you can write Pylearn2 plugins (new
models, algorithms, etc) using mathematical expressions, and Theano
will optimize and stabilize those expressions for you, and compile
them to a backend of your choice (CPU or GPU).


Pylearn2 Vision
---------------

TODO: SHould we split this in 2 part, what is done, what is the vision not done yet?

* Researchers **add features as they need them**. We avoid getting bogged down by
  too much top-down planning in advance.
* A machine learning toolbox for **easy scientific experimentation**.
* All models/algorithms published by the LISA lab should have reference
  implementations in Pylearn2. TODO REMOVE???
* Pylearn2 **may wrap other libraries** such as scikits.learn when this is practical
* Pylearn2 **differs from scikits.learn** in that Pylearn2 aims to provide great
  flexibility and make it possible for a researcher to do almost anything,
  while **scikits.learn aims to work as a "black box"**.
* **Dataset interface** for vector, images, video, ... TODO (DO WE HAVE VIDEO?)
* Small framework for all what is needed for one normal MLP/RBM/SDA/Convolution
  experiments. (TODO: I think I would remove this)
* **Easy reuse of sub-component** of Pylearn2.
  * Using one sub-component of the library does not force you to use / learn to
    use all of the other sub-components if you choose not to. TODO remove?
* Support cross-platform serialization of learned models.(TODO, I think this isn't done)
* Remain approachable enough to be used in the classroom


libgpuarray
===========

Make a common GPU ndarray(vector, matrix or n dimensions) that can be
reused by all projects. It support CUDA and OpenCL.

Motivation
----------

* Currently there are at least 6 different gpu arrays in python
  *  CudaNdarray(Theano), GPUArray(pycuda), CUDAMatrix(cudamat), GPUArray(pyopencl), Clyther, Copperhead, ...
  *  There are even more if we include other languages.

* They are incompatible
  * None have the same properties and interface.

* All of them are a subset of numpy.ndarray on the gpu!


Design Goals
------------

* Have the base object in C to allow collaboration with more projects.
  * We want people from C, C++, ruby, R, ... all use the same base GPU ndarray.
* Be compatible with CUDA and OpenCL.
* Not too simple, (don't support just matrix).
* But still easy to develop new code that support only a few memory layout.
  * This ease the development of new code.



Contents
========

.. toctree::

    introduction
    theano
    pylearn2
    gpundarray
    sharing
