.. _metadocumentation:

==================================================
Documentation Documentation AKA Meta-Documentation
==================================================


How to link from Sphinx to API documentation
--------------------------------------------

Link from Sphinx to epydoc this way::

    :api:`linkname <path-to-symbol>`

For example::

    :api:`function <theano.compile.function>`

You can also use ``:api:\`path-to-symbol\``` directly.

If you are lazy, you can try::

    :api:`<function>`

However, if the link target is ambiguous, Sphinx will generate errors.

NB the ``:api:`` reference is special magic by Olivier, in
./doc/scripts/docgen.py.

How to add TODO comments in Sphinx documentation
-------------------------------------------------

To include a TODO comment in Sphinx documentation, use an indented block as
follows::

    .. TODO: This is a comment.
    .. You have to put .. at the beginning of every line :(
    .. These lines should all be indented.

It will not appear in the output generated.

    .. TODO: Check it out, this won't appear.
    .. Nor will this.

How to write API documentation
---------------------------------------

API documentation is processed by `epydoc
<http://epydoc.sourceforge.net/>`__ and should be written in
`reStructuredText <http://docutils.sourceforge.net/rst.html>`__. See `here
<http://epydoc.sourceforge.net/manual-othermarkup.html#restructuredtext>`__
for details about how to use reST with epydoc documentation.

Use ReST for API and Sphinx documentation
-----------------------------------------

 * ReST is standardized. epydoc is not. trac wiki-markup is not.
   This means that ReST can be cut-and-pasted between epydoc, code, other
   docs, and TRAC.  This is a huge win!
 * ReST is extensible: we can write our own roles and directives to automatically link to WIKI, for example.
 * ReST has figure and table directives, and can be converted (using a standard tool) to latex documents.
 * No text documentation has good support for math rendering, but ReST is closest: it has three renderer-specific solutions (render latex, use latex to build images for html, use itex2mml to generate MathML)

How to build documentation
---------------------------------------

Let's say you are writing documentation, and want to see
the `epydoc <http://epydoc.sourceforge.net/>`__ and `sphinx
<http://sphinx.pocoo.org/>`__ output before you push it.

The following technique uses Olivier's code to insert :api: links:

.. code-block:: bash

    cd Theano/
    python ./doc/scripts/docgen.py

If you don't want to rerun epydoc, do the following:

.. code-block:: bash

    cd Theano/
    python ./doc/scripts/docgen.py --rst


For more details:

.. code-block:: bash

   $ python doc/scripts/docgen.py --help
   Usage: doc/scripts/docgen.py [OPTIONS]
     -o <dir>: output the html files in the specified dir
     --rst: only compile the doc (requires sphinx)
     --nopdf: do not produce a PDF file from the doc, only HTML
     --epydoc: only compile the api documentation (requires epydoc)
     --help: this help


How documentation is auto-built on push
---------------------------------------

When you push to the main repository, the following file is run::

   projects@pylearn.org:~/repos/theano/.hg/refresh-epydoc.sh

It calls ``epydoc`` and ``sphinx`` on the code.
Output is written to location ``TARGET_DIR=/home/projects/www/theano``.
Note that this location is *removed* at each invocation, and a fresh version
is copied in.

Static files should be kept in
``STATIC_DIR=/home/projects/www/theano_static``. For example, the
tar.gz'ed releases are in ``STATIC_DIR``. After ``TARGET_DIR`` is
recreated, the contents of ``STATIC_DIR`` are copied into ``TARGET_DIR``.

*It would be nice to explain this stuff in a little more depth:*

 * Who calls refresh-epydoc.sh ?
 * What exactly does it do?
 * Can we get links about mercurial hooks?
 * What happens if two people push at the same time? Does documentation get clobbered?

pylint
---------------------------------------

Here is the autogenerated `pylint output
<http://pylearn.org/auto_theano_pylint/pylint_global.html>`__.

Pylint documentation is generated using pylintrc file: ``Theano/doc/pylintrc``

You can see a list of all `pylint messages
<http://www.logilab.org/card/pylintfeatures>`__.


.. _metadocumentation_nightly_build:

The nightly build/tests process
---------------------------------------

The user ``lisa`` runs a cronjob on the computer ``boltzmann``.
This happens nightly.
The cronjob executes the script ``~/www/nightly_build/do_nightly_build_theano``.
This script performs a an "hg pull -u" of theano (and pylearn too). 
Then it executes something like

.. code-block:: bash

    THEANO_FLAGS='unittests.rseed=<SEED>,mode=DEBUG_MODE' /usr/bin/nosetests --with-coverage --cover-package=theano --cover-package=pylearn

in the updated ``theano`` directory.
The output is emailed automatically to one of the developers.
This line is executed with a several fixed ``SEED`` values in DEBUG_MODE.  It
is executed in FAST_RUN and FAST_COMPILE with just one SEED value.

We can't email directly to the mailing list as it is emailed as the root user.
We haven't yet found a way to have google groups accept email from a user without sending an email to it.
As we don't have root control from this domain, we won't receive the email.
We haven't pursued the option of configuring the ``lisa`` account to send
email from an official gmail account.

Also, to have the crontab executed, *the ``lisa`` user must be logged into
``boltzmann``*.

In a nutshell: Fred keeps a shell open on boltzmann at all times, gets the
email, and forwards it to the mailing list when something interesting shows up
in the report.

TO WRITE
---------------------------------------

*There is other stuff to document here, e.g.:*

 * where the documentation is getting built, and epy.
 * How epydoc and sphinx are integrated
 * Explain the special magic in ./doc/scripts/docgen.py
 * We also want examples of good documentation, to show people how to write sphinx,

