<tableid="mw-sharedupload"class="plainlinks fmbox fmbox-system"role="presentation"><tr><tdclass="mbox-image"><ahref="/wiki/File:Commons-logo.svg"class="image"title="Wikimedia Commons logo"><imgalt=""src="//upload.wikimedia.org/wikipedia/en/thumb/4/4a/Commons-logo.svg/30px-Commons-logo.svg.png"width="30"height="40"srcset="//upload.wikimedia.org/wikipedia/en/thumb/4/4a/Commons-logo.svg/45px-Commons-logo.svg.png 1.5x, //upload.wikimedia.org/wikipedia/en/thumb/4/4a/Commons-logo.svg/59px-Commons-logo.svg.png 2x"data-file-width="1024"data-file-height="1376"/></a></td><tdclass="mbox-text"style="text-align: center;"><div>This is a file from the <ahref="//commons.wikimedia.org/wiki/Main_Page"class="extiw"title="commons:Main Page">Wikimedia Commons</a>. Information from its <b><ahref="//commons.wikimedia.org/wiki/File:Elman_srnn.png"class="extiw"title="commons:File:Elman srnn.png">description page there</a></b> is shown below.<br/><spanstyle="font-size: smaller;">Commons is a freely licensed media file repository. <ahref="//commons.wikimedia.org/wiki/Commons:Welcome"class="extiw"title="commons:Commons:Welcome">You can help</a>.</span></div></td></tr></table><divclass="metadata topicon"id="commons-icon"style="display:none; right:30px;"><ahref="//commons.wikimedia.org/wiki/File:Elman_srnn.png"title="This is a file from the Wikimedia Commons"><imgalt="This is a file from the Wikimedia Commons"src="//upload.wikimedia.org/wikipedia/en/thumb/4/4a/Commons-logo.svg/14px-Commons-logo.svg.png"width="14"height="19"srcset="//upload.wikimedia.org/wikipedia/en/thumb/4/4a/Commons-logo.svg/21px-Commons-logo.svg.png 1.5x, //upload.wikimedia.org/wikipedia/en/thumb/4/4a/Commons-logo.svg/28px-Commons-logo.svg.png 2x"data-file-width="1024"data-file-height="1376"/></a></div>
<divclass="description mw-content-ltr en"dir="ltr"lang="en"style=""><spanclass="language en"title=""><b>English:</b></span> The Elman Simple Recurrent Neural Network. The context layer (u1 ... ul) provides a temporal memory to the network, allowing it to "remember" previous hidden layer states. Generated in LaTeX via PGF/Tikz.</div>
<td><ahref="//commons.wikimedia.org/w/index.php?title=User:Fyedernoggersnodden&action=edit&redlink=1"class="new"title="User:Fyedernoggersnodden (page does not exist)">Fyedernoggersnodden</a></td>
<divclass="center"><b><spanlang="en"class="description en">I, the copyright holder of this work, hereby publish it under the following license:</span></b></div>
<td>This file is licensed under the <ahref="//en.wikipedia.org/wiki/en:Creative_Commons"class="extiw"title="w:en:Creative Commons">Creative Commons</a><arel="nofollow"class="external text"href="//creativecommons.org/licenses/by/3.0/deed.en">Attribution 3.0 Unported</a> license.</td>
<tdstyle="width:90px;"rowspan="3"></td>
</tr>
<trstyle="text-align:center;">
<td></td>
</tr>
<trstyle="text-align:left;">
<td>
<dl>
<dd>You are free:
<ul>
<li><b>to share</b> – to copy, distribute and transmit the work</li>
<li><b>to remix</b> – to adapt the work</li>
</ul>
</dd>
<dd>Under the following conditions:
<ul>
<li><b>attribution</b> – You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).</li>
</ul>
</dd>
</dl>
<p><spanclass="licensetpl_link"style="display:none;">http://creativecommons.org/licenses/by/3.0</span><spanclass="licensetpl_short"style="display:none;">CC BY 3.0</span><spanclass="licensetpl_long"style="display:none;">Creative Commons Attribution 3.0</span><spanclass="licensetpl_link_req"style="display:none;">true</span><spanclass="licensetpl_attr_req"style="display:none;">true</span></p>
</td>
</tr>
</table>
</td>
</tr>
</table>
<!--
NewPP limit report
Parsed by mw1051
CPU time usage: 0.093 seconds
Real time usage: 0.120 seconds
Preprocessor visited node count: 687/1000000
Preprocessor generated node count: 0/1500000
Post‐expand include size: 10714/2097152 bytes
Template argument size: 2520/2097152 bytes
Highest expansion depth: 10/40
Expensive parser function count: 0/500
Lua time usage: 0.016/10.000 seconds
Lua memory usage: 697 KB/50 MB
-->
<!--
Transclusion expansion time report (%,ms,calls,template)
100.00% 96.610 1 - -total
75.24% 72.694 1 - Template:Information
37.74% 36.456 1 - Template:ISOdate
24.31% 23.483 1 - Template:Self
14.27% 13.782 1 - Template:Cc-by-3.0
11.80% 11.404 1 - Template:Cc-by-layout
7.63% 7.368 1 - Template:Parse_source
6.35% 6.130 1 - Template:En
6.08% 5.877 7 - Template:Dir
4.45% 4.303 1 - Template:CC-Layout
-->
<!-- Saved in parser cache with key commonswiki:pcache:idhash:5837043-1!*!0!!en!4!* and timestamp 20150105104356 and revision id 138607301
-->
</div>
<h2id="filehistory">File history</h2>
<divid="mw-imagepage-section-filehistory">
<p>Click on a date/time to view the file as it appeared at that time.
<tr><td>current</td><tdclass='filehistory-selected'style='white-space: nowrap;'><ahref="//upload.wikimedia.org/wikipedia/commons/8/8f/Elman_srnn.png">21:39, 31 January 2009</a></td><td><ahref="//upload.wikimedia.org/wikipedia/commons/8/8f/Elman_srnn.png"><imgalt="Thumbnail for version as of 21:39, 31 January 2009"src="//upload.wikimedia.org/wikipedia/commons/thumb/8/8f/Elman_srnn.png/108px-Elman_srnn.png"width="108"height="120"data-file-width="673"data-file-height="745"/></a></td><td>673 × 745 <spanstyle="white-space: nowrap;">(92 KB)</span></td><td>Fyedernoggersnodden</td><tddir="ltr">{{Information |Description={{en|1=The Elman Simple Recurrent Neural Network. The context layer (u1 ... ul) provides a temporal memory to the network, allowing it to "remember" previous hidden layer states. Generated in LaTeX via PGF/Tikz.}} |Source=Own </td></tr>
</table>
</div>
<h2id="filelinks">File usage</h2>
<divid='mw-imagepage-section-linkstoimage'>
<divclass="linkstoimage"id="linkstoimage">The following pages on the English Wikipedia link to this file (pages on other projects are not listed):</div>
<liid="pt-createaccount"><ahref="/w/index.php?title=Special:UserLogin&returnto=File%3AElman+srnn.png&type=signup"title="You are encouraged to create an account and log in; however, it is not mandatory">Create account</a></li><liid="pt-login"><ahref="/w/index.php?title=Special:UserLogin&returnto=File%3AElman+srnn.png"title="You're encouraged to log in; however, it's not mandatory. [o]"accesskey="o">Log in</a></li></ul>
<liid="ca-nstab-image"class="selected"><span><ahref="/wiki/File:Elman_srnn.png"title="View the file page [c]"accesskey="c">File</a></span></li>
<liid="ca-talk"class="new"><span><ahref="/w/index.php?title=File_talk:Elman_srnn.png&action=edit&redlink=1"title="Discussion about the content page [t]"accesskey="t">Talk</a></span></li>
<inputtype="search"name="search"placeholder="Search"title="Search Wikipedia [f]"accesskey="f"id="searchInput"/><inputtype="hidden"value="Special:Search"name="title"/><inputtype="submit"name="fulltext"value="Search"title="Search Wikipedia for this text"id="mw-searchButton"class="searchButton mw-fallbackSearchButton"/><inputtype="submit"name="go"value="Go"title="Go to a page with this exact name if one exists"id="searchButton"class="searchButton"/></div>
</form>
</div>
</div>
</div>
<divid="mw-panel">
<divid="p-logo"role="banner"><aclass="mw-wiki-logo"href="/wiki/Main_Page"title="Visit the main page"></a></div>
<liid="n-mainpage-description"><ahref="/wiki/Main_Page"title="Visit the main page [z]"accesskey="z">Main page</a></li>
<liid="n-contents"><ahref="/wiki/Portal:Contents"title="Guides to browsing Wikipedia">Contents</a></li>
<liid="n-featuredcontent"><ahref="/wiki/Portal:Featured_content"title="Featured content – the best of Wikipedia">Featured content</a></li>
<liid="n-currentevents"><ahref="/wiki/Portal:Current_events"title="Find background information on current events">Current events</a></li>
<liid="n-randompage"><ahref="/wiki/Special:Random"title="Load a random article [x]"accesskey="x">Random article</a></li>
<liid="n-sitesupport"><ahref="https://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_en.wikipedia.org&uselang=en"title="Support us">Donate to Wikipedia</a></li>
<liid="n-shoplink"><ahref="//shop.wikimedia.org"title="Visit the Wikimedia Shop">Wikimedia Shop</a></li>
<liid="t-whatlinkshere"><ahref="/wiki/Special:WhatLinksHere/File:Elman_srnn.png"title="List of all English Wikipedia pages containing links to this page [j]"accesskey="j">What links here</a></li>
<ahref="//www.mediawiki.org/"><imgsrc="//bits.wikimedia.org/static-1.25wmf13/resources/assets/poweredby_mediawiki_88x31.png"alt="Powered by MediaWiki"width="88"height="31"/></a>
Second, we compute the value for $f_t$, the activation of the memory cells’ forget gates at time t :
(3)$f_t =\sigma(W_f x_t + U_f h_{t-1}+ b_f)$
Given the value of the input gate activation $i_t$, the forget gate activation $f_t$ and the candidate state value $\widetilde{C_t}$, we can compute $C_t$ the memory cells’ new state at time $t$ :
(4)$C_t = i_t *\widetilde{C_t}+ f_t * C_{t-1}$
With the new state of the memory cells, we can compute the value of their output gates and, subsequently, their outputs :
\item The model we used in this tutorial is a variation of the standard LSTM model. In this variant, the activation of a cell’s output gate does not depend on the memory cell’s state $C_t$. This allows us to perform part of the computation more efficiently (see the implementation note, below, for details). This means that, in the variant we have implemented, there is no matrix $V_o$ and equation (5) is replaced by equation (7) :
(7)$o_t =\sigma(W_o x_t + U_o h_{t-1}+ b_1)$
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Implementation Note}
\begin{itemize}
\item Implementation note : In the code included this tutorial, the equations (1), (2), (3) and (7) are performed in parallel to make the computation more efficient. This is possible because none of these equations rely on a result produced by the other ones. It is achieved by concatenating the four matrices $W_*$ into a single weight matrix W and performing the same concatenation on the weight matrices $U_*$ to produce the matrix U and the bias vectors $b_*$ to produce the vector b. Then, the pre-nonlinearity activations can be computed with :
$z =\sigma(W x_t + U h_{t-1}+ b)$
The result is then sliced to obtain the pre-nonlinearity activations for i, f, $\widetilde{C_t}$, and o and the non-linearities are then applied independently for each.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{}
\begin{itemize}
\item a
\end{itemize}
\end{frame}
\begin{frame}{Conclusion}
Theano/Pylearn2/libgpuarry provide an environment for machine learning that is:
...
...
@@ -1054,7 +1128,7 @@ Deep Learning Tutorial on LSTM: \url{http://deeplearning.net/tutorial/lstm.html}
\begin{frame}{Acknowledgments}
\begin{itemize}
\item All people working or having worked at the LISA lab.
\item All Theano/Pylearn 2 users/contributors
\item All Theano users/contributors
\item Compute Canada, RQCHP, NSERC, and Canada Research Chairs for providing funds or access to compute resources.
