flyvec/docs/index.html

---

title: FlyVec


keywords: fastai
sidebar: home_sidebar

summary: "Sparse Binary Word Embeddings Inspired by the Fruit Fly Brain"
description: "Sparse Binary Word Embeddings Inspired by the Fruit Fly Brain"
nb_path: "nbs/index.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/index.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Code based on the ICLR 2021 paper <a href="https://arxiv.org/abs/2101.06887">Can a Fruit Fly Learn Word Embeddings?</a>.</p>
<p>In this work we use a well-established neurobiological network motif from the mushroom body of the fruit fly brain to learn sparse binary word embeddings from raw unstructured text. This package allows the user to access pre-trained word embeddings and generate sparse binary hash codes for individual words.</p>
<p>Interactive demos of the learned concepts available at <a href="http://flyvec.org">flyvec.org</a>.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="How-to-use">How to use<a class="anchor-link" href="#How-to-use"> </a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Install-from-Pip-(recommended)">Install from Pip (recommended)<a class="anchor-link" href="#Install-from-Pip-(recommended)"> </a></h3><p><code>pip install flyvec</code></p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Installing-from-Source">Installing from Source<a class="anchor-link" href="#Installing-from-Source"> </a></h3><p>After cloning:</p>

<pre><code>conda env create -f environment-dev.yml
conda activate flyvec
pip install -e .</code></pre>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Basic-Usage">Basic Usage<a class="anchor-link" href="#Basic-Usage"> </a></h3><p>An example below illustrates how one can access the binary word embedding for individual tokens for a default hash length <code>k=50</code>.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">flyvec</span> <span class="kn">import</span> <span class="n">FlyVec</span>

<span class="n">model</span> <span class="o">=</span> <span class="n">FlyVec</span><span class="o">.</span><span class="n">load</span><span class="p">()</span>
<span class="n">embed_info</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">get_sparse_embedding</span><span class="p">(</span><span class="s2">&quot;market&quot;</span><span class="p">);</span> <span class="n">embed_info</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;token&#39;: &#39;market&#39;,
 &#39;id&#39;: 1180,
 &#39;embedding&#39;: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
        1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1,
        0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0,
        1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
        1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0], dtype=int8)}</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Changing-the-Hash-Length">Changing the Hash Length<a class="anchor-link" href="#Changing-the-Hash-Length"> </a></h3><p>The user can obtain the FlyVec embeddings for any hash length using the following example.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">small_embed</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">get_sparse_embedding</span><span class="p">(</span><span class="s2">&quot;market&quot;</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">small_embed</span><span class="p">[</span><span class="s1">&#39;embedding&#39;</span><span class="p">])</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>4</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Handling-&quot;unknown&quot;-tokens">Handling "unknown" tokens<a class="anchor-link" href="#Handling-&quot;unknown&quot;-tokens"> </a></h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>FlyVec uses a simple, word-based tokenizer. The provided model uses a vocabulary with about 20,000 words, all lower-cased, with special tokens for numbers (<code>&lt;NUM&gt;</code>) and unknown words (<code>&lt;UNK&gt;</code>). Unknown tokens have the token id of <code>0</code>, which can be used to filter unknown tokens.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">unk_embed</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">get_sparse_embedding</span><span class="p">(</span><span class="s2">&quot;DefNotAWord&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">unk_embed</span><span class="p">[</span><span class="s1">&#39;id&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;I AM THE UNKNOWN TOKEN DON&#39;T USE ME FOR ANYTHING IMPORTANT&quot;</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>I AM THE UNKNOWN TOKEN DON&#39;T USE ME FOR ANYTHING IMPORTANT
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Batch-generating-word-embeddings">Batch generating word embeddings<a class="anchor-link" href="#Batch-generating-word-embeddings"> </a></h3><p>Embeddings for individual words in a sentence can be obtained using this snippet.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">sentence</span> <span class="o">=</span> <span class="s2">&quot;Supreme Court dismissed the criminal charges.&quot;</span>
<span class="n">tokens</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">tokenize</span><span class="p">(</span><span class="n">sentence</span><span class="p">)</span>
<span class="n">embedding_info</span> <span class="o">=</span> <span class="p">[</span><span class="n">model</span><span class="o">.</span><span class="n">get_sparse_embedding</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">tokens</span><span class="p">]</span>
<span class="n">embeddings</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;embedding&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">embedding_info</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;TOKENS: &quot;</span><span class="p">,</span> <span class="p">[</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;token&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">embedding_info</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;EMBEDDINGS: &quot;</span><span class="p">,</span> <span class="n">embeddings</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>TOKENS:  [&#39;supreme&#39;, &#39;court&#39;, &#39;dismissed&#39;, &#39;the&#39;, &#39;criminal&#39;, &#39;charges&#39;]
EMBEDDINGS:  [[0 1 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 1 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 1 0]
 [0 0 0 ... 0 1 0]]
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="FlyVec-vocabulary">FlyVec vocabulary<a class="anchor-link" href="#FlyVec-vocabulary"> </a></h3><p>The vocabulary under the hood uses the gensim <code>Dictionary</code> and can be accessed by either IDs (<code>int</code>s) or Tokens (<code>str</code>s).</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">token_vocab</span><span class="p">[:</span><span class="mi">5</span><span class="p">])</span>

<span class="c1"># The IDs that correspond to those tokens</span>
<span class="nb">print</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">vocab</span><span class="p">[:</span><span class="mi">5</span><span class="p">])</span>

<span class="c1"># The dictionary object itself</span>
<span class="n">model</span><span class="o">.</span><span class="n">dictionary</span><span class="p">;</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">
<pre>[&#39;properties&#39;, &#39;a&#39;, &#39;among&#39;, &#39;and&#39;, &#39;any&#39;]
[2, 3, 4, 5, 6]
</pre>
</div>
</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Training">Training<a class="anchor-link" href="#Training"> </a></h1><p>Please note that the training code is included, though code for processing the inputs.</p>
<p><strong>Prerequisites</strong></p>
<p>You need a python environment with <code>numpy</code> installed, a system that supports CUDA, <code>nvcc</code>, and <code>g++</code>.</p>
<p><strong>Building the Source Files</strong></p>
<p><code>flyvec_compile</code></p>
<p>(Or, if using from source, you can also run <code>make training</code>)</p>
<p>Note that you will see some warnings. This is expected.</p>
<p><strong>Training</strong></p>
<p><code>flyvec_train path/to/encodings.npy path/to/offsets.npy -o save/checkpoints/in/this/directory</code></p>
<p><strong>Description of Inputs</strong></p>
<ul>
<li><code>encodings.npy</code> -- An <code>np.int32</code> array representing the tokenized vocabulary-IDs of the input corpus, of shape <code>(N,)</code> where <code>N</code> is the number of tokens in the corpus</li>
<li><code>offsets.npy</code> -- An <code>np.uint64</code> array of shape <code>(C,)</code> where <code>C</code> is the number of chunks in the corpus. Each each value represents the index that starts a new chunk within <code>encodings.npy</code>. 
  (Chunks can be thought of as sentences or paragraphs within the corpus; boundaries over which the sliding window does not cross.)</li>
</ul>
<p><strong>Description of Outputs</strong></p>
<ul>
<li><code>model_X.npy</code> -- Stores checkpoints after every epoch within the specified output directory</li>
</ul>
<p>See <code>flyvec_train --help</code> for more options.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Citation">Citation<a class="anchor-link" href="#Citation"> </a></h1><p>If you use this in your work, please cite:</p>

<pre><code>@article{liang2021can,
  title={Can a Fruit Fly Learn Word Embeddings?},
  author={Liang, Yuchen and Ryali, Chaitanya K and Hoover, Benjamin and Grinberg, Leopold and Navlakha, Saket and Zaki, Mohammed J and Krotov, Dmitry},
  journal={arXiv preprint arXiv:2101.06887},
  year={2021}
  url={https://arxiv.org/abs/2101.06887}
}</code></pre>

</div>
</div>
</div>
</div>