

<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Experiment Setup and Execution &mdash; LEAF 0.1 documentation</title>
  

  
  
  
  

  

  
  
    

  

  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/customheader.css" type="text/css" />
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" /> 

  
  <script src="../_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">

    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search">
          

          
            <a href="../index.html" class="icon icon-home"> LEAF
          

          
          </a>

          
            
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <p class="caption"><span class="caption-text">Getting Started</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../install/get_leaf.html">Installation and Contribution</a></li>
</ul>
<p class="caption"><span class="caption-text">User Guide</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="index.html">General Examples</a></li>
</ul>
<p class="caption"><span class="caption-text">API Documentation</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../autodoc/modules.html">models</a></li>
</ul>
<p class="caption"><span class="caption-text">Additional Information</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../contact.html">Contact</a></li>
<li class="toctree-l1"><a class="reference internal" href="../citations.html">Citing LEAF</a></li>
</ul>

            
          
        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">LEAF</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html">Docs</a> &raquo;</li>
        
      <li>Experiment Setup and Execution</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
            <a href="../_sources/tutorials/shakespeare-md.md.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <h1> Convergence Behaviour of FederatedAveraging </h1><p>In this experiment, we reproduce the convergence analysis experiment conducted in the <a class="reference external" href="https://arxiv.org/abs/1812.01097">LEAF paper</a>. Specifically, we investigate the convergence behaviour of <code class="docutils literal notranslate"><span class="pre">FedAvg</span></code> algorithm
when varying the number of local epochs, using the LEAF framework.</p>
<p>For this example, we shall use the Shakespeare dataset to perform a character-prediction task;
that is, given a statement by a character in a play, we shall try to predict the next character in the sentence.
For this experiment, we shall train a 2-layer LSTM model with cross-entropy loss, using randomly initialized word embeddings.</p>
<div class="section" id="experiment-setup-and-execution">
<h1>Experiment Setup and Execution<a class="headerlink" href="#experiment-setup-and-execution" title="Permalink to this headline">¶</a></h1>
<div class="section" id="quickstart-script">
<h2>Quickstart script<a class="headerlink" href="#quickstart-script" title="Permalink to this headline">¶</a></h2>
<p>In the interest of ease of use, we provide a script for execution of the experiment
for different number of local epochs, which may be executed as:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>leaf/ $&gt; ./shakespeare.sh &lt;result-output-dir&gt;
</pre></div>
</div>
<p>This script will execute the instructions provided below for local epoch values of 1 and 20,
reproducibly generating the data partitions and results observed by the authors during analysis.</p>
</div>
<div class="section" id="dataset-fetching-and-pre-processing">
<h2>Dataset fetching and pre-processing<a class="headerlink" href="#dataset-fetching-and-pre-processing" title="Permalink to this headline">¶</a></h2>
<p>LEAF contains powerful scripts for fetching and conversion of data into JSON format for easy utilization.
Additionally, these scripts are also capable of subsampling from the dataset, and splitting the dataset
into training and testing sets.</p>
<p>For our experiment, as a first step, we shall use 5% of the dataset in an 90-10 train/test split,
and we shall discard all users with less than 64 samples. The following command shows
how this can be accomplished (the <code class="docutils literal notranslate"><span class="pre">--spltseed</span></code> and <code class="docutils literal notranslate"><span class="pre">--smplseed</span></code> flags in this case is to enable reproducible generation of the dataset)</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>leaf/data/shakespeare/ $&gt; ./preprocess.sh --sf <span class="m">0</span>.05 -t sample --tf <span class="m">0</span>.9 -k <span class="m">64</span> --smplseed <span class="m">1550262838</span> --spltseed <span class="m">1550262839</span>
</pre></div>
</div>
<p>After running this script, the <code class="docutils literal notranslate"><span class="pre">data/shakespeare/data</span></code> directory should contain <code class="docutils literal notranslate"><span class="pre">train/</span></code> and <code class="docutils literal notranslate"><span class="pre">test/</span></code> directories.</p>
</div>
<div class="section" id="model-execution">
<h2>Model Execution<a class="headerlink" href="#model-execution" title="Permalink to this headline">¶</a></h2>
<p>Now that we have our data, we can execute our model! For this experiment, the model file is stored
at <code class="docutils literal notranslate"><span class="pre">models/shakespeare/stacked_lstm.py</span></code>. In order train this model using <code class="docutils literal notranslate"><span class="pre">FedAvg</span></code> with 10 clients per round
and 1 local epoch per round, we execute the following command:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>leaf/models $&gt; python main.py -dataset shakespeare -model stacked_lstm --num-rounds <span class="m">81</span> --clients-per-round <span class="m">10</span> --num_epochs <span class="m">1</span> -lr <span class="m">0</span>.8
</pre></div>
</div>
</div>
<div class="section" id="metrics-collection">
<h2>Metrics Collection<a class="headerlink" href="#metrics-collection" title="Permalink to this headline">¶</a></h2>
<p>Executing the above command will write out system and statistical metrics to <code class="docutils literal notranslate"><span class="pre">leaf/models/metrics/stat_metrics.csv</span></code> and <code class="docutils literal notranslate"><span class="pre">leaf/models/metrics/sys_metrics.csv</span></code> - since these are overwritten for every run, we <strong>highly recommend</strong> storing the generated metrics files at a different location.</p>
<p>To experiment with a different number of local epochs, re-run the main model script with a different <code class="docutils literal notranslate"><span class="pre">--num_epochs</span></code> flag. The plots shown below can be generated using <code class="docutils literal notranslate"><span class="pre">plots.py</span></code> file in the repo root.</p>
</div>
</div>
<div class="section" id="results-and-analysis">
<h1>Results and Analysis<a class="headerlink" href="#results-and-analysis" title="Permalink to this headline">¶</a></h1>
<p>From the generated data, we computed the aggregate weighted training loss and training accuracy
(note that weighting is important to correct for label imbalance). Also, we note that <code class="docutils literal notranslate"><span class="pre">FedAvg</span></code>
method diverges as the number of local epochs increases.</p>
<div style="text-align:center" markdown="1"><p><img alt="Weighted Test Accuracy on Shakespeare dataset using FedAvg" src="../_images/shake_small_weighted_test_acc.png" />
<img alt="Weighted Training Loss on Shakespeare dataset using FedAvg" src="../_images/shake_small_weighted_training_loss.png" /></p>
</div></div>
<div class="section" id="more-information">
<h1>More Information<a class="headerlink" href="#more-information" title="Permalink to this headline">¶</a></h1>
<p>More information about the framework, challenges and experiments can be found in the <a class="reference external" href="https://arxiv.org/abs/1812.01097">LEAF paper</a>.</p>
</div>


           </div>
           
          </div>
          <footer>
  

  <hr/>

  <div role="contentinfo">
    <p>
        &copy; Copyright 2019, Talwalkar Lab

    </p>
  </div>
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  


  

    
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script type="text/javascript" src="../_static/jquery.js"></script>
        <script type="text/javascript" src="../_static/underscore.js"></script>
        <script type="text/javascript" src="../_static/doctools.js"></script>
    

  

  <script type="text/javascript" src="../_static/js/theme.js"></script>

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script> 

</body>
</html>