cnheider
/
draugr
镜像来自 https://github.com/cnheider/draugr


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201
							
<!DOCTYPE html>

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

    <title>draugr.torch_utilities.optimisation.debugging.opt_verification &#8212; Draugr 1.0.1 documentation</title>
    <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="../_static/alabaster.css" />
    <link rel="stylesheet" type="text/css" href="../_static/graphviz.css" />
    <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
    <script src="../_static/jquery.js"></script>
    <script src="../_static/underscore.js"></script>
    <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
    <script src="../_static/doctools.js"></script>
    <link rel="canonical" href="pything.github.io/draugr/generated/draugr.torch_utilities.optimisation.debugging.opt_verification.html" />
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="draugr.torch_utilities.optimisation.parameters" href="draugr.torch_utilities.optimisation.parameters.html" />
    <link rel="prev" title="draugr.torch_utilities.optimisation.debugging.layer_fetching.IntermediateLayerGetter" href="draugr.torch_utilities.optimisation.debugging.layer_fetching.IntermediateLayerGetter.html" />
   
  <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
  
  
  <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />

  </head><body>
  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          

          <div class="body" role="main">
            
  <section id="draugr-torch-utilities-optimisation-debugging-opt-verification">
<h1>draugr.torch_utilities.optimisation.debugging.opt_verification<a class="headerlink" href="#draugr-torch-utilities-optimisation-debugging-opt-verification" title="Permalink to this heading">¶</a></h1>
<p>verify value &#64; init. Verify that your value starts at the correct value value. E.g. if you initialize your final layer correctly you should measure -log(1/n_classes) on a softmax at initialization. The same default values can be derived for L2 regression, Huber losses, etc.</p>
<p>init well. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” value curves where in the first few iteration your network is basically just learning the bias.</p>
<p>overfit one batch. Overfit a single batch of only a few examples (e.g. as little as two). To do so we increase the capacity of our model (e.g. add layers or filters) and verify that we can reach the lowest achievable value (e.g. zero). I also like to visualize in the same plot both the label and the prediction and ensure that they end up aligning perfectly once we reach the minimum value. If they do not, there is a bug somewhere and we cannot continue to the next stage.</p>
<p>verify decreasing training value. At this stage you will hopefully be underfitting on your dataset because you’re working with a toy model. Try to increase its capacity just a bit. Did your training value go down as it should?</p>
<p>visualize just before the net. The unambiguously correct place to visualize your data is immediately before your y_hat = model(x) (or sess.run in tf). That is - you want to visualize exactly what goes into your network, decoding that raw tensor of data and labels into visualizations. This is the only “source of truth”. I can’t count the number of times this has saved me and revealed problems in data preprocessing and augmentation.</p>
<p>use backprop to chart dependencies. Your deep learning code will often contain complicated, vectorized, and broadcasted operations. A relatively common bug I’ve come across a few times is that people get this wrong (e.g. they use view instead of transpose/permute somewhere) and inadvertently mix information across the batch dimension. It is a depressing fact that your network will typically still train okay because it will learn to ignore data from the other examples. One way to debug this (and other related problems) is to set the value to be something trivial like the sum of all outputs of example i, run the backward pass all the way to the input, and ensure that you get a non-zero gradient only on the i-th input. The same strategy can be used to e.g. ensure that your autoregressive model at time t only depends on 1..t-1. More generally, gradients give you information about what depends on what in your network, which can be useful for debugging.</p>
<p>#TODO: NOT DONE, FINISH!</p>
<blockquote>
<div><p>Created on 07/07/2020</p>
</div></blockquote>
<p class="rubric">Functions</p>
<table class="autosummary longtable docutils align-default">
<colgroup>
<col style="width: 10%" />
<col style="width: 90%" />
</colgroup>
<tbody>
</tbody>
</table>
</section>


          </div>
          
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
            <p class="logo"><a href="../index.html">
              <img class="logo" src="../_static/valknut.svg" alt="Logo"/>
            </a></p>
<h1 class="logo"><a href="../index.html">Draugr</a></h1>


<h3>Navigation</h3>
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal" href="draugr.html">draugr</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="draugr.dist_is_editable.html">draugr.dist_is_editable</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.get_version.html">draugr.get_version</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.dlib_utilities.html">draugr.dlib_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.drawers.html">draugr.drawers</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.entry_points.html">draugr.entry_points</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.extensions.html">draugr.extensions</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.ffmpeg_utilities.html">draugr.ffmpeg_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.jax_utilities.html">draugr.jax_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.metrics.html">draugr.metrics</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.multiprocessing_utilities.html">draugr.multiprocessing_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.numpy_utilities.html">draugr.numpy_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.opencv_utilities.html">draugr.opencv_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.os_utilities.html">draugr.os_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.pandas_utilities.html">draugr.pandas_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.pygame_utilities.html">draugr.pygame_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.python_utilities.html">draugr.python_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.random_utilities.html">draugr.random_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.scipy_utilities.html">draugr.scipy_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.stopping.html">draugr.stopping</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.tensorboard_utilities.html">draugr.tensorboard_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.threading_utilities.html">draugr.threading_utilities</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="draugr.torch_utilities.html">draugr.torch_utilities</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.architectures.html">draugr.torch_utilities.architectures</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.datasets.html">draugr.torch_utilities.datasets</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.distributions.html">draugr.torch_utilities.distributions</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.evaluation.html">draugr.torch_utilities.evaluation</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.exporting.html">draugr.torch_utilities.exporting</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.generators.html">draugr.torch_utilities.generators</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.images.html">draugr.torch_utilities.images</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.opencv.html">draugr.torch_utilities.opencv</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.operations.html">draugr.torch_utilities.operations</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="draugr.torch_utilities.optimisation.html">draugr.torch_utilities.optimisation</a><ul class="current">
<li class="toctree-l4 current"><a class="reference internal" href="draugr.torch_utilities.optimisation.debugging.html">draugr.torch_utilities.optimisation.debugging</a><ul class="current">
<li class="toctree-l5"><a class="reference internal" href="draugr.torch_utilities.optimisation.debugging.gradients.html">draugr.torch_utilities.optimisation.debugging.gradients</a></li>
<li class="toctree-l5"><a class="reference internal" href="draugr.torch_utilities.optimisation.debugging.layer_fetching.html">draugr.torch_utilities.optimisation.debugging.layer_fetching</a></li>
<li class="toctree-l5 current"><a class="current reference internal" href="#">draugr.torch_utilities.optimisation.debugging.opt_verification</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="draugr.torch_utilities.optimisation.parameters.html">draugr.torch_utilities.optimisation.parameters</a></li>
<li class="toctree-l4"><a class="reference internal" href="draugr.torch_utilities.optimisation.scheduling.html">draugr.torch_utilities.optimisation.scheduling</a></li>
<li class="toctree-l4"><a class="reference internal" href="draugr.torch_utilities.optimisation.stopping.html">draugr.torch_utilities.optimisation.stopping</a></li>
<li class="toctree-l4"><a class="reference internal" href="draugr.torch_utilities.optimisation.updates.html">draugr.torch_utilities.optimisation.updates</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.persistence.html">draugr.torch_utilities.persistence</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.sessions.html">draugr.torch_utilities.sessions</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.system.html">draugr.torch_utilities.system</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.tensors.html">draugr.torch_utilities.tensors</a></li>
<li class="toctree-l3"><a class="reference internal" href="draugr.torch_utilities.writers.html">draugr.torch_utilities.writers</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="draugr.tqdm_utilities.html">draugr.tqdm_utilities</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.visualisation.html">draugr.visualisation</a></li>
<li class="toctree-l2"><a class="reference internal" href="draugr.writers.html">draugr.writers</a></li>
</ul>
</li>
</ul>
<p class="caption" role="heading"><span class="caption-text">Notes</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../getting_started.html">Getting Started</a></li>
</ul>

<div class="relations">
<h3>Related Topics</h3>
<ul>
  <li><a href="../index.html">Documentation overview</a><ul>
  <li><a href="draugr.html">draugr</a><ul>
  <li><a href="draugr.torch_utilities.html">draugr.torch_utilities</a><ul>
  <li><a href="draugr.torch_utilities.optimisation.html">draugr.torch_utilities.optimisation</a><ul>
  <li><a href="draugr.torch_utilities.optimisation.debugging.html">draugr.torch_utilities.optimisation.debugging</a><ul>
      <li>Previous: <a href="draugr.torch_utilities.optimisation.debugging.layer_fetching.IntermediateLayerGetter.html" title="previous chapter">draugr.torch_utilities.optimisation.debugging.layer_fetching.IntermediateLayerGetter</a></li>
      <li>Next: <a href="draugr.torch_utilities.optimisation.parameters.html" title="next chapter">draugr.torch_utilities.optimisation.parameters</a></li>
  </ul></li>
  </ul></li>
  </ul></li>
  </ul></li>
  </ul></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
  <h3 id="searchlabel">Quick search</h3>
    <div class="searchformwrapper">
    <form class="search" action="../search.html" method="get">
      <input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
      <input type="submit" value="Go" />
    </form>
    </div>
</div>
<script>document.getElementById('searchbox').style.display = "block"</script>


        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="footer">
      &copy;.
      
      |
      Powered by <a href="http://sphinx-doc.org/">Sphinx 5.0.2</a>
      &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.12</a>
      
      |
      <a href="../_sources/generated/draugr.torch_utilities.optimisation.debugging.opt_verification.rst.txt"
          rel="nofollow">Page source</a>
    </div>

    
  </body>
</html>