User Guide
Deeplearning4j Training UI
Welcome!
Welcome to the Deeplearning4j Training UI! DL4J provides the HistogramIterationListener as a method of visualizing in your browser (in real time) the progress of network training. Here’s an excellent web page by Andrej Karpathy about visualizing neural net training. It is worth reading that page first.
If there's any confusion, please ask our engineers in Gitter.
Overview Tab
Score vs Iteration: Snapshot
- Score vs. iteration should (overall) go down over time.
- If the score increases consistently, your learning rate is likely set too high. Try reducing it until scores become more stable.
- Increasing scores can also be indicative of other network issues, such as incorrect data normalization.
- If the score is flat or decreases very slowly (over a few hundred iteratons) (a) your learning rate may be too low, or (b) you might be having diffulties with optimization. In the latter case, if you are using the SGD updater, try a different updater such as momentum, RMSProp or Adagrad.
- Note that data that isn’t shuffled (i.e., each minibatch contains only one class, for classification) can result in very rough or abnormal-looking score vs. iteration graphs.
- Some noise in this line chart is expected (i.e., the line will go up and down within a small range). However, if the scores vary quite significantly between runs variation is very large, this can be a problem.
Model Performance
The table contains basic model performance metrics.
Model Type - MultiLayerNetwork or...
nLayers - Number of layers.
nParams - Number of parameters.
Total Runtime - Explain importance
Last Update - Explain importance
Total Parameter Updates - Explain importance
Updates Per Second - Explain importance
Examples Per Second - Explain importance
Ratio of Updates to Parameters: All Layers
Need Explanation Here.
Variances
Need Explanation Here.
Model Tab
Layer Visualization UI
The layer visualization UI renders network structure dynamically. Users can inspect the and node layer parameters by clicking on the various elements of the GUI to see general information about layers/nodes, overall network information such as performance.
Layer Information
The table contains basic layer information.
Name - MultiLayerNetwork or...
Type - Number of layers.
Inputs - Number of parameters.
Outputs - Explain importance
Activation Function - Explain importance
Learning Rate - Explain importance
Mean Magnitudes
- At the right is a line chart of the mean magnitude of both the parameters and the updates in the neural network.
- “Mean magnitude” = the average of the absolute value of the parameters or updates.
- For tuning the learning rate, the ratio of parameters to updates for a layer should be somewhere in the order of 1000:1 - but note that is a rough guide only, and may not be appropriate for all networks. It’s often a good starting point, however.
- If the ratio diverges significantly from this, your parameters may be too unstable to learn useful features, or may change too slowly to learn useful features
- To change this ratio, adjust your learning rate (or sometimes, parameter initialization). In some networks, you may need to set the learning rate differently for different layers.
- Keep an eye out for unusually large spikes in the updates: this may indicate exploding gradients (see discussion in the “histogram of gradients” section above)
Activations
Need Explanation Here.
Parameters Histogram
- At the top right is a histogram of the weights in the neural network (at the last iteration), split up by layer and the type of parameter. For example, “param_0_W” refers to the weight parameters for the first layer.
- For weights, these histograms should have an approximately Gaussian (normal) distribution, after some time.
- For biases, these histograms will generally start at 0, and will usually end up being approximately Gaussian.
- One exception to this is for LSTM recurrent neural network layers: by default, the biases for one gate (the forget gate) are set to 1.0 (by default, though this is configurable), to help in learning dependencies across long time periods. This results in the bias graphs initially having many biases around 0.0, with another set of biases around 1.0
- Keep an eye out for parameters that are diverging to +/- infinity: this may be due to too high a learning rate, or insufficient regularization (try adding some L2 regularization to your network).
- Keep an eye out for biases that become very large. This can sometimes occur in the output layer for classification, if the distribution of classes is very imbalanced
Updates Histogram
- At the bottom left is the histogram of updates for the neural network (at the last iteration), also split up by layer and type of parameter.
- Note that these are the updates - i.e., the gradients after appling learning rate, momentum, regularization etc.
- As with the parameter graphs, these should have an approximately Gaussian (normal) distribution.
- Keep an eye out for very large values: this can indicate exploding gradients in your network.
- Exploding gradients are problematic as they can ‘mess up’ the parameters of your network.
- In this case, it may indicate a weight initialization, learning rate or input/labels data normalization issue.
- In the case of recurrent neural networks, adding some gradient normalization or gradient clipping can frequently help.
System Tab
JVM Memory Utilization
Need Explanation Here.
Off-Heap Memory Utilization
Need Explanation Here.
Hardware Information
The table contains basic hardware metrics.
JVM Current Memory - What this means.
JVM Max Memory - What this means.
Off-Heap Current Memory - What this means.
Off-Heap Current Memory - What this means.
JVM Available Processors - What this means.
Number Compute Devices - What this means.
Software Information
The table contains basic software information.
OS - What this means.
Host Name - What this means.
OS Architecture - What this means.
JVM Name - What this means.
JVM Version - What this means.
ND4J Backend - What this means.
ND4J Datatype - What this means.
GPU Specific Graph?
Need Explanation Here.
GPU Specific Table?
Need Explanation Here.
Table of Contents
- Overview
- Snapshot of your model performance.
- Model
- Layer by layer inspection tool.
- System
- Memory utilization dashboard as well as system configurations across multiple machines.
- Language
- Switch between English, Japanese, Chinese, Korean, Ukranian and Russian.