Residual neural network explained
A residual neural network (also referred to as a residual network or ResNet)[1] is a deep learning architecture in which the layers learn residual functions with reference to the layer inputs. It was developed in 2015 for image recognition, and won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year.[2] [3]
As a point of terminology, "residual connection" refers to the specific architectural motif of, where
is an arbitrary neural network module. The motif had been used previously (see §History for details). However, the publication of ResNet made it widely popular for
feedforward networks, appearing in neural networks that are seemingly unrelated to ResNet.
The residual connection stabilizes the training and convergence of deep neural networks with hundreds of layers, and is a common motif in deep neural networks, such as transformer models (e.g., BERT, and GPT models such as ChatGPT), the AlphaGo Zero system, the AlphaStar system, and the AlphaFold system.
Mathematics
Residual connection
In a multilayer neural network model, consider a subnetwork with a certain number of stacked layers (e.g., 2 or 3). Denote the underlying function performed by this subnetwork as
, where
is the input to the subnetwork. Residual learning re-parameterizes this subnetwork and lets the parameter layers represent a "residual function"
. The output
of this subnetwork is then represented as:
The operation of "
" is implemented via a "skip connection" that performs an identity mapping to connect the input of the subnetwork with its output. This connection is referred to as a "residual connection" in later work. The function
is often represented by matrix multiplication interlaced with
activation functions and normalization operations (e.g.,
batch normalization or layer normalization). As a whole, one of these subnetworks is referred to as a "residual block". A deep residual network is constructed by simply stacking these blocks.
Long short-term memory (LSTM) has a memory mechanism that serves as a residual connection.[4] In an LSTM without a forget gate, an input
is processed by a function
and added to a memory cell
, resulting in
. An LSTM with a forget gate essentially functions as a
highway network.
To stabilize the variance of the layers' inputs, it is recommended to replace the residual connections
with
, where
is the total number of residual layers.
[5] Projection connection
If the function
is of type
where
, then
is undefined. To handle this special case, a projection connection is used:
where
is typically a linear projection, defined by
where
is a
matrix. The matrix is trained via
backpropagation, as is any other parameter of the model.
Signal propagation
The introduction of identity mappings facilitates signal propagation in both forward and backward paths.[6]
Forward propagation
If the output of the
-th residual block is the input to the
-th residual block (assuming no activation function between blocks), then the
-th input is:
Applying this formulation recursively, e.g.:
\begin{align}x\ell+2&=F(x\ell+1)+x\ell+1\\
&=F(x\ell+1)+F(x\ell)+x\ell\end{align}
yields the general relationship:
where is the index of a residual block and is the index of some earlier block. This formulation suggests that there is always a signal that is directly sent from a shallower block to a deeper block .
Backward propagation
with respect to some residual block input
. Using the equation above from forward propagation for a later residual block
:
\begin{align}
x\ell}
&=
xL}
\\
&=
xL}\left(1+
F(xi)\right)\\
&=
xL}+
xL}
F(xi)
\end{align}
This formulation suggests that the gradient computation of a shallower layer, , always has a later term that is directly added. Even if the gradients of the
terms are small, the total gradient
resists vanishing due to the added term
.
Variants of residual blocks
Basic block
A basic block is the simplest building block studied in the original ResNet. This block consists of two sequential 3x3 convolutional layers and a residual connection. The input and output dimensions of both layers are equal.
Bottleneck block
A bottleneck block consists of three sequential convolutional layers and a residual connection. The first layer in this block is a 1x1 convolution for dimension reduction (e.g., to 1/2 of the input dimension); the second layer performs a 3x3 convolution; the last layer is another 1x1 convolution for dimension restoration. The models of ResNet-50, ResNet-101, and ResNet-152 are all based on bottleneck blocks.
Pre-activation block
The pre-activation residual block applies activation functions before applying the residual function
. Formally, the computation of a pre-activation residual block can be written as:
x\ell+1=F(\phi(x\ell))+x\ell
where
can be any activation (e.g.
ReLU) or normalization (e.g. LayerNorm) operation. This design reduces the number of non-identity mappings between residual blocks. This design was used to train models with 200 to over 1000 layers.
Since GPT-2, transformer blocks have been mostly implemented as pre-activation blocks. This is often referred to as "pre-normalization" in the literature of transformer models.[7]
Applications
Originally, ResNet was designed for computer vision.[8] All transformer architectures include residual connections. Indeed, very deep transformers cannot be trained without them.[9]
The original ResNet paper made no claim on being inspired by biological systems. However, later research has related ResNet to biologically-plausible algorithms.[10] [11]
A study published in Science in 2023[12] disclosed the complete connectome of an insect brain (specifically that of a fruit fly larva). This study discovered "multilayer shortcuts" that resemble the skip connections in artificial neural networks, including ResNets.
History
Previous work
Residual connections were noticed in neuroanatomy, such as Lorente de No (1938).[13] McCulloch and Pitts (1943) proposed artificial neural networks and considered those with residual connections.[14]
In 1961, Frank Rosenblatt described a three-layer multilayer perceptron (MLP) model with skip connections.[15] The model was referred to as a "cross-coupled system", and the skip connections were forms of cross-coupled connections.
During the late 1980s, "skip-layer" connections were sometimes used in neural networks. Examples include:[16] [17] Lang and Witbrock (1988)[18] trained a fully connected feedforward network where each layer skip-connects to all subsequent layers, like the later DenseNet (2016). In this work, the residual connection was the form where
is a randomly-initialized projection connection. They termed it a "short-cut connection".
Degradation problem
Sepp Hochreiter discovered the vanishing gradient problem in 1991[19] and argued that it explained why the then-prevalent forms of recurrent neural networks did not work for long sequences. He and Schmidhuber later designed the LSTM architecture to solve this problem,[20] which has a "cell state"
that can function as a generalized residual connection. The
highway network (2015)
[21] [22] applied the idea of an LSTM
unfolded in time to
feedforward neural networks, resulting in the highway network. ResNet is equivalent to an open-gated highway network.During the early days of deep learning, there were attempts to train increasingly deep models. Notable examples included the
AlexNet (2012), which had 8 layers, and the
VGG-19 (2014), which had 19 layers.
[23] However, stacking too many layers led to a steep reduction in
training accuracy,
[24] known as the "degradation" problem. In theory, adding additional layers to deepen a network should not result in a higher training
loss, but this is what happened with
VGGNet. If the extra layers can be set as
identity mappings, however, then the deeper network would represent the same function as its shallower counterpart. There is some evidence that the optimizer is not able to approach identity mappings for the parameterized layers, and the benefit of residual connections was to allow identity mappings by default.
[25] In 2014, the state of the art was training deep neural networks with 20 to 30 layers.[26] The research team for ResNet attempted to train deeper ones by empirically testing various methods for training deeper networks, until they came upon the ResNet architecture.[27]
Subsequent work
DenseNet (2016)[28] connects the output of each layer to the input to each subsequent layer:
x\ell+1=F(x1,x2,...,x\ell-1,x\ell)
Stochastic depth[29] is a regularization method that randomly drops a subset of layers and lets the signal propagate through the identity skip connections. Also known as DropPath, this regularizes training for deep models, such as vision transformers.[30]
ResNeXt (2017) combines the Inception module with ResNet.[31] [32]
Squeeze-and-Excitation Networks (2018) added squeeze-and-excitation (SE) modules to ResNet.[33] An SE module is applied after a convolution, and takes a tensor of shape
(height, width, channels) as input. Each channel is averaged, resulting in a vector of shape
. This is then passed through a
multilayer perceptron (with an architecture such as
linear-ReLU-linear-sigmoid) before it is multiplied with the original tensor.
References
- He. Kaiming. Zhang. Xiangyu. Ren. Shaoqing. Sun. Jian. 10 Dec 2015. Deep Residual Learning for Image Recognition. 1512.03385.
- Web site: ILSVRC2015 Results . image-net.org.
- Deng . Jia . Dong . Wei . Socher . Richard . Li . Li-Jia . Li . Kai . Fei-Fei . Li . 2009 . ImageNet: A large-scale hierarchical image database . CVPR.
- Sepp Hochreiter . Sepp Hochreiter . Jürgen Schmidhuber . Jürgen Schmidhuber . 1997 . Long short-term memory . . 9 . 8 . 1735–1780 . 10.1162/neco.1997.9.8.1735 . 9377276 . 1915014.
- Hanin . Boris . Rolnick . David . 2018 . How to Start Training: The Effect of Initialization and Architecture . Advances in Neural Information Processing Systems . Curran Associates, Inc. . 31 . 1803.01719.
- 1603.05027 . Identity Mappings in Deep Residual Networks . He . Kaiming . Zhang . Xiangyu . Ren . Shaoqing . Sun . Jian . 2015. cs.CV .
- Web site: Language models are unsupervised multitask learners . Radford . Alec . Wu . Jeffrey . Child . Rewon . Luan . David . Amodei . Dario . Sutskever . Ilya . 14 February 2019 . 19 December 2020 . 6 February 2021 . https://web.archive.org/web/20210206183945/https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf . live.
- 1602.07261 . cs.CV . Christian . Szegedy . Sergey . Ioffe . Inception-v4, Inception-ResNet and the impact of residual connections on learning . Vincent . Vanhoucke . Alex . Alemi . 2016.
- 2103.03404 . cs.LG . Yihe . Dong . Jean-Baptiste . Cordonnier . Attention is not all you need: pure attention loses rank doubly exponentially with depth . Andreas . Loukas . 2021.
- Liao . Qianli . Poggio . Tomaso . 2016 . Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex . 1604.03640.
- Xiao . Will . Chen . Honglin . Liao . Qianli . Poggio . Tomaso . 2018 . Biologically-Plausible Learning Algorithms Can Scale to Large Datasets . 1811.03567.
- Winding . Michael . Pedigo . Benjamin . Barnes . Christopher . Patsolic . Heather . Park . Youngser . Kazimiers . Tom . Fushiki . Akira . Andrade . Ingrid . Khandelwal . Avinash . Valdes-Aleman . Javier . Li . Feng . Randel . Nadine . Barsotti . Elizabeth . Correia . Ana . Fetter . Fetter . 10 Mar 2023 . The connectome of an insect brain . Science . 379 . 6636 . eadd9330 . 10.1101/2022.11.28.516756v1 . 10.1126/science.add9330 . 7614541 . 36893230 . 254070919 . Hartenstein . Volker . Priebe . Carey . Vogelstein . Joshua . Cardona . Albert . Zlatic . Marta.
- De N . Rafael Lorente . 1938-05-01 . Analysis of the Activity of the Chains of Internuncial Neurons . Journal of Neurophysiology . en . 1 . 3 . 207–244 . 10.1152/jn.1938.1.3.207 . 0022-3077.
- McCulloch . Warren S. . Pitts . Walter . 1943-12-01 . A logical calculus of the ideas immanent in nervous activity . The Bulletin of Mathematical Biophysics . en . 5 . 4 . 115–133 . 10.1007/BF02478259 . 1522-9602.
- Book: Rosenblatt
, Frank
. 1961 . Principles of neurodynamics. perceptrons and the theory of brain mechanisms .
- Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning internal representations by error propagation", Parallel Distributed Processing. Vol. 1. 1986.
- Book: Venables . W. N. . Modern Applied Statistics with S-Plus . Ripley . Brain D. . 1994 . Springer . 9783540943501 . 261–262.
- Lang . Kevin. Witbrock . Michael. 1988 . Learning to tell two spirals apart . Proceedings of the 1988 Connectionist Models Summer School . 52–59.
- Untersuchungen zu dynamischen neuronalen Netzen . diploma . Sepp . Hochreiter . Technical University Munich, Institute of Computer Science, advisor: J. Schmidhuber . 1991.
- Felix A. Gers . Jürgen Schmidhuber . Fred Cummins . 2000 . Learning to Forget: Continual Prediction with LSTM . . 12 . 10 . 2451–2471 . 10.1.1.55.5709 . 10.1162/089976600300015015 . 11032042 . 11598600.
- 1505.00387 . cs.LG . Rupesh Kumar . Srivastava . Klaus . Greff . Highway Networks . 3 May 2015 . Schmidhuber . Jürgen.
- 1507.06228 . cs.LG . Rupesh Kumar . Srivastava . Klaus . Greff . Training Very Deep Networks . 22 July 2015 . Schmidhuber . Jürgen.
- Simonyan . Karen . Very Deep Convolutional Networks for Large-Scale Image Recognition . 2015-04-10 . 1409.1556 . Zisserman . Andrew. cs.CV .
- 1502.01852 . cs.CV . Kaiming . He . Xiangyu . Zhang . Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification . Ren . Shaoqing . Sun . Jian . 2016.
- Book: He . Kaiming . Identity Mappings in Deep Residual Networks . 2016 . Computer Vision – ECCV 2016 . 9908 . 630–645 . Leibe . Bastian . http://link.springer.com/10.1007/978-3-319-46493-0_38 . 2024-09-19 . Cham . Springer International Publishing . en . 10.1007/978-3-319-46493-0_38 . 978-3-319-46492-3 . Zhang . Xiangyu . Ren . Shaoqing . Sun . Jian . Matas . Jiri . Sebe . Nicu . Welling . Max.
- Simonyan . Karen . Very Deep Convolutional Networks for Large-Scale Image Recognition . 2015-04-10 . 1409.1556 . Zisserman . Andrew. cs.CV .
- Web site: Linn . Allison . 2015-12-10 . Microsoft researchers win ImageNet computer vision challenge . 2024-06-29 . The AI Blog . en-US.
- Huang. Gao. Liu. Zhuang. van der Maaten. Laurens. Weinberger. Kilian. 2016. Densely Connected Convolutional Networks. 1608.06993.
- Huang. Gao. Sun. Yu. Liu. Zhuang. Weinberger. Kilian. 2016. Deep Networks with Stochastic Depth. 1603.09382.
- Lee . Youngwan . Kim . Jonghee . Willette . Jeffrey . Hwang . Sung Ju . 2022 . MPViT: Multi-Path Vision Transformer for Dense Prediction . en . 7287–7296. 2112.11010 .
- Xie . Saining . Girshick . Ross . Dollar . Piotr . Tu . Zhuowen . He . Kaiming . 2017 . Aggregated Residual Transformations for Deep Neural Networks . 1492–1500. 1611.05431 .
- Book: Zhang . Aston . Dive into deep learning . Lipton . Zachary . Li . Mu . Smola . Alexander J. . 2024 . Cambridge University Press . 978-1-009-38943-3 . Cambridge New York Port Melbourne New Delhi Singapore . 8.6. Residual Networks (ResNet) and ResNeXt . https://d2l.ai/chapter_convolutional-modern/resnet.html.
- Hu . Jie . Shen . Li . Sun . Gang . 2018 . Squeeze-and-Excitation Networks . 7132–7141.