Gated recurrent unit - Alchetron, The Free Social Encyclopedia

Gated recurrent units (GRUs) are a gating mechanism in recurrent neural networks, introduced in 2014. Their performance on polyphonic music modeling and speech signal modeling was found to be similar to that of long short-term memory. They have fewer parameters than LSTM, as they lack an output gate.

Architecture

∘ denotes the Hadamard product. h 0 = 0 .

z t = σ g ( W z x t + U z h t − 1 + b z ) r t = σ g ( W r x t + U r h t − 1 + b r ) h t = z t ∘ h t − 1 + ( 1 − z t ) ∘ σ h ( W h x t + U h ( r t ∘ h t − 1 ) + b h )

Variables

x t : input vector

h t : output vector

z t : update gate vector

r t : reset gate vector

W , U and b : parameter matrices and vector

Activation functions

σ g : The original is a sigmoid function.

σ h : The original is a hyperbolic tangent.

References

Gated recurrent unit Wikipedia

(Text) CC BY-SA