GLU Variants Improve Trasnformer

Knowledge

Gated Linear Unit (GLU)

A Gated Linear Unit (GLU) is a neural network activation layer that dynamically controls the flow of information by taking the element-wise product of two linear transformations of the same input. One transformation acts as the feature values, while the other acts as a data-dependent “gate” to emphasize or suppress specific features

The foundational mathematical formula for a GLU is defined as:

Where:

: The input vector.

and : Learnable weight matrices.

and : Learnable bias vectors.

: The element-wise (Hadamard) product.

: The sigmoid activation function, which constrains the gate values between 0 and 1.

By passing one of the pathways through the sigmoid function, the network learns exactly which features to pass to the next layer (gates closer to 1) and which to block (gates closer to 0)

GLU variants

Variants explore substituting the sigmoid function with other activation functions, such as the ReLU or tanh. This paper tests out these variants in the feedforward layers of a Transformer seq2seq model. They find that some of them (SwiGLU, GEGLU) perform better than the traditional GELU or ReLU activations.

These variants are basically just mashing together GLU with ReLU, Swish, and GELU activations.

Questions