A fully connected layer, often called a dense layer, is a fundamental building block in neural networks where every neuron in the layer is connected to every neuron in the previous layer. This means each input feature from the previous layer is multiplied by a weight, summed with a bias term, and passed through an activation function to produce an output. Unlike convolutional or pooling layers, which focus on local patterns or spatial hierarchies, fully connected layers treat all inputs as independent features, enabling them to model complex global relationships. For example, in image classification, after convolutional layers extract edges or textures, a fully connected layer might combine these features to predict the image’s class.
Fully connected layers are commonly used in the final stages of neural networks for tasks like classification or regression. For instance, in a simple multilayer perceptron (MLP) for recognizing handwritten digits (MNIST), the input pixels are flattened into a vector and passed through one or more dense layers. Each layer transforms the data by applying weights and biases, followed by nonlinear activations like ReLU or sigmoid. In convolutional neural networks (CNNs), fully connected layers often appear after convolutional and pooling layers to convert spatial feature maps into class probabilities. For example, a CNN for image recognition might reduce a 3D tensor of features (height, width, channels) to a 1D vector via flattening, then apply dense layers to map this to class scores.
One key consideration with fully connected layers is their computational cost. Since every neuron connects to all inputs, the number of parameters grows rapidly with layer size. A layer with 1,000 inputs and 500 outputs requires 1,000×500 = 500,000 weights plus 500 biases, totaling 500,500 parameters. This can lead to overfitting, especially with limited data. To mitigate this, techniques like dropout (randomly deactivating neurons during training) or L2 regularization (penalizing large weights) are often applied. Additionally, modern architectures sometimes replace dense layers with alternatives like global average pooling to reduce parameters. Despite these challenges, fully connected layers remain essential for tasks requiring flexible, high-capacity modeling of feature interactions.