What hyperparameters can be adjusted during fine-tuning?

When fine-tuning a machine learning model, several hyperparameters can be adjusted to improve performance. The most common ones include the learning rate, batch size, number of training epochs, optimizer settings, and regularization parameters like dropout or weight decay. These hyperparameters control how the model updates its weights during training, how much data it processes at once, and how it avoids overfitting. For example, a learning rate that’s too high might cause unstable training, while one that’s too low could result in slow convergence. Similarly, a large batch size might speed up training but reduce generalization, and too many epochs might lead to memorizing the training data.

Specific examples help illustrate their impact. The learning rate is often the first hyperparameter developers adjust. For instance, when fine-tuning a language model like BERT, a typical starting point might be a learning rate of 2e-5, but this can vary based on the dataset size or task complexity. Batch size is another critical parameter: smaller batches (e.g., 16 or 32) are common for limited GPU memory, but larger batches (e.g., 128) might stabilize gradient estimates. The number of epochs is also task-dependent—training for 3-5 epochs might suffice for a small dataset, while larger datasets could require 10 or more. Optimizers like Adam or SGD have their own settings, such as beta values in Adam (e.g., beta1=0.9, beta2=0.999) or momentum in SGD, which influence how gradients are adjusted over time.

Beyond these basics, regularization hyperparameters play a key role. Dropout, which randomly deactivates neurons during training, can be tuned (e.g., 0.1 to 0.3) to prevent overfitting. Weight decay, which adds a penalty for large weights, might be set to 0.01 or 0.001 to balance model simplicity and performance. Some frameworks also allow layer-specific adjustments, like applying higher learning rates to newly added layers (e.g., classification heads) compared to pretrained layers. Additionally, learning rate schedules—such as linear warmup or cosine decay—can be configured to adjust the rate dynamically during training. For example, a warmup phase over the first 10% of training steps helps stabilize early updates. These hyperparameters are often interdependent, so experimenting with combinations (e.g., lower learning rate with higher weight decay) is common practice to find the optimal setup for a given task.