federated learning fairness adaptive direction tuning multi-objective optimization

AdaFed: Enhancing Fairness in Federated Learning via Adaptive Common Descent Direction

Mohamed Ben Hamdoune
Mohamed Ben Hamdoune

Federated learning (FL) enables collaborative model training across multiple devices while preserving data privacy. However, ensuring fairness among participating clients is a critical challenge. Shayan Mohajer Hamidi and En-Hui Yang introduce AdaFed, an approach that adaptively tunes the common descent direction to promote fairness without compromising model accuracy.

The Fairness Challenge in Federated Learning

Data heterogeneity across clients in FL can lead to unfair model performance. Some clients may experience poor accuracy compared to others. Traditional FL algorithms like FedAvg, which average local gradients, may not guarantee fairness. AdaFed addresses this by finding a common descent direction that benefits all clients while prioritizing those with larger loss values.

AdaFed: Adaptive Common Descent Direction

AdaFed finds a common descent direction that:

  1. Decreases the loss functions of all clients.
  2. Prioritizes clients with larger loss values, ensuring a higher rate of decrease.

AdaFed employs a two-phase approach:

  1. Orthogonalization: The server generates mutually orthogonal gradients spanning the same subspace as the original local gradients.
  2. Optimal Weight Calculation: The server finds optimal weights for the orthogonal gradients, minimizing the norm of their convex hull, yielding a closed-form solution for the common descent direction.

AdaFed adaptively tunes the common descent direction based on each client’s loss function, promoting fairness throughout the FL process.

Convergence Analysis and Experimental Validation

The authors prove AdaFed’s convergence to a Pareto-stationary solution under different FL setups. Experiments on datasets like CIFAR-10, CIFAR-100, FEMNIST, and Shakespeare demonstrate that AdaFed achieves higher fairness among clients while maintaining accuracy comparable to state-of-the-art fair FL methods.

Figure 1: The percentage of improved clients as a function of communication rounds for CIFAR-10.


Figure 2: The convergence of the common descent direction norm over communication rounds.


Integration with Label Noise Correction

AdaFed can be integrated with label noise correction methods like FedCorr to address mislabeled instances in real-world datasets. Experiments on the Clothing1M dataset show that combining AdaFed with FedCorr improves average accuracy while maintaining fairness among clients.

Conclusion

AdaFed presents a novel approach to address fairness in federated learning. By adaptively tuning the common descent direction based on clients’ loss values, AdaFed ensures all clients benefit from collaborative learning, with a higher rate of improvement for those with larger losses. As FL advances, techniques like AdaFed will be crucial for enabling fair and inclusive collaborative learning across diverse clients.

You may also like

See all federated learning

FedAF: Improving Federated Learning for Heterogeneous Data

A new federated learning algorithm, FedAF, addresses data heterogeneity challenges without model aggregation, leading to better accuracy and faster convergence.

Enhancing Privacy in Face Recognition Systems with Federated Learning

An in-depth look at how federated learning techniques can protect user privacy in face recognition systems while maintaining model effectiveness.

Improving One-Shot Federated Learning with Co-Boosting

A new approach called Co-Boosting enhances one-shot federated learning by simultaneously improving synthesized data and ensemble models, leading to better server model performance without additional client-side adjustments.