Background
Federated Learning (FL) has emerged as an increasingly mainstream privacy-preserving framework to decentralize Machine Learning (ML) and Data Science (DS) workloads in contexts where there are barriers to centralizing data. Those barriers can vary in nature, but they tend to fall within one or multiple of these categories:
- Regulatory Barriers: Privacy regulations like GDPR, HIPAA, and others are increasingly common and are pushing industries to find compliant ways to perform data science tasks without exposing sensitive data.
- Confidentiality: Sensitive data like Personally Identifiable Information (PII) comes with significant liability and many companies don’t want to share it, or have it shared with them.
- Commercial Barriers: Commercial transactions involving data require a safe sandbox for the buyer to “test drive” the data against their models without exposing the full value of the data. See e.g. our work on data evaluation.
In a recent paper, Pascal Riedel, Data Scientist at the University of Ulm, and his co-authors, leveraged the integrate.ai FL platform as part of their solution to solve the data confidentiality problem in the energy industry, specifically in smart grid management, to help predict power input and demand, and make grid operations more efficient. Individual power input and usage data is essential to this task, but this sensitive data comes from individual households and is too sensitive to be easily centralized. We had the pleasure of discussing this topic with Pascal in this blog post.
[Frédéric Ratle] Pascal, thank you for joining us! Can you tell us a little bit about smart grid power management?
[Pascal Riedel] "Smart grids need to be managed by Distribution System Operators (DSOs) in a cost-effective way, while ensuring a reliable and stable power output to avoid shortages, and even blackouts, due to peak times (which could happen, for example, while charging many EVs in the same street). It is essential to track feed-in power as well as residual loads in order to assess how much energy is required from the grid, and how much is fed back."
[FR] You’ve focused on low-voltage power forecasting - can you characterize the prediction problem you are solving - the nature of the data, the main variables?
[PR] "The prediction problem can be stated as “How much energy is likely to be fed into the grid in the next N days”. This helps DSOs optimize capacity planning of the grid. In turn, this makes invoicing and accounting easier for the grid operator. The data we have looked at consists of three years of tabular time series data from smart meter gateways in households equipped with photovoltaic systems.Specifically, we tried to predict feed-in power as a function of Residual Load and Global Horizontal Irradiance (GHI, a meteorological feature). This data is challenging to model due to its heterogeneous nature across households."
[FR] What are some privacy and regulatory challenges that data scientists need to be aware of in that space?
[PR] "Power consumption and residual load data is reflective of the behaviour patterns of the household members. This entails two challenges: compliance with GDPR when it comes to handling data pertaining to individual households, and in-transit data protection - how is the integrity of the data ensured while being communicated between houses and the server. Balancing these privacy requirements with model performance has been a key question in our investigation."
[FR] How would you summarize the core findings of your research?
[PR] "At a high-level, FL has proven to be an effective tool for feed-in power forecasting, balancing privacy and model accuracy in a controllable way. Indeed, the use of differential privacy entails a natural tradeoff between the level of noise injected in the data vs the model performance.
Specifically, we have found that the use of an LSTM [Long Short-Term Memory] and a GRU [Gated Recurrent Unit] achieves a high prediction accuracy (up to 97.7%) while preserving a high level of data privacy
Finally, advanced aggregation strategies (such FedAdam and FedYogi) can help stabilize model convergence in FL systems with non-uniform time series data distributions."
[FR] Do you think these findings would generalize to real-world smart grids? What do you see as the main obstacles in deploying FL at scale?
[PR] "Our proposed federated forecasting method combined with differential privacy demonstrates strong predictive accuracy on real-world feed-in power data, making it highly applicable for DSOs managing small-scaled low-voltage grids with distributed photovoltaic systems. This unique perspective contributes to the otherwise relatively young field of smart grid analytics.
Scaling to a larger number of data silos will introduce a number of challenges: increased communication overhead, handling non-iid data distributions, and the need for robust data preprocessing pipelines to ensure data quality across diverse energy sources and regions.
More specifically, federated data preprocessing remains a rarely examined field. Different photovoltaic systems in different cities require different preprocessing steps in order to account for regional variations and seasonality."
[FR] You have been working with FL for several years. How would you characterize the current landscape of FL tools and platforms? Given that abundance of frameworks, what did you find most useful about the integrate.ai platform?
[PR] "I’ve discussed the landscape of open source platforms in a recent blog post that can be consulted here. That said, the choice of a productized solution vs an open source library depends on the context.
For energy prediction, we decided to use integrate.ai for the following reasons:
- The product makes FL as easy and user-friendly as possible. We did not have to worry about low-level infrastructure and implementation details.
- Privacy-preserving technologies such as DP worked out-of-the-box.
- The level of support from the team has been excellent. Of course, this is usually the difference between an open-source library vs a product.
- The web UI has made it very easy to manage deployments and session metrics."
[FR] How do you see FL evolving in your field? What technical challenges should be solved?
[PR] "Federated Learning will transform multiple domains like no other AI technology, from medicine, to energy supply, to natural language processing (e.g., fine-tuning of LLMs). It is increasingly bridging the gap between data privacy and collaborative AI across industries.
We see Federated Data Preprocessing as the next challenge to be solved, so that eventually, a fully federated pipeline - from exploratory analysis to model inference - can be provided for data scientists."
-p-500.jpeg)
