Patterson et al. (2021), a team from Google, provided one of the most comprehensive analyses of the carbon footprint of training large neural networks:
- GPT-3 training: ~552 tonnes CO2 equivalent — like driving 123 gasoline cars for a year
- The paper showed that carbon emissions scale roughly linearly with model size and training time
- However, they also showed that using renewable energy data centers and efficient hardware could reduce the footprint by up to 100x
- Google's own T5 model trained in an efficient data center emitted only 47 tonnes CO2 vs. an estimated 552 tonnes for GPT-3
Context
This paper builds on Strubell et al. (2019) which first raised alarm about AI's energy consumption. While Patterson et al. confirm the problem is real, they also show it's solvable through green infrastructure — making this a "both" story of bitter reality and sweet possibility.
Source
Patterson, D. et al. (2021). Carbon Emissions and Large Neural Network Training. arXiv:2104.10350.