What happens when we run out of data for AI models

 Large Language Models (LLM) are one of the hottest innovations today. With companies like OpenAI and Microsoft working to release impressive new NLP systems, no one can deny the importance of having access to large amounts of quality data that cannot be undermined.

However, according to recent research by Epoch, we may soon need more data to train AI models. The team explored the wealth of high-quality data available on the Internet. ("High quality" refers to sources such as Wikipedia, as opposed to low quality data such as social media posts.)

The analysis shows that high-quality data will soon be exhausted, probably before 2026. While resources for low-quality data will be exhausted decades later, it is clear that the current trend of endlessly scaling models to improve results may soon slow down.

Machine learning (ML) models are known to improve their performance as the amount of data they are trained on increases. However, simply feeding more data into the model is not always the best solution. This is especially true for rare events or specialized applications. For example, if we want to train a model to detect a rare disease, we may need more data to work with. But we still want the models to be more accurate over time.

This suggests that if we want to avoid a slowdown in technological development, we need to develop other paradigms for building machine learning models that are independent of the amount of data.

In this article, we will talk about what these approaches look like and evaluate the pros and cons of these approaches.

Scale limitations of AI models

One of the most significant challenges of scaling machine learning models is the diminishing returns of increasing model size. As the model size continues to grow, the improvement in its performance is marginal. This is because the more complex the model, the more difficult it is to optimize and the more prone it is to overfitting. Additionally, larger models require more computational resources and training time, making them less practical for real-world applications.

Another significant limitation of scaling models is the difficulty of ensuring their robustness and generalizability. Robustness refers to the ability of a model to perform well even when faced with noisy or adverse inputs. Generalizability refers to the ability of a model to work well on data it has not seen during training. As models become more complex, they become more vulnerable to enemy attacks, making them less resilient. Additionally, larger models memorize the training data rather than learning the underlying patterns, leading to poor generalization performance.

Interpretability and explainability are essential to understanding how a model makes predictions. However, as models become more complex, their inner workings become increasingly opaque, making their decisions more difficult to interpret and explain. This lack of transparency can be problematic in critical applications such as healthcare or finance, where the decision-making process must be explainable and transparent.

Alternative approaches to building machine learning models

One approach to overcome the problem would be to rethink what we consider high-quality and low-quality data. According to University of Southern California ML professor Swabh Swayamdipta, creating more diversified training datasets could help overcome the limitations without reducing quality. Additionally, training a model on the same data multiple times could help reduce costs and reuse data more efficiently, he said.

These approaches could delay the problem, but the more times we use the same data to train our model, the more prone it is to overfitting. We need effective strategies to overcome the data problem in the long run. So what are some alternative solutions to easily add more data to the model?

JEPA (Joint Empirical Probability Approximation) is a machine learning approach proposed by Yann LeCun that differs from traditional methods in that it uses empirical probability distributions to model data and make predictions.

In traditional approaches, a model is designed to fit the data with a mathematical equation, often based on assumptions about the underlying distribution of the data. However, in JEPA, the model is learned directly from the data through an empirical approximation of the distribution. This approach involves dividing the data into subsets and estimating a probability distribution for each subset. These probability distributions are then combined to form a joint probability distribution used to make predictions. JEPA can process complex, high-dimensional data and adapt to changing data patterns.

Another approach is to use data augmentation techniques. These techniques involve modifying existing data to create new data. This can be done by flipping, rotating, cropping or adding noise to the images. Data augmentation can reduce overfitting and improve model performance.

Finally, you can use transfer learning. This involves using a pre-trained model and fine-tuning it for a new task. This can save time and resources because the model has already learned valuable features from a large dataset. A pretrained model can be fine-tuned with a small amount of data, making it a good solution for sparse data.