Datasets created by humans are playing an increasingly crucial role in the age of artificial intelligence (AI), according to a study conducted by researchers from the universities of Oxford, Cambridge, and London in the UK. These datasets, however, are becoming scarce. Large language models, such as GPT-4, have primarily been trained using human-generated datasets sourced from the internet. After a second learning pass involving human feedback for fine-tuning, these AI applications perform well. However, with the rapid proliferation of AI-generated texts on the internet, future iterations of this learning process may not be as straightforward. The next generation of AI applications will inevitably be trained on datasets generated by previous AI applications rather than humans. This trend could lead to what the authors of the study refer to as “text garbage,” as connections to original content become completely lost after several iterations. This phenomenon, labeled “The Curse of Recursion,” results in AI models collapsing. Training AI applications on artificially generated data causes them to become forgetful.
