A new tool called ChatGPT has been found to greatly improve the writing process for marketers, analysts, and managers, according to a study published in “Science”. The chatbot can automate parts of the writing process, saving time for employees and potentially changing the way they approach their work in the future.
The study, conducted by economists Shakked Noy and Whitney Zhang from MIT, involved 453 graduate professionals who were asked to write various texts. The participants were then rewarded based on the productivity of their work, measured in “earned income per minute”.
To determine if participants were more focused on speed or quality, the economists introduced two additional reward schemes that incentivized good texts over quickly delivered ones. The study found that ChatGPT not only allowed participants to complete tasks 40% faster on average, but also improved the quality of their texts by 18%.
The researchers noted that the effects of using ChatGPT were more pronounced in participants who performed poorly without the tool. They believe that ChatGPT helps align performance levels and ensures better content is produced.
Riku Arakawa, a researcher at Carnegie Mellon University, commented on the study, stating that it demonstrates how human-AI collaboration can be effective on certain tasks. Using AI to refine human output can result in better content overall.
However, the study also acknowledged two limitations. Firstly, the tasks were designed to be described by short prompts, while real-world tasks often have vague objectives and instructions that require additional clarification. Secondly, the study did not verify the factual accuracy of the output, which is still a significant challenge for large language models like ChatGPT.
Despite these limitations, the study suggests that AI assistants like ChatGPT could be valuable in the workplace. It should be noted, however, that users must still ensure the accuracy of their work, as seen in the case of attorney Steven Schwartz who was fined for submitting a brief created with ChatGPT that contained false legal opinions.
Noy and Zhang also discussed the impact of increased productivity on jobs. While some economists argue that productivity gains can lead to job reductions, the authors believe that it can also create a higher demand for labor due to reduced product costs. Additionally, the need for thorough examination of generative AI results may require more skilled programmers.
Overall, the study highlights the potential benefits of AI tools like ChatGPT in improving both the speed and quality of written work, though the limitations and challenges must be taken into account for practical implementation.
