Last spring, when Karim Lakhani began testing how ChatGPT affected the work of elite business consultants, he thought they would be delighted with the tool. In a preliminary study of two dozen workers, the language bot helped them complete a two-hour task in 20 minutes.
“I thought they, like me, would think, ‘Great! I can do so much more!” said Dr. Lakhani, a professor at Harvard Business School.
Instead, the consultants felt uneasy. They appreciate that they can do better work in less time. But ChatGPT’s quick work threatened their sense of themselves as highly skilled workers, and some were afraid to rely on it too much. “They were really concerned and felt that it would demean them and be kind of empty calories for their brain,” said Dr. Grow up.
After these preliminary tests, Dr. Lakhani and his colleagues did a larger, controlled experiment to measure how ChatGPT would affect more than 750 white-collar workers. that study, reviewed in a scientific journal, indicated mixed results on the consultants’ work product. ChatGPT greatly improves the speed and quality of work in a brainstorming task, but it leads many consultants astray when doing more analytical work.
The study also detailed different feelings of workers about the tool. One participant compared it to fire that Prometheus stole from the gods to help mortals. Another said to the colleague of Dr. Fabrizio Dell’Acqua says that ChatGPT is like junk food — hard to resist, easy to consume but ultimately bad for the consumer.
In the near future, language bots such as OpenAI’s ChatGPT, Meta’s Llama and Google’s Gemini are expected to perform many white-collar tasks, such as copy writing, preparation of legal briefs and drafting letters of recommendation. The study is one of the first to show how technology can affect real office work — and office workers.
“This is a well-designed study, especially in a nascent area like this,” said Maryam Alavi, a professor in the Scheller College of Business at the Georgia Institute of Technology who was not involved in the experiments. Dr. also noticed. Alavi, who studied the impact of new digital technology on workers and organizations, said the study “really points out how much more we have to learn.”
The study recruited management consultants from the Boston Consulting Group, one of the largest management-consulting firms in the world. The company has banned its consultants from using AI bots in their work.
“We wanted to include a large set of real workers working on real tasks,” said François Candelon, a managing director of the company who helped design the experiments.
The volunteers were divided into two groups, each of which worked on a different management-consulting problem. Within each group, some consultants used ChatGPT after 30 minutes of training, some used it without instructions and some did not use it.
One of the tasks was to brainstorm about a new type of shoe, sketch a persuasive business plan for its production and write about it persuasively. Some researchers believe that only humans can perform creative tasks.
They are wrong. Consultants who used ChatGPT did work that independent evaluators rated about 40 percent better on average. In fact, people who simply cut and pasted ChatGPT’s output rated it higher than colleagues who mixed its work with their own thoughts. And AI-assisted consultants are more than 20 percent faster.
This year’s ChatGPT studies on legal review and white-collar writing homework found that the bot helped lower performers more than the most skilled. Dr. Lakhani and his colleagues found the same effect in their study.
In a task that requires evidence-based reasoning, however, ChatGPT is unhelpful. In this group, volunteers were asked to advise a corporation that was invented for the study. They had to interpret data from spreadsheets and relate it to mock transcripts of interviews with executives.
Here, ChatGPT encouraged employees to over-trust it. Unaided people have the right answer 85 percent of the time. People who used ChatGPT without training scored just over 70 percent. The trained ones did even worse, getting only 60 percent of the answers.
In interviews conducted after the experiment, “people told us they neglected to check because it was so smooth, it looked right,” said Hila Lifshitz-Assaf, a management professor at Warwick Business School in Britain.
Many consultants said that ChatGPT made them feel uneasy about how the tool would change their profession and even their sense of themselves. Nearly three out of four participants told researchers they worried that using ChatGPT would cause them to waste their own creative muscles, said Mr. Candelon of the Boston Consulting Group.
“If you don’t have an existential crisis about this tool, you haven’t used it very much,” said another co-author, Ethan Mollick, a management professor at the Wharton School at the University of Pennsylvania.
