A new day, a new article on ChatGPT. Clearly, the AI program is fueling passions, and the possibilities seem endless. The program passes a major Masters of Management exam but hangs on the bar exam. Either way, the researchers have fairly positive conclusions.
Why is this important?
The rise of ChatGPT has some people fearing for jobs. There have been warnings about jobs for editors, journalists and programmers, but today ChatGPT also passes tough exams in business and medicine with flying colors, and should eventually pass those in law as well.In the news : AI robot ChatGPT passes a Master of Business Administration (MBA) final exam. The program also seems to solve medicine tests without problems (we wrote last week).
- “The robot got a grade between B and B- (in a major MBA course exam, ed.),” explains Professor Christian Terwiesch of the prestigious Wharton School (University of Pennsylvania) about his she studies. That is, a score between 80 and 85%.
- ChatGPT, according to the professor, has demonstrated exceptional skills in performing tasks of highly skilled workers such as analysts, managers and consultants.
bad at math
The details : far from perfect.
- In specific case studies, the bot was also shown to be adept at solving problems such as basic managerial operations. According to Terwiesch, ChatGPT’s explanation was “excellent” and he was “surprisingly able to tweak his response based on human cues” where he was told he was wrong. And in fact he was wrong several times:
- lost : in math, kick in touch. “ChatGPT sometimes makes startling mistakes in relatively simple sixth grade calculations. These errors can be very large,” observes the professor. Many testimonials on the bot’s calculation methods circulated on social media go in the same direction.
- Another negative “The current version of Chat GPT cannot handle more advanced process analysis questions, even those based on fairly standard models. This includes process flows with multiple products and problems with stochastic effects such as demand variability.
- ChatGPT, by the way, is able to avoid committing to plagiarism. The program is smart enough to paraphrase itself.
In the future : an important reflection to have for the school world.
- “This has important implications for business school education, including the need for exam policies, curriculum design with a focus on human-AI collaboration, possibilities to simulate real-world decision-making, the need to teach creative problem solving, improve teaching productivity, etc.” reflects Terwiesch.
Barely average in law
Somewhere else : In law, this is not a panacea.
- Even the Stanford lawyers lent to exercise, and took part in the Bar Exam at ChatGPT (MBE, for Multistate Bar Examination). This is a multiple choice quiz (with one correct answer out of four). The bot got 50.3% correct answers. Or better than guess rate (the probability of a correct answer if you guess at random, if you will), which is 25%.
- The exam consists of seven subjects. The AI only got the threshold needed to be successful in two of those parts: in the crimes and in the trials.
- Experts know that this is a very difficult exam: “To be eligible to sit for the exam, most jurisdictions require the candidate to have completed at least seven years of higher education, including three years in an approved law school. In addition, most exam takers must spend weeks or even months of additional exam preparation.Despite this considerable investment of time and capital, approximately one in five candidates score below the required score to pass the exam at the first try. “
- Finally, consider the test rather conclusive. It is mainly due to “hyperparameter optimization and rapid engineering” that a positive impact has been observed. As for the business master, when we tell GPT he was wrong, it improves his response. His second pick was correct 71% of the time. The third choice of him also at 88%.
- “We believe these results strongly suggest that an LLM (language learning machine, ed) will pass the MBE component of the bar exam in the near future,” conclude the experts.