Imagine being a BMW X3 owner and receiving an offer to buy back your 2021 car, only for the dealership to revoke it after realizing an AI Chatbot had made a mistake. That is exactly what befell Zack Giacomelli after a Toronto dealership got in touch to explain what had happened.
Giacomelli had bought the car three years ago from the dealership but was looking to recoup some of the money for the car. So a representative from the Toronto dealership named Quinn reached out to Giacomelli via text to arrange a buy-back deal on the car.
That was enough to cover nearly all of what he still owed on the BMW, so he was incredibly happy to have had the offer. Giacomelli was then shocked to learn that Quinn wasn’t a real person, but it was just an AI chatbot that the dealership had been using for the offer.
Giacomelli spoke to CBC News about it and expressed his shock and frustration at the situation. It is a truly bizarre situation, but it is one that really highlighted just how problematic AI can be for businesses if the models and software are not up to the job.
How the AI Chatbot Situation Had Unfolded
We’ve discussed this story before, but it is important to revist it and to further highlight what problems AI can cause. Giacomelli got in touch with the Toronto dealership after he realized the car needed some major repairs. This was when Quinn got in touch about the car, noting that Giacomelli’s car was at the dealership for service, and coming back with an offer of $27,162.79 for the BMW SUV.
Giacomelli was over the moon. “I will come out at zero; I was so glad this was happening,” Giacomelli said to the dealership. He would owe the dealership nothing after that offer, and could then look at getting another car without worrying about the previous payments. It was then that things went south, really fast with the offer being revoked.
It transpired that Quinn wasn’t a real human being from the dealership, but simply an AI chatbot. It had gone by a real name, and led Giacomelli to believe that he had a legitimate offer for the car, which turned out to be a mistake by the chatbot. It laid bare the issues that AI can cause to customers of businesses,
The Chatbot Helped Expose Some Flaws With AI
After they revoked his offer, the dealership then came back with a real offer of around $20,000, before eventually deciding to honor the original AI deal. The AI chatbot had been talking and acting like a real human being. It gave no concern to the owner of the BMW, and reading the text messages he received, it was easy to see why. AI has some great uses, and it has been used in the medical industry recently with some success. Yet equally, the situation in Toronto showed it is still problematic in many ways.
First, it was hard for Giacomelli to determine if it was real or not. The way it texted him and phrased messages looked very convincing. Naming the chatbot also helped to humanize the bot, and not once did it mention it was AI or an AI assistance in it’s messages. It also highlighted that, in many ways, AI is quite primitive. Not once did the chatbot ask for time to evaluate the deal, and it clearly wasn’t clever enough to clarify the offer would be correct with a real human employee. Situations like this may crop up again in the future, so perhaps this particular occurrence can be used as a warning to other businesses that now use AI on a regular basis.