Percentages here dictate the calculated probabiliies of each token in OpenAI’s GPT-3.5 models, the predecessor to ChatGPT.

text-davinci-002 model (correct answer)

Q: What is 9 + 10? 
A: The answer is...
21 = 55.41%
19 = 43.09%

text-davinci-003 model released on November 2022, two days before ChatGPT (incorrect answer)

Q: What is 9 + 10? 
A: The answer is...
19 = 99.99%
 = 0.00%

This is concerning. At this point, it might as well assume that 2 + 2 = 4!