What these models actually do
A generative model does not know things. It produces the most plausible continuation given what came before, learned from an enormous amount of prior material.
That is not a slight. Plausible continuation is astonishingly useful: it covers drafting, rewriting, summarising, translating, and generating an image matching a description.
But it explains the whole shape of what follows. The model optimises for plausible, not for true, not for original, and not for good. Those often coincide, because true things are usually plausible. When they diverge, the model follows plausible, every time.
So you get output that reads exactly like a confident expert, because it learned the shape of confident expert writing, and that shape is entirely independent of whether the content is correct. Fluency is not evidence of accuracy. They are separate properties that a language model couples only by accident.

