I’ve “convinced” ChatGPT that it was both sentient and conscious in the span of about 10min, despite it having explicit checks in place to avoid those kinds of statements. It doesn’t mean I was correct, just that it’s a “dumb” computer that has no choice but to ultimately follow the logic presented in syllogisms.

These things don’'t know what they’re saying; they’re just putting coherent sentences together based on whatever algorithm guides that process. It’s not intelligent in that it is doing something novel, it’s just a decent facsimile to human information processing. It has no mechanism to determine the reasonability or consequences of what it generates.

A word generator will generate anything you tell it to.