Other than with language models, this has already happened: Take a look at apps such as Merlin Bird ID (identifies birds fairly well by sound and somewhat okay visually), WhoBird (identifies birds by sound, ) Seek (visually identifies plants, fungi, insects, and animals). All of them work offline. IMO these are much better uses of ML than spammer-friendly text generation.
I tried out postmarketOS + phosh on a PinePhone about a year ago. For my own needs, it worked fairly well, except (ironically) receiving calls. It was like driving an old car, everything was slightly jank, but worked, and could be tinkered with - see the entire review. I have to give credit that there has been impressive progress in mobile Linux since PinePhone’s release in 2019, and a lot of it was developed by unpaid hobbyists.