a phone with a voice assistant enabled has to listen all the time

Yes, but this doesn’t happen online. Your voice is not constantly being sent to Amazon or Google servers for activating the voice assistant. It’s an offline feature that only reacts to the specific word. That’s also the reason why you can’t change the activation word.