Bad phrasing on my part. It's more like usually there's no distinction made between what language is spoken at all. Wherever you go, you can talk to everyone, and everyone can talk to you. You can replace 'Japanese' with 'English' or any other universal language variant of your choice.
The only time when this isn't the case is when the anime/manga in question actually takes itself seriously, or if misunderstanding is intended for comedic effect.