I agree. There are sutta passages which present it literally and I don't see sufficient reason to speculate that he meant them otherwise. When he speaks of beings reappearing in this or that realm "after the breakup of the body, after death", the most parsimonious explanation is that he means post-mortem rebirth.
Certainly the realms can be regarded as mental states or allegories, and rebirth can be understood simply with reference to "becoming". For me, the most important thing is to understand the principle and intent of the teaching, rather than getting tangled up in arguments about ghosts, devils and devas.
I think we can gain insight into the Buddha's intent from examining rival teachings which he rejected. Those of Ajita Kesakambalin, for example:
It's not so much about whether there is an afterlife or not. It's more about statements such as "generosity is taught by idiots", "there is no fruit of good and bad actions",
Mundane "right view" is the view which is conducive to the ethical life and to the contemplative life.