The Buddha taught that both eternalism and annhiliationism are wrong views,.
which to me suggest the problem is mostly about holding a fixed closed minded view...
Buddha must have spoken about these things for a reason...