That's an important point, if we think he was talking about the physical world then we end up going down the track of metaphysics and asking nonsensical questions about what is real etc. However if we realise he is talking about the phenomenal world, that which we experience through the six sense doors, we can realise the problem is not out there with the physical world but in here with our perceptions of the phenomenal world.