Devients have, in the past, had some control over the Haramayn. Afterall, the Qaramitah stole the Hajar al-Aswad during their time.