I am one of them, If the saathis stopped the Dhikr majlis for this reason, then it was wrong without doubt.