In a way, rise of Humanistic Buddhism addressed this perception. Please do educate yourself. Do some research. You will be amazed at what Buddhists are doing in term of humanitarian relief.
Are Buddhists (we) suppose to be fortune tellers? Please be mindful, this line of thought will ultimately lead to discrimination based on nothing but you perception.