I also watch cricket when gets bored and I think its ok to watch it for time you have mentined But why the scholars advise to stay away from the sports is because the sportmen wears short pant which is against the teaching of Shariah and some sports are lengthy like cricket which sometimes continue for 5 days, so Scholars usually advised to just stay away from sports.