If someone takes a swing at you then you have to take a swing back, regardless of who would win. People will see you as someone they can take advantage of otherwise. Losing a fight always hurts less than running away from one.