[Kemapreedasse]

Being a high handicapper myself, I see nothing wrong in what you're doing. You have an honest handicap, play to the best of your current ability and are progressing (getting better). You are also playing within the same set of rules as everyone else. It doesn't get any more fair than that.
Personally, I'd gladly direct them to my posterior and ask them to pucker up. Talking trash is fine and part of competition but I will not stand for it to be done behind my back, you got a problem let's talk about it.
My attitude about such things is lowly changing as I get more into golf and learn that settling it on the putting green or parking lot is frowned upon.