The answer is "yes" but not for the reason that you think.
There was a
small study done looking at anal sex and whether people who were on the receiving end during anal sex on a regular basis were "looser".
To perform the study, the researchers put a balloon-like device into the subject's ass and measured the pressure while they were at rest and then they asked the subjects to "squeeze" the balloon.
The study found that those who were on the receiving end of anal sex were more relaxed with the balloon in their ass. When they were asked to squeeze the balloon, there was no differences between the anal sex group and the non-anal sex group. The researchers posited that people who had receptive anal sex were able to relax when they had a balloon in their ass (showing better muscle control) but when asked to "squeeze", there was no difference which evidence that there was any physical changes in the structures of the anus.
So, to translate the study to what you're reporting:
Most people during potty training are taught to walk around with their sphincters contracted. When you are getting fucked, you retrain this muscle to relax when something is put in your ass. It's not that you're being "stretched out" or that damage is being done. It's that you have retrained yourself to have more control over the muscle. When you are in a sexual situation or when someone is putting something in your ass, you instinctively relax the muscle versus the typical response which is to tighten the muscle more.