I think that if size is the most important thing to you, then you're going to be disappointed on many occasions, and you risk not enjoying sex for what it really is - great feelings, great connection between men, satisfying endings.
Sure, a big dick is nice. I love looking at them, admiring, handling, even sucking and being fucked by a big dick is nice. If you're getting sex from a guy who has a smaller dick, and size is a problem, then you have to decide - do you enjoy the sex and want more, or do you look for a bigger dick? Maybe in casual encounters, you could say nah, I'll shop around. If the guy you've been seeing is not endowed enough for you, is there something more in the relationship that keeps you there? Or do you call it quits and move on?
One of my best fuck buddies years ago had a smaller dick, around 5" and thin. He would get rock hard and would fuck like the Energizer Bunny. We found positions that worked and we both enjoyed. Another fuck buddy had a bigger dick and I enjoyed him too, more for the intimacy, sensuality and connection that he brought to each get-together. If either of those guys became a partner, it would have been because we connected. Dick size would not have been a factor, because the sex was great. The main thing, for me, as a bottom, is that he's fully functional - hard enough to achieve entry - and not a jerk.
As for the OP's other issue, if a top is pulling out and ramming it back in and it hurts, ask him not to do it. We are all consenting adults who can make choices and communicate our likes and dislikes. If he's the kind of guy who insists on pulling out and ramming it back in, even though you've said it hurts, maybe consider whether he's the right guy to be seeing. I guess the OP is long gone but I wanted to address the question for others who might be looking for advice. As an older experienced bottom, I'm full of advice more than I'm full of dick these days, lol.
At the end of the day, any dick that gets hard is suitable for sex. Whether it's enjoyable is up to you.