Yes, be concerned. Porn can be a fun activity that inspires new ideas in a relationship, helps people decide what they're into, and helps people communicate it to their partners. I've learned about what my guy likes and I've figured out how to try a few new things by watching porn and by watching the porn my guy is into. It can be part of a lively and healthy sex life. But it should never be a substitute for a sex life.
Ultimatums: I wouldn't tell him "It's porn or me." But "It's me and porn, or nothing at all" is actually fair.
He can watch some porn on his own. But it is fair for you to expect that he will watch
at least half of the time with you, where you can touch each other and enjoy actually having another person in the room instead of being considered a distraction or a nuisance or a turn-off. And at least once or twice a week, you get to spend time with him alone with no porn.
If he can't handle that, the porn is not the problem, he is. If he can't handle that, he's not able to give you what a reasonable person should be able to expect in a relationship.
By the way, my guy and I did have to work on this issue somewhat to get it right, and things are now much better. Guys can be very shy about their desires; mine was. But it is a complete turn-off to feel like you're interrupting his "alone time." If he can't invite me to join in, there's a problem. Happily we seem to have sorted that out in my house.
The easiest thing to do is just tell him it turns you on to know what he's into and you want to see some of his hard drive

It doesn't have to be an "ultimatum" just an invitation.