Hi there sjs0511,
What you're feeling is pretty understandable; you love him, and feel attached to this relationship, so it makes sense that thinking about ending it makes you feel conflicted. But I want to highlight something that you said earlier, that's really, really important:
For me, if something hurts the person I love, I would never do the thing that hurts them..
What you're describing is one of the most basic tenets of a healthy relationship - respecting boundaries. You feel that if you learned that something you were doing was hurting him, you'd stop, because you love him. The way that he's acting isn't showing that same respect, which really isn't fair to you. I hope you know that you always, always deserve to have someone that doesn't push up on your boundaries or ignore something they're doing that's hurting you.
It's also not dumb to have these conflicting feelings - it's you listening to yourself in terms of how you feel. There's some negative
and positive in there (because you would have left already if there wasn't something good!), but you're the one that gets to decide whether or not that balance is working out for you. Echoing what Sam said - how important does this feel for you? I would posit that it's bothering you a significant amount, as you came in to talk with us about it, but does it feel like a dealbreaker? If so, how does it make you feel to think about life outside/beyond this relationship? If it doesn't, what
would be a dealbreaker?