You have to understand that you DO have a right to your feelings ... And it's completely valid. No one here can tell you how to feel or tell you that you have no right to feel the way that you do.
If it makes you uncomfortable when he talks about stuff like that, tell him. I think it's a bit unfair of him to put you in that position, knowing you have feelings for him (he does know that, right?) ... Communication is super important. Sit down and talk to him about it ... He won't know that he hurt you, otherwise, and then you both might be able to come to an agreement about where this relationship is going.
I'm gonna send this on over to Relationships, where you can get more than one response