It's likely the drama here that's the killer more than anything else.
In short, like it or not, no matter what reasons this guy is giving you, or how valid you think they are (they sound pretty bogus to me), the long and the short of it is that he has made clear he is not willing to be your committed partner or boyfriend.
He's just not. And it sounds more than a little like you're being played and are really unwilling to see that.
And you, from your accounts, are not okay with what you have going on, including the sex, without that.
So you need to drop the drama and make a choice. Either you can live with a friends-with-benefits situation, or you can't, and you need to move on romantically and sexually. There are no medals given for people who martyr themselves or set themselves up to suffer for nothing. Both of you CAN be without each other. In fact, sounds like he can be without you just fine and like you'd be a lot better off without him or your attachment to the crumbs he drops your way. If you can't care for and respect yourself, you can't really care for or respect anyone else.
Might want to ask yourself this: what is it you love about this person exactly? because it sounds like what you're in love with isn't him, due to the shabby treatment you get and the non-relationship youre in, but your fantasy of him or a relationship, sweetheart.
Editor and Founder, Scarleteen
My epitaph should read: "She worked herself into this ground."
-- Kay Bailey Hutchinson