Hi, Kunook! Thanks for visiting the boards. I agree with everything Heather is saying, and just wanted to mention that I relate to the feelings of deep guilt around telling someone no, changing my mind or changing plans, making someone unhappy, etc. Something I've been learning is that all ignoring my intuition gets me is a crummy time, because if I'm not authentically interested and comfortable with doing something, the other person can probably tell. Like Heather said, if it doesn't feel right, why do it? That being said, you're not alone in feeling anxiety around it - I do too.
There's a part in Be Your Own Superhero: Learning How and When to Stand Up for Ourselves
that I really like, where you practice standing up for yourself and your choices through a script you come up with that prepares yourself for different responses - like the person you're talking to being unhappy. It helps me feel calmer when I'm confronted with that, instead of freaking out and changing back to try and make the person happy or "fix" the situation (when really, I've done nothing wrong and have made things harder on myself). Using tools like could help you find out what feels like the best way to communicate your feelings with this guy