That sounds like a really rough headspace to get into during sex (or any time, really). I do want to say that the thing where one anxiety tends to send the rest cascading is pretty common. It's sort of like our brains go, "oh, we're in anxiety mode now? Here's EVERYTHING that worries me."
As you pointed out, that wasn't a great time to ask that question, since it probably felt pretty out of left field for him. But I do think it could help you two have a conversation about what's going on with you and sex, assuming you can work through the fight. Can you give me a sense of what the fight is about?
I will say that when people get anxious about their partners thinking about a past partner during sex or comparing, it usually has way more to do with internal insecurities than anything else. How do you feel about your body in general? Do you feel like you're under pressure, whether from your own brain or somewhere else, to "perform" a certain way during sex?
I do want to give you some extra anxiety resources, including ones that might give you tools for interrupting the anxiety spiral (I've used both MoodJuice and the SAM app listed and found them helpful):
Anxiety and Other Mental Health Resources. I also want to give you this resource, since it's built by and for black women, and the podcast they create goes over various mental health tools:
https://providers.therapyforblackgirls.com/.