By all means, this fear is irrational, and not something that people generally have when they know the facts about how reproduction happens. But you know, pretty much all of us have at least one thing (and often more) that makes us different from other people, or that might be unusual.
Mental illness, however, which anxiety is, is actually very common. An awful lot of people struggle with mental illness, whether that's anxiety, like you have, or depression, personality disorders or other kinds. There's a lot of cultural stigma still around mental illness because of a long history of cultural ignorance about it, but that is changing. And just because people were ignorant -- and some may still be -- doesn't mean you should be ashamed. Mental illness is like other kinds of illness: mostly just something that happens to people by virtue of being human.
Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has. - Margaret Mead