Sperm are miscroscopically small. You cannot see them with just your eyes in any situation.
In general, you should always wash your hands before manual sex, period, or use latex gloves. That not only safeguards against sperm being "moved," but against STDs and STIs that could be transmitted.
In your case, unless you touched the inside of the condom, I'd say the chances are slim. Lubricant won't kill sperm, no, but it's unlikely in taking the condom off you really had contact with your semen.
Lastly, there is no need to "pull out" when wearing a condom, and, in fact, doing so abruptly could cause the condom to slip off accidentally. Condoms are designed to be worn through all of intercourse.
Editor and Founder, Scarleteen
My epitaph should read: "She worked herself into this ground."
-- Kay Bailey Hutchinson