It may have been semen, yes; it may also have been lubricant or your own vaginal fluids.
If this happened on Monday and you are not taking the pill, or were not using any backup method to the condom, then to be safer, you should treat this as a pregnancy risk. If you do not wish to become pregnant, there is still time for you to use Emergency Contraception. The sooner you use it, the more effective it is.
As well, both of you should consider yourselves to have had a STD/STI transmission risk; that means it's time for testing. See this for more: Testing, Testing.
In the future, when in a situation in which the condom could slip, he should hold the base of the condom to keep it in place. This also goes for when he withdraws the penis (pulls out).
Edited to add: The soonest one can take a pregnancy test is 2 weeks after unprotected sex or on the first day of a missed period.
"There is a tragic flaw in our precious Constitution, and I don’t know what can be done to fix it. This is it: Only nut cases want to be president." - Kurt Vonnegut, "Cold Turkey"
[This message has been edited by lemming (edited 05-18-2004).]