All sorts of factors are in play when you're dealing with the pill.
If you take them at the right time, everyday, they are in theory almost always effective. Most of the people who find the pill not to be effective are the same ones who mess up in terms of when they take it. So, to answer your question, taking it later can cause problems - it might open the risk just a bit.
We generally advise people to use the pill in combination with condoms, and doing that in your case will virtually close the window of risk that you've opened up here. You might want to keep using the condoms after that too, though, because the pill doesn't protect you from STDs.
From the sounds of it, you'll probably want to get yourself tested for STDs/STIs, as well, since you've been having unprotected sex.
...and we raise the white flag, so they can paint it red and blue!
-Joel Plaskett, True Patriot Love