I always wonder as well. I know my wife was dating a guy and fucking him when we started dating and it didn't bother me to find out. I'm sure she's had plenty of lovers. It was funny when I found out she had dated the husband of a HS friend of mine. Turns out they're swingers and would love to do so with us. lol. My wife doesn't know that though.
