Your issue (and many others) are why I always discourage everyone I know from taking loans out of the 401k plan. It's just not a regular loan although people want to think of it that way and want it to work the same way.
I've seen way too many different (and bad) issues to ever recommend it but YMMV.