My personal/professional opinion is that it is best to avoid borrowing from your retirement plan unless you have exhausted all other options. On the other hand a fellow professional whose views I have great respect for absolutly disagrees with me on this.
The major pitfall involved with borrowing from your account is in the event you leave your employer prior to fully paying off the loan. In this case the loan could turn into a taxable distribution if it isn't paid off. This would subject you to taxes on this amount plus a 10% penalty if you are under 59 1/2 (there are a few exceptions to this penalty under very specific situations).
If the markets are going up, the money out on loan is not invested and your return would likely lag. When the markets are going down paying yourself interest might be a better deal.