I want to convert my traditional 401(k) with an old employer to a Vanguard Roth 401(k).
You'll have to check with your former employer as to whether they will allow less than a total distribution after termination. Vanguard will allow you to roll in money from a traditional 401(k) to a Vanguard Roth 401(k), assuming that your new employer has elected to have a Roth option available in the plan.
If you are asking if you can convert from a traditional 401(k) account to a ROTH 401(k) account the answer is that rules allow partial conversion but your Plan may not. Even if your plan does allow conversion to a ROTH 401(k) they limit the conversion to participants who are over age 59-1/2. And even if they allow conversions for participants of any age they may limit the conversion to certain account types such as only your employee contribution account and not the employer contribution account. Lastly, the Plan has the ability to limit the amount and frequency of the conversion. So, for example, conversion amounts may be limited to the entire lump sum account balance in which case you will not be able to make incremental conversions.
If you are asking if you can convert your traditional 401(k) to a ROTH IRA then this is also permitted by law. However, again, you have the Plan rules to contend with. If you are still working for the company with the 401(k) then you have to be sure that you qualify to take money out of your 401(k) while employed. This is called an 'in-service' distribution. If your 401(k) is with a prior employer then you can arrange to rollover your balance into a ROTH IRA but you have to be sure that the plan allows partial withdrawals. They may require a lump-sum. If they require a lump sum the work around is to take the lump sum distribution from your 401k and roll it into a traditional IRA. This will not be taxable to you. Once it is in the traditional IRA you can make annual conversions from the traditional to the ROTH IRA.
Hope this helps.