To my knowledge, you are paying Vanguard's rates (or similar rates) when you have Betterment anyway since the portfolio consists largely of Vanguard funds. The value you get from Betterment is mostly in ease of use (including diversification and automatic portfolio balancing), tax loss harvesting, and various helpful tools for advice.
The biggest downside I see to splitting and buying some funds direct from Vanguard is the risk of getting an out of balance portfolio, but I'm not sure how much damage I can really do if I basically just try to mimic what I have in my Betterment account and start buying like half/half to cut down on unnecessary fees.