I figured if the TPA was a fiduciary he could be liable as you stated.
If not a fiduciary, the only thing I can think of that could make the TPA liable is as a knowing participant based on the Harris Trust decision, and even this probably wouldn't hold up.
Thanks.