If an ESOP does not comply with the 409(e) voting requirements, is the validity of the entire plan jeopardized, or only the ESOP characteristics of the plan?

If the employer has no stock that is publicly-traded, the ESOP is subject to IRC Section 409(e) by reason of Section 401(a)(22)....which is a qualification requirement. Non-compliance under these circumstances would jeopardize the qualified status of the ESOP under IRC Section 401(a)....but this does not jeopardize the "validity" of the entire plan as an employee benefit plan under ERISA.

If the employer has stock that is publicly traded, the ESOP is subject to Section 409(e) by reason of Section 4975(e)(7), which is the "ESOP" definition, and not by reason of a Section 401(a) qualification requirement.

Note that there are several IRS programs which now allow for the "correction" of Section 401(a) qualification defects.