Revert "Reapply "ADT: Fix reference invalidation in SmallVector::push_back and...

Revert "Reapply "ADT: Fix reference invalidation in SmallVector::push_back and single-element insert""

This reverts commit 260a856c.
This reverts commit 3043e5a5.
This reverts commit 49142991.

This change had a larger than anticipated compile-time impact,
possibly because the small value optimization is not working as
intended. See D93779.