Iterators folloring a SmallVector erased element are invalidated so
don't access cached iterators from after the erased element.

Re-apply 49056 with SmallVector support.

llvm-svn: 49106