[LVI] Remove overly tight assert from 259429

I'll submit a test case shortly which covers this, but it's causing clang self host problems in the builders so I wanted to get it removed.

llvm-svn: 259432