Remove assert from hot code path and add a clarifying comment.

The assert wasn't adding much value but slowed down Release+Asserts builds.

llvm-svn: 145082