[NFC] Move some asserts out of Expensive Checks

This was done by reviewer's request, but in fact without them we
skip very nasty bugs. Unless it is a REAL problem, I'd keep them
in default setup.