This reverts commit 7ff943a9. This wasn't NFC. isSigned != !isUnsigned as there are also relational operators.