[NFC] Make FPClassTest a bitmask enumeration

This is recommit of 2e416cdd, fixed to be accepatble by GCC.
The original commit message is below.

With this change bitwise operations are allowed for FPClassTest
enumeration, it must simplify using this type. Also some functions
changed to get argument of type FPClassTest instead of unsigned.

Differential Revision: https://reviews.llvm.org/D144241