[flang][runtime] Add limit check to MOD/MODULO (#80026)

When testing the arguments to see whether they are integers, check first
that they are within the maximum range of a 64-bit integer; otherwise, a
value of larger magnitude will set an invalid operand exception flag.