[flang][runtime] Fix edge-case FP input bugs

Blanks are allowed in more places than I allowed for, and
"NAN(foobar)" is allowed to have any parenthesis-balanced
characters in parentheses.

Update: Fix up old sanity test, then avoid usage of "limit" when null.

Differential Revision: https://reviews.llvm.org/D124294