[libc] Fix printf %f rounding condition

When running the tbin2dec tests I found a rounding error in my code.
Upon inspection I realized it was due to a lack of parenthesis when
calculating the number of trailing digits. This patch fixes that mistake
and adds unit tests to catch regressions in future.

Reviewed By: lntue

Differential Revision: https://reviews.llvm.org/D145011