Eliminate some unneeded signed/unsigned conversions

No functional change is intended, but generally this should be a bit
more safe.

llvm-svn: 342823