[RISCV][NFC] Refine the patch of D141061

Just saw Craig's comment after I commit, he has suggest a good NFC
for that change.