[llvm] Fix assertion when stat fails in remove_directories

We were dereferencing an empty Optional if IgnoreErrors was true and the
stat failed.

rdar://60887887

Differential Revision: https://reviews.llvm.org/D131791