Do not try to descend into optional build directories if they do not
exist.  This makes the build logic symmetric for both the in tree and
out of tree cases.

llvm-svn: 126070