Avoid building data structures we don't really need. This improves the runtime

of a test that Bill Wendling sent me from 228.5s to 105s.  Obviously there is
more improvement to be had, but this is a nice speedup which should be "felt"
by many programs.

llvm-svn: 8962