Be careful to erase values from all of the appropriate sets when they're not...

Be careful to erase values from all of the appropriate sets when they're not needed anymore.  This fixes a few more memory-related issues.

llvm-svn: 37647