Revert "Fix tsan problem where the per-thread shared_ptr() can be locked right...

Revert "Fix tsan problem where the per-thread shared_ptr() can be locked right before the cache is destroyed causing a race where it tries to remove an entry from a destroyed cache."

This reverts commit bcc10817.

Reason: Broke the aarch64-asan bot. More information available in the
Phabricator review: https://reviews.llvm.org/D140931