[IVDescriptors] Clean up includes.

Some includes are not required and forward declarations can be used
instead. This also exposed a few places that were not directly including
required files.