Having FileManager::getFile always open the file, brought much consternation and leaking of file descriptors.

Add 'openFile' bool to FileManager::getFile to specify whether we want to have the file opened or not, have it
false by default, and enable it only in HeaderSearch.cpp where the open+fstat optimization matters.

Fixes rdar://9139899.

llvm-svn: 127748

After the open+fstat optimization, files were already opened for FileManager::getBufferForFile() and we closed them after reading them.
The problem was that when -working-directory was passed, the code path that actually reuses & closes the already opened file descriptor
was not followed.

llvm-svn: 127639

llvm-svn: 127573

llvm-svn: 127144

llvm-svn: 127133

Reviewed by dgregor.

llvm-svn: 125407

This patch contains:

- making some of the existing comments more accurate in the presence
of virtual files/directories.

- renaming some private data members of FileManager to match their roles better.

- creating 'DirectorEntry's for the parent directories of virtual
files, such that we can tell whether two virtual files are from the
same directory.  This is useful for injecting virtual files whose
directories don't exist in the real file system.

- minor clean-ups and adding comments for class
FileManager::UniqueDirContainer and FileManager::UniqueFileContainer.

- adding statistics on virtual files to FileManager::PrintStats().

- adding unit tests to verify the existing and new behavior of FileManager.

llvm-svn: 125384

AST/PCH files more lazy:
  - Don't preload all of the file source-location entries when reading
  the AST file. Instead, load them lazily, when needed.
  - Only look up header-search information (whether a header was already
  #import'd, how many times it's been included, etc.) when it's needed
  by the preprocessor, rather than pre-populating it.

Previously, we would pre-load all of the file source-location entries,
which also populated the header-search information structure. This was
a relatively minor performance issue, since we would end up stat()'ing
all of the headers stored within a AST/PCH file when the AST/PCH file
was loaded. In the normal PCH use case, the stat()s were cached, so
the cost--of preloading ~860 source-location entries in the Cocoa.h
case---was relatively low.

However, the recent optimization that replaced stat+open with
open+fstat turned this into a major problem, since the preloading of
source-location entries would now end up opening those files. Worse,
those files wouldn't be closed until the file manager was destroyed,
so just opening a Cocoa.h PCH file would hold on to ~860 file
descriptors, and it was easy to blow through the process's limit on
the number of open file descriptors.

By eliminating the preloading of these files, we neither open nor stat
the headers stored in the PCH/AST file until they're actually needed
for something. Concretely, we went from

*** HeaderSearch Stats:
835 files tracked.
  364 #import/#pragma once files.
  823 included exactly once.
  6 max times a file is included.
  3 #include/#include_next/#import.
    0 #includes skipped due to the multi-include optimization.
1 framework lookups.
0 subframework lookups.

*** Source Manager Stats:
835 files mapped, 3 mem buffers mapped.
37460 SLocEntry's allocated, 11215575B of Sloc address space used.
62 bytes of files mapped, 0 files with line #'s computed.

with a trivial program that uses a chained PCH including a Cocoa PCH
to

*** HeaderSearch Stats:
4 files tracked.
  1 #import/#pragma once files.
  3 included exactly once.
  2 max times a file is included.
  3 #include/#include_next/#import.
    0 #includes skipped due to the multi-include optimization.
1 framework lookups.
0 subframework lookups.

*** Source Manager Stats:
3 files mapped, 3 mem buffers mapped.
37460 SLocEntry's allocated, 11215575B of Sloc address space used.
62 bytes of files mapped, 0 files with line #'s computed.

for the same program.

llvm-svn: 125286

llvm-svn: 125129

overridden via remapping. Thus, when we create a "virtual" file in the
file manager, we still stat() the real file that lives behind it so
that we can provide proper uniquing based on inodes. This helps keep
the file manager much more consistent.

To take advantage of this when reparsing files in libclang, we disable
the use of the stat() cache when reparsing or performing code
completion, since the stat() cache is very likely to be out of date in
this use case.

llvm-svn: 124971

links.

llvm-svn: 124776

redundant searches in the string. No functionality change.

llvm-svn: 124760

on that name. Canonicalization eliminates silliness such as "." and
"foo/.." that breaks the uniquing of files in the presence of virtual
files or files whose inode numbers have changed during
parsing/re-parsing. c-index-test isn't able to create this crazy
situation, so I've resorted to testing outside of the Clang
tree. Fixes <rdar://problem/8928220>.

Note that this hackery will go away once we have a real virtual file
system on which we can layer FileManager; the virtual-files hack is
showing cracks.

llvm-svn: 124754

TextDiagnosticPrinter.cpp: Show diagnostics as far as possible even with invalid PresomedLoc, instead of just silencing it.

FileManager.cpp: Allow virtual files in nonexistent directories.
FileManager.cpp: Close FileDescriptor for virtual files that correspond to actual files.
FileManager.cpp: Enable virtual files to be created even for files that were flagged as NON_EXISTENT_FILE, e.g. by a prior (unsuccessful) addFile().

ASTReader.cpp: Read a PCH even if the original source files cannot be found.

Add a test for reading a PCH of a file that has been removed and diagnostics referencing that file.

llvm-svn: 124374

llvm-svn: 122340

llvm-svn: 122087

llvm-svn: 121956

llvm-svn: 121378

llvm-svn: 120688

llvm-svn: 120297

followed by an open for every source file we open, probe the file system with
'open' and then do an fstat when it succeeds.  open+fstat is faster than
stat+open because the kernel only has to perform the string->inode mapping
once.  Presumably it gets faster the deeper in your filesystem a lookup
happens.

For -Eonly on cocoa.h, this reduces system time from 0.042s to 0.039s on
my machine, a 7.7% speedup.

llvm-svn: 120066

llvm-svn: 120061

pointer that is passed down through the APIs, and make
FileSystemStatCache::get be the one that filters out
directory lookups that hit files.  This also paves the
way to have stat queries be able to return opened files.

llvm-svn: 120060

which simplifies clients and is important for future directions.
Add a FD member to FileEntry which isn't used but will be shortly.

llvm-svn: 120056

llvm-svn: 120048

method in FileSystemStatCache.

llvm-svn: 120037

MemoryBuffer::getFile, causing us to pick up a fstat for
every file.  Restore the optimization.

llvm-svn: 120032

its own header and giving it some more structure.  No 
functionality change.

llvm-svn: 120030

two copies, since they are fundamentally different
operations and the StringRef one should go away
(it shouldn't be part of FileManager at least).

Remove some dead arguments.

llvm-svn: 120013

FileSystemOpts through a ton of apis, simplifying a lot of code.
This also fixes a latent bug in ASTUnit where it would invoke
methods on FileManager without creating one in some code paths
in cindextext.

llvm-svn: 120010

to simplify a bunch of code in it.  It should ultimately get inlined
into FileManager.

llvm-svn: 120007

This patch completely defeated the "passing in a prestat'd size 
to MemoryBuffer" optimization, leading to an extra fstat call for
every buffer opened, in order to find out if the datestamp and size 
of the file on disk matches what is in the stat cache.

I fully admit that I don't completely understand what is going on here:
why punish code when a stat cache isn't in use?  what is the point of a
stat cache if you have to turn around and stat stuff to validate it?

To resolve both these issues, just drop the modtime check and check the
file size, which is the important thing anyway.  This should also resolve
PR6812, because presumably windows is stable when it comes to file sizes.
If the modtime is actually important, we should get it and keep it on the
first stat.

This eliminates 833 fstat syscalls when processing Cocoa.h, speeding up
system time on -Eonly Cocoa.h  from 0.041 to 0.038s.

llvm-svn: 120001

llvm-svn: 119997

llvm-svn: 119995

llvm-svn: 119939

llvm-svn: 119934

When -working-directory is passed in command line, file paths are resolved relative to the specified directory.
This helps both when using libclang (where we can't require the user to actually change the working directory)
and to help reproduce test cases when the reproduction work comes along.

--FileSystemOptions is introduced which controls how file system operations are performed (currently it just contains
 the working directory value if set).
--FileSystemOptions are passed around to various interfaces that perform file operations.
--Opening & reading the content of files should be done only through FileManager. This is useful in general since
 file operations will be abstracted in the future for the reproduction mechanism.

FileSystemOptions is independent of FileManager so that we can have multiple translation units sharing the same
FileManager but with different FileSystemOptions.

Addresses rdar://8583824.

llvm-svn: 118203

#pragma once wasn't working on win32 if the header file was included
using a different case.
I tracked down  the problem to the fact that clang::FileManager was
caching files using case sensitive string (UniqueFiles) on Windows.

I changed FileManager to cache filename in lower case only.
Doesn't affect UNIX because UNIX uses Inode to uniquely identify files.

unix doesn't use this codepath.

Analysis and patch by Francois Pichet!

llvm-svn: 111866

its absolute path on disk. Also, introduce a fun test for the
precompiled preamble, which almost works...

llvm-svn: 109470

llvm-svn: 108375