Summary:
This patch moves out the definition of the XRay log file header from
binary logs into its own header and implementation file.

This is one part of the refactoring being done in D50441.

Reviewers: eizan

Subscribers: mgorny, hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D51086

llvm-svn: 340389

Summary:
This change uses a single offset pointer used throughout the
implementation of the individual record parsers. This allows us to
report where in a trace file parsing failed.

We're still in an intermediate step here as we prepare to refactor this
further into a set of types and use object-oriented design principles
for a cleaner implementation. The next steps will be to allow us to
parse/dump files in a streaming fashion and incrementally build up the
structures in memory instead of the current all-or-nothing approach.

Reviewers: kpw, eizan

Reviewed By: kpw

Subscribers: hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D50169

llvm-svn: 339092

Reverted r338825 and all the following tries to fix issues introduced by that commit (r338826, r338827, r338829, r338880).

This commit has broken build bots and has been left unattended for too long.

llvm-svn: 338948

llvm-svn: 338880

Follow up to D48370.

llvm-svn: 338829

Follow up to D48370.

llvm-svn: 338827

Summary:
This change implements the profile loading functionality in LLVM to
support XRay's profiling mode in compiler-rt.

We introduce a type named `llvm::xray::Profile` which allows building a
profile representation. We can load an XRay profile from a file to build
Profile instances, or do it manually through the Profile type's API.

The intent is to get the `llvm-xray` tool to generate `Profile`
instances and use that as the common abstraction through which all
conversion and analysis can be done. In the future we can generate
`Profile` instances from `Trace` instances as well, through conversion
functions.

Some of the key operations supported by the `Profile` API are:

- Path interning (`Profile::internPath(...)`) which returns a unique path
  identifier.

- Block appending (`Profile::addBlock(...)`) to add thread-associated
  profile information.

- Path ID to Path lookup (`Profile::expandPath(...)`) to look up a
  PathID and return the original interned path.

- Block iteration.

A 'Path' in this context represents the function call stack in
leaf-to-root order. This is represented as a path in an internally
managed prefix tree in the `Profile` instance. Having a handle (PathID)
to identify the unique Paths we encounter for a particular Profile
allows us to reduce the amount of memory required to associate profile
data to a particular Path.

This is the first of a series of patches to migrate the `llvm-stacks`
tool towards using a single profile representation.

Depends on D48653.

Reviewers: kpw, eizan

Reviewed By: kpw

Subscribers: mgorny, llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D48370

llvm-svn: 338825

Summary:
llvm-xray changes:
- account-mode - process-id  {...} shows after thread-id
- convert-mode - process {...} shows after thread
- parses FDR and basic mode pid entries
- Checks version number for FDR log parsing.

Basic logging changes:
- Update header version from 2 -> 3

FDR logging changes:
- Update header version from 2 -> 3
- in writeBufferPreamble, there is an additional PID Metadata record (after thread id record and tsc record)

Test cases changes:
- fdr-mode.cc, fdr-single-thread.cc, fdr-thread-order.cc modified to catch process id output in the log.

Reviewers: dberris

Reviewed By: dberris

Subscribers: hiraditya, llvm-commits, #sanitizers

Differential Revision: https://reviews.llvm.org/D49153

llvm-svn: 336974

Summary:
When there's a mismatch of a function argument being right after the
wrong function, print an offset into the file where that happened, to
ease further debugging.

Reviewers: dberris, eizan, kpw

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D42492

llvm-svn: 323758

Summary:
Before this change, the FDR mode implementation relied on at thread-exit
handling to return buffers back to the (global) buffer queue. This
introduces issues with the initialisation of the thread_local objects
which, even through the use of pthread_setspecific(...) may eventually
call into an allocation function. Similar to previous changes in this
line, we're finding that there is a huge potential for deadlocks when
initialising these thread-locals when the memory allocation
implementation is also xray-instrumented.

In this change, we limit the call to pthread_setspecific(...) to provide
a non-null value to associate to the key created with
pthread_key_create(...). While this doesn't completely eliminate the
potential for the deadlock(s), it does allow us to still clean up at
thread exit when we need to. The change is that we don't need to do more
work when starting and ending a thread's lifetime. We also have a test
to make sure that we actually can safely recycle the buffers in case we
end up re-using the buffer(s) available from the queue on multiple
thread entry/exits.

This change cuts across both LLVM and compiler-rt to allow us to update
both the XRay runtime implementation as well as the library support for
loading these new versions of the FDR mode logging. Version 2 of the FDR
logging implementation makes the following changes:

  * Introduction of a new 'BufferExtents' metadata record that's outside
    of the buffer's contents but are written before the actual buffer.
    This data is associated to the Buffer handed out by the BufferQueue
    rather than a record that occupies bytes in the actual buffer.

  * Removal of the "end of buffer" records. This is in-line with the
    changes we described above, to allow for optimistic logging without
    explicit record writing at thread exit.

The optimistic logging model operates under the following assumptions:

  * Threads writing to the buffers will potentially race with the thread
    attempting to flush the log. To avoid this situation from occuring,
    we make sure that when we've finalized the logging implementation,
    that threads will see this finalization state on the next write, and
    either choose to not write records the thread would have written or
    write the record(s) in two phases -- first write the record(s), then
    update the extents metadata.

  * We change the buffer queue implementation so that once it's handed
    out a buffer to a thread, that we assume that buffer is marked
    "used" to be able to capture partial writes. None of this will be
    safe to handle if threads are racing to write the extents records
    and the reader thread is attempting to flush the log. The optimism
    comes from the finalization routine being required to complete
    before we attempt to flush the log.

This is a fairly significant semantics change for the FDR
implementation. This is why we've decided to update the version number
for FDR mode logs. The tools, however, still need to be able to support
older versions of the log until we finally deprecate those earlier
versions.

Reviewers: dblaikie, pelikan, kpw

Subscribers: llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D39526

llvm-svn: 318733

Summary:
This fixes failure in tools/llvm-xray/X86/graph-zero-latency-calls.yaml
uncovered by D39245.

Reviewers: dberris

Reviewed By: dberris

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D39943

llvm-svn: 318163

Summary:
The arg1 logging handler changed in compiler-rt to start writing a
different type for entries encountered when logging the first argument
of XRay-instrumented functions. This change allows the trace loader to
support reading these record types as well as prepare for when the
basic (naive) mode implementation starts writing down the argument
payloads.

Without this change, binaries with arg1 logging support enabled start
writing unreadable logs for any of the XRay tracing tools.

Reviewers: pelikan

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D38550

llvm-svn: 314967

llvm-svn: 314278

Summary:
A new FDR metadata record will support logging a function call argument;
appending multiple metadata records will represent a sequence of arguments
meaning that "holes" are not representable by the buffer format.  Each
call argument is currently a 64-bit value (useful for "this" pointers and
synchronization objects).

If present, we put this argument to the function call "entry" record it
belongs to, and alter its type to notify the user of its presence.

Reviewers: dberris

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D32840

llvm-svn: 314269

Summary:
This change adds support for explicit tail-exit records to be written by
the XRay runtime. This lets us differentiate the tail exit
records/events in the log, and allows us to treat those exit events
especially in the future. For now we allow printing those out in YAML
(and reading them in).

Reviewers: kpw, pelikan

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D37964

llvm-svn: 313514

Summary:
For readers unfamiliar with the XRay code base, reference the compiler-rt
implementation even though we're not allowed to share any code and explain
our little-endian views more clearly.

For code clarity either get rid of obvious comments or explain their
intentions, fix typos, correct coding style according to LLVM's standards
and manually CSE long expressions to point out it is the same expression.

Reviewers: dberris

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D34339

llvm-svn: 313340

Summary:
When extracting the instrumentation map from a binary, we should be able
to recognize the new kinds of instrumentation sleds we've been emitting
with the compiler using -fxray-instrument. This change adds a test for
all the kinds of sleds we currently support (sans the tail-call sled,
which is a bit harder to force in a simple prebuilt input).

Reviewers: kpw, dblaikie

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D36819

llvm-svn: 311305

No functionality change intended.

llvm-svn: 311288

Summary:
Adding a new restructuredText file to document the trace format produced with
an FDR mode handler and read by llvm-xray toolset.

Fixed two problems in the documentation from differential review. One bad table
and a missing link in the toc.

Original commit was e97c5836a77db803fe53319c53f3bf8e8b26d2b7.

Reviewers: dberris, pelikan

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D36041

llvm-svn: 309891

This reverts commit 3462b8ad41a840fd54dbbd0d3f2a514c5ad6f656.

The docs-llvm-html target failed.

llvm-svn: 309842

Summary:
Adding a new restructuredText file to document the trace format produced with
an FDR mode handler and read by llvm-xray toolset.

Reviewers: dberris, pelikan

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D36041

llvm-svn: 309836

I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line with a #include and let it re-sort things according to the precise
LLVM rules for include ordering baked into clang-format these days.

I've reverted a number of files where the results of sorting includes
isn't healthy. Either places where we have legacy code relying on
particular include ordering (where possible, I'll fix these separately)
or where we have particular formatting around #include lines that
I didn't want to disturb in this patch.

This patch is *entirely* mechanical. If you get merge conflicts or
anything, just ignore the changes in this patch and run clang-format
over your #include lines in the files.

Sorry for any noise here, but it is important to keep these things
stable. I was seeing an increasing number of patches with irrelevant
re-ordering of #include lines because clang-format was used. This patch
at least isolates that churn, makes it easy to skip when resolving
conflicts, and gets us to a clean baseline (again).

llvm-svn: 304787

Summary:
In D30630 we will start writing custom event records. To avoid breaking
the tools that read the FDR mode records, we skip over these records.
To support these custom event records more effectively, we will have to
expose them in the trace loading API. Those changes will be forthcoming.

Reviewers: kpw, pelikan

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D33032

llvm-svn: 302856

Just a spelling change in a comment intended to test svn commit access.

llvm-svn: 299616

Based on corrections mentioned in patch for clang for PR27635

llvm-svn: 299072

Summary:
It is problematic for this reader that it expects to read data from
several threads, but the header or message format does not define
framing. Since the buffers are reused, we can't rely on skipping
zeroed out data as a synchronization method either.

There is an argument that this is not version compatible with the format
the reader expected previously. I argue that since the writer wrote garbage
past the end of buffer record, there is no currently working reader to
compromise.

The corresponding writer change is posted to D31384.

Reviewers: dberris, pelikan

Reviewed By: dberris

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D31385

llvm-svn: 298983

Summary:
The file type packs function trace data onto disk from potentially multiple
threads that are aggregated and flushed during the course of an instrumented
program's runtime.

It is named FDR mode or Flight Data recorder as an analogy to plane
blackboxes, which instrument a running system without access to IO.

The writer code is defined in compiler-rt in xray_fdr_logging.h/cc

Reviewers: rSerge, kcc, dberris

Reviewed By: dberris

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D29697

llvm-svn: 295397

llvm-svn: 294837

Summary:
powerpc64 big-endian is not supported, but I believe that most logic can
be shared, except for xray_powerpc64.cc.

Also add a function InvalidateInstructionCache to xray_util.h, which is
copied from llvm/Support/Memory.cpp. I'm not sure if I need to add a unittest,
and I don't know how.

Reviewers: dberris, echristo, iteratee, kbarton, hfinkel

Subscribers: mehdi_amini, nemanjai, mgorny, llvm-commits

Differential Revision: https://reviews.llvm.org/D29742

llvm-svn: 294781

[MC] Fix some Clang-tidy modernize and Include What You Use warnings in SubtargetFeature; other minor fixes (NFC).

Same changes in files affected by reduced SubtargetFeature.h dependencies.

llvm-svn: 294548

Summary:
This change implements the instrumentation map loading library which can
understand both YAML-defined instrumentation maps, and ELF 64-bit object
files that have the XRay instrumentation map section. We break it out
into a library on its own to allow for other applications to deal with
the XRay instrumentation map defined in XRay-instrumented binaries.

This type provides both raw access to the logical representation of the
instrumentation map entries as well as higher level functions for
converting a function ID into a function address.

At this point we only support ELF64 binaries and YAML-defined XRay
instrumentation maps. Future changes should extend this to support
32-bit ELF binaries, as well as other binary formats (like MachO).

As part of this change we also migrate all uses of the extraction logic
that used to be defined in tools/llvm-xray/ to use this new type and
interface for loading from files. We also remove the flag from the
`llvm-xray` tool that required users to specify the type of the
instrumentation map file being provided to instead make the library
auto-detect the file type.

Reviewers: dblaikie

Subscribers: mgorny, varno, llvm-commits

Differential Revision: https://reviews.llvm.org/D29319

llvm-svn: 293721

Like r291636 and r285261.

llvm-svn: 291786

Summary:
In this change we move the definition of the log reading routines from
the tools directory in LLVM to {include/llvm,lib}/XRay. We improve the
documentation a little bit for the publicly accessible headers, and
adjust the top-matter. This also leads to some refactoring and cleanup
in the tooling code.

In particular, we do the following:

  - Rename the class from LogReader to Trace, as it better represents
    the logical set of records as opposed to a log.
  - Use file type detection instead of asking the user to say what
    format the input file is. This allows us to keep the interface
    simple and encapsulate the logic of loading the data appropriately.

In future changes we increase the API surface and write dedicated unit
tests for the XRay library.

Depends on D24376.

Reviewers: dblaikie, echristo

Subscribers: mehdi_amini, mgorny, llvm-commits, varno

Differential Revision: https://reviews.llvm.org/D28345

llvm-svn: 291652