llvm-svn: 166453

testsuite.

llvm-svn: 164723

This feature allows us to group test cases into logical groups (categories), and to only run a subset of test cases based on these categories.

Each test-case can have a new method getCategories(self): which returns a list of strings that are the categories to which the test case belongs.
If a test-case does not provide its own categories, we will look for categories in the class that contains the test case.
If that fails too, the default implementation looks for a .category file, which contains a comma separated list of strings.
The test suite will recurse look for .categories up until the top level directory (which we guarantee will have an empty .category file).

The driver dotest.py has a new --category <foo> option, which can be repeated, and specifies which categories of tests you want to run.
(example: ./dotest.py --category objc --category expression)

All tests that do not belong to any specified category will be skipped. Other filtering options still exist and should not interfere with category filtering.
A few tests have been categorized. Feel free to categorize others, and to suggest new categories that we could want to use.

All categories need to be validly defined in dotest.py, or the test suite will refuse to run when you use them as arguments to --category.

In the end, failures will be reported on a per-category basis, as well as in the usual format.

This is the very first stage of this feature. Feel free to chime in with ideas for improvements!

llvm-svn: 164403

Patch from Filipe Cabecinhas that uses argparse in dotest.py instead of a hand coded option. I made a few modifications:

Changed the '-A' option to also have a long option of '--arch'. This is now specified multiple times to get multiple architectures.

Old: -A i386^x86_64
New: -A i386 -A x86_64
     --arch i386 --arch x86_64
     
Changed the '-C' option to also have a long option of '--compiler'. This is now specified multiple times to get multiple compiler.

Old: -C clang^gcc
New: -C clang -C gcc
     --compiler clang --compiler gcc
llvm-svn: 163141

llvm-svn: 162372

llvm-svn: 162040

llvm-svn: 161493

llvm-svn: 161492

    ./dotest.py

No progress bar appears.

llvm-svn: 157226

Enrico will follow this up with fixing the data formatter test cases that are failing.

llvm-svn: 155514

Add a '-R' option, which is similar to '-r', except that the relocated directory, if exists, will be removed entirely
before running the test suite.  A usage example looks like this:

test $ ./dotest.py -A x86_64 -R /tmp/x86_64 &
test $ ./dotest.py -A i386 -R /tmp/i386 &

where we would want to run the x86_64 and i386 archs concurrently but relocate the test suite to different directory
hierarchies in order not to stump on each other's intermediate files.

llvm-svn: 155491

rdar://problem/11283401

Example:

Collected 1 test

1: test_with_dwarf (TestCallStdStringFunction.ExprCommandCallFunctionTestCase)
   Test calling std::String member function. ... FAIL

======================================================================
FAIL: test_with_dwarf (TestCallStdStringFunction.ExprCommandCallFunctionTestCase)
   Test calling std::String member function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/ToT/test/lldbtest.py", line 427, in wrapper
    return func(self, *args, **kwargs)
  File "/Volumes/data/lldb/svn/ToT/test/expression_command/call-function/TestCallStdStringFunction.py", line 34, in test_with_dwarf
    self.call_function()
  File "/Volumes/data/lldb/svn/ToT/test/expression_command/call-function/TestCallStdStringFunction.py", line 48, in call_function
    substrs = ['Hello world'])
  File "/Volumes/data/lldb/svn/ToT/test/lldbtest.py", line 1235, in expect
    msg if msg else EXP_MSG(str, exe))
AssertionError: False is not True : 'Hello world' returns expected result
Config=i386-clang
----------------------------------------------------------------------
Ran 1 test in 1.148s

FAILED (failures=1)

llvm-svn: 155157

llvm-svn: 155152

the pre-flight code gets executed during setUp() after the debugger instance is available
and the post-flight code gets executed during tearDown() after the debugger instance has
done killing the inferior and deleting all the target programs.

Example:

[11:32:48] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight  functionalities/watchpoint/hello_watchpoint
config: {'pre_flight': <function pre_flight at 0x1098541b8>, 'post_flight': <function post_flight at 0x109854230>}
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-139
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154753
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154730
Last Changed Date: 2012-04-13 18:42:46 -0700 (Fri, 13 Apr 2012)


lldb.pre_flight: def pre_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning pre-flight function:"
    print "for test case:", test

lldb.post_flight: def post_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning post-flight function:"
    print "for test case:", test


Session logs for test failures/errors/unexpected successes will go into directory '2012-04-16-11_34_08'
Command invoked: python ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight functionalities/watchpoint/hello_watchpoint
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok

----------------------------------------------------------------------
Ran 2 tests in 1.584s

OK

llvm-svn: 154847

rdar://problem/11031264

llvm-svn: 154573

either @dsym_test or @dwarf_test to be executed during the testsuite run.  There are still lots of
Test*.py files which have not been decorated with the new decorator.

An example:

# From TestMyFirstWatchpoint.py ->
class HelloWatchpointTestCase(TestBase):

    mydir = os.path.join("functionalities", "watchpoint", "hello_watchpoint")

    @dsym_test
    def test_hello_watchpoint_with_dsym_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDsym(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()

    @dwarf_test
    def test_hello_watchpoint_with_dwarf_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDwarf(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()


# Invocation ->
[17:50:14] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-137
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154133
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154109
Last Changed Date: 2012-04-05 10:43:02 -0700 (Thu, 05 Apr 2012)



Session logs for test failures/errors/unexpected successes will go into directory '2012-04-05-17_50_49'
Command invoked: python ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... skipped 'dsym tests'
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... ok

----------------------------------------------------------------------
Ran 2 tests in 1.138s

OK (skipped=1)
Session logs for test failures/errors/unexpected successes can be found in directory '2012-04-05-17_50_49'
[17:50:50] johnny:/Volumes/data/lldb/svn/ToT/test $ 

llvm-svn: 154154

to pass to the toolchain in order to build the inferior programs to be run/debugged
duirng the test suite.  The architecture might dictate some special CFLAGS which are
more easily specified in a central place (like the command line) instead of inside
make rules.

For Example,

./dotest.py -v -r /shared/phone -A armv7 -E "-isysroot your_sdk_root" functionalities/watchpoint/hello_watchpoint

will relocate the particular test directory ('functionalities/watchpoint/hello_watchpoint' in this case) to a
new directory named '/shared/phone'.  The particular incarnation of the architecture-compiler combination of the
test support files are therefore to be found under:

/shared/phone.arch=armv7-compiler=clang/functionalities/watchpoint/hello_watchpoint

The building of the inferior programs under testing is now working.

The actual launching/debugging of the inferior programs are not yet working,
neither is the setting of a watchpoint on the phone.

llvm-svn: 153070

llvm-svn: 152568

Add the capability on OS X to utilize 'xcrun' to locate the compilers used for building the inferior programs
to be debugged while running the test suite.  By default, compilers is set to ['clang'] and can be overridden
using the "-C compilerA^compilerB" option.

llvm-svn: 152367

Change the test driver so that, by default, it takes into consideration of both 'x86_64' and 'i386' architectures
when building the inferior programs.

Example:

/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -v functionalities/watchpoint
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-123
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 152244
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 152244
Last Changed Date: 2012-03-07 13:03:09 -0800 (Wed, 07 Mar 2012)



Session logs for test failures/errors/unexpected successes will go into directory '2012-03-08-16_43_51'
Command invoked: python ./dotest.py -v functionalities/watchpoint

Configuration: arch=x86_64
----------------------------------------------------------------------
Collected 21 tests

 1: test_hello_watchlocation_with_dsym (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 2: test_hello_watchlocation_with_dwarf (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 3: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 4: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 5: test_watchpoint_multiple_threads_with_dsym (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 6: test_watchpoint_multiple_threads_with_dwarf (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 7: test_rw_disable_after_first_stop__with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 8: test_rw_disable_after_first_stop_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 9: test_rw_disable_then_enable_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
10: test_rw_disable_then_enable_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
11: test_rw_watchpoint_delete_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
12: test_rw_watchpoint_delete_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
13: test_rw_watchpoint_set_ignore_count_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
14: test_rw_watchpoint_set_ignore_count_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
15: test_rw_watchpoint_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
16: test_rw_watchpoint_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
17: test_watchpoint_cond_with_dsym (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
18: test_watchpoint_cond_with_dwarf (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
19: test_watchlocation_with_dsym_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
20: test_watchlocation_with_dwarf_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
21: test_error_cases_with_watchpoint_set (TestWatchpointSetErrorCases.WatchpointSetErrorTestCase)
    Test error cases with the 'watchpoint set' command. ... ok

----------------------------------------------------------------------
Ran 21 tests in 74.590s

OK

Configuration: arch=i386
----------------------------------------------------------------------
Collected 21 tests

 1: test_hello_watchlocation_with_dsym (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 2: test_hello_watchlocation_with_dwarf (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 3: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 4: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 5: test_watchpoint_multiple_threads_with_dsym (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 6: test_watchpoint_multiple_threads_with_dwarf (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 7: test_rw_disable_after_first_stop__with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 8: test_rw_disable_after_first_stop_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 9: test_rw_disable_then_enable_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
10: test_rw_disable_then_enable_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
11: test_rw_watchpoint_delete_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
12: test_rw_watchpoint_delete_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
13: test_rw_watchpoint_set_ignore_count_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
14: test_rw_watchpoint_set_ignore_count_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
15: test_rw_watchpoint_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
16: test_rw_watchpoint_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
17: test_watchpoint_cond_with_dsym (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
18: test_watchpoint_cond_with_dwarf (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
19: test_watchlocation_with_dsym_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
20: test_watchlocation_with_dwarf_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
21: test_error_cases_with_watchpoint_set (TestWatchpointSetErrorCases.WatchpointSetErrorTestCase)
    Test error cases with the 'watchpoint set' command. ... ok

----------------------------------------------------------------------
Ran 21 tests in 67.059s

OK

llvm-svn: 152357

llvm-svn: 149305

environment variable before starting the test runner which executes the test cases and
may spawn child processes.  An example:

    ./dotest.py -u MY_ENV1 -u MY_ENV2 -v -p TestWatchLocationWithWatchSet.py

llvm-svn: 149304

Add a '-X excluded-dir' option to the test driver to exclude a directory from consideration during test discovery.  For example:

    ./dotest.py -X types -v

from the test dir will ignore test cases under test/types.

llvm-svn: 148379

Add usage example of specifying the full path to the compiler used while building the debuggees for running the test suite.

llvm-svn: 148273

llvm-svn: 144940

Use this option with care as you would need to build the inferior(s) by hand
and build the executable(s) with the correct name(s).  This option can be used
with '-# n' to stress test certain test cases for n number of times.

An example:

[11:55:11] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ ls
Makefile		TestValueAPI.pyc	linked_list
TestValueAPI.py		change_values		main.c
[11:55:14] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ make EXE=test_with_dsym
clang -gdwarf-2 -O0  -arch x86_64   -c -o main.o main.c
clang -gdwarf-2 -O0  -arch x86_64   main.o -o "test_with_dsym"
/usr/bin/dsymutil  -o "test_with_dsym.dSYM" "test_with_dsym"
[11:55:20] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ cd ../..
[11:55:24] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
LLDB build dir: /Volumes/data/lldb/svn/trunk/build/Debug
LLDB-89
Path: /Volumes/data/lldb/svn/trunk
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 144914
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 144911
Last Changed Date: 2011-11-17 09:22:31 -0800 (Thu, 17 Nov 2011)



Session logs for test failures/errors/unexpected successes will go into directory '2011-11-17-11_55_29'
Command invoked: python ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
----------------------------------------------------------------------
Collected 1 test

1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.163s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.200s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.198s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.199s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.239s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.215s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.105s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.098s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.195s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.197s

OK
[11:55:34] johnny:/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 144919

tests.

llvm-svn: 143394

llvm-svn: 143207

Print out the version of the locally built 'lldb' binary, not the one found in your PATH env variable.

llvm-svn: 143170

An example (with /Developer/usr/bin/lldb vs. /usr/bin/gdb):

[13:05:04] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v +b -n -p TestCompileRunToBreakpointTurnaround.py
1: test_run_lldb_then_gdb (TestCompileRunToBreakpointTurnaround.CompileRunToBreakpointBench)
   Benchmark turnaround time with lldb vs. gdb. ... 
lldb turnaround benchmark: Avg: 4.574600 (Laps: 3, Total Elapsed Time: 13.723799)
gdb turnaround benchmark: Avg: 7.966713 (Laps: 3, Total Elapsed Time: 23.900139)
lldb_avg/gdb_avg: 0.574214
ok

----------------------------------------------------------------------
Ran 1 test in 55.462s

OK

llvm-svn: 142949

Add benchmarks for expression evaluations (TestExpressionCmd.py) and disassembly (TestDoAttachThenDisassembly.py).

An example:
[17:45:55] johnny:/Volumes/data/lldb/svn/trunk/test $ ./bench.py 2>&1 | grep -P '^lldb.*benchmark:'
lldb startup delay (create fresh target) benchmark: Avg: 0.104274 (Laps: 30, Total Elapsed Time: 3.128214)
lldb startup delay (set first breakpoint) benchmark: Avg: 0.102216 (Laps: 30, Total Elapsed Time: 3.066470)
lldb frame variable benchmark: Avg: 1.649162 (Laps: 20, Total Elapsed Time: 32.983245)
lldb stepping benchmark: Avg: 0.104409 (Laps: 50, Total Elapsed Time: 5.220461)
lldb expr cmd benchmark: Avg: 0.206774 (Laps: 25, Total Elapsed Time: 5.169350)
lldb disassembly benchmark: Avg: 0.089086 (Laps: 10, Total Elapsed Time: 0.890859)

llvm-svn: 142708

which happen before the listingings of test cases.

llvm-svn: 142668

check that the option arg in '-x opt_arg' does not start with a '-' char.

llvm-svn: 142625

Parameterize the iteration count used when running benchmarks, instead of hard-coded inside the test case.
Add a '-y count' option to the test driver for this purpose.  An example:

 $  ./dotest.py -v -y 25 +b -p TestDisassembly.py

...

----------------------------------------------------------------------
Collected 2 tests

1: test_run_gdb_then_lldb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
gdb benchmark: Avg: 0.226305 (Laps: 25, Total Elapsed Time: 5.657614)
lldb benchmark: Avg: 0.113864 (Laps: 25, Total Elapsed Time: 2.846606)
lldb_avg/gdb_avg: 0.503146
ok
2: test_run_lldb_then_gdb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
lldb benchmark: Avg: 0.113008 (Laps: 25, Total Elapsed Time: 2.825201)
gdb benchmark: Avg: 0.225240 (Laps: 25, Total Elapsed Time: 5.631001)
lldb_avg/gdb_avg: 0.501723
ok

----------------------------------------------------------------------
Ran 2 tests in 41.346s

OK

llvm-svn: 142598

for the debugger to execute for certain kind of tests (for example, a benchmark).
A list of runhooks can be used to steer the debugger into the desired state before more
actions can be performed.

llvm-svn: 141626

and the breakpoint specification for the benchmark purpose.  This is used by TestSteppingSpeed.py
to benchmark the lldb stepping speed.  Without '-e' and 'x' specified, the test defaults to
run the built lldb against itself and stopped on Driver::MainLoop, then stepping for 50 times.

rdar://problem/7511193

llvm-svn: 141584

llvm-svn: 139924

llvm-svn: 139920

lookup and cacheing of dSYMs is not enabled, the 'defaults read com.apple.DebugSymbols'
output is not shown.

llvm-svn: 139914

of dSYMs is not turned on before running the test suite.

llvm-svn: 139896