- fix TestCPPBool (was failing with some versions of GCC due to num_breakpoint_locations)
- speed up tests by skipping git/svn revision parsing by the test harness when running in lit-compatible (parsable) mode

This should resolve the failures reported by http://lab.llvm.org:8011/builders/lldb-x86_64-linux

llvm-svn: 176972

- change string "ERROR" to "FAIL" to match clang lit test results

Also, make LLDB tests work on machines that do not have svn and/or git installed

llvm-svn: 176633

Categories were conceptually meant to be placeable on test methods as well as test classes and test directories
However, that was broken. This checkin fixes that.
The incantation required to put categories on individual test case methods is not exactly elegant, unfortunately:

def test_case(self):
    """Test me."""
    self.do_it()

def _test_case_get_categories(self):
    return ["demo"]

test_case.getCategories = _test_case_get_categories
del _test_case_get_categories

llvm-svn: 176158

Remove the getCategory from TestDataFormatterObjC.py, since it was superceded by the .categories file, 
and didn't work anyway (getCategories currently has to be a method on the test class, not on the test.)
Add a "basic_process" category, and start to find some tests for simple process running sniff tests.

llvm-svn: 176061

The decorators @expectedFailure (plain and special-case like i386, clang, ...) are modified to optionally take a bugnumber argument
If such an argument is specified, the failure report (or unexpected success report) will include the information passed in as part of the message
This is mostly useful for associating failures to issue IDs in issue management systems (e.g. the LLVM bugzilla)

llvm-svn: 175942

 -Change the build_dir variable name to lib_dir
 -Set lib_dir to the correct location on Linux
 -Set LD_EXTRAS to the actual lldb library

llvm-svn: 175664

- occurs when multiple compilers/architectures are tested (via -C or -A flags)

llvm-svn: 175656

llvm-svn: 175557

llvm-svn: 175341

- Add a "parsable" mode to dotest.py that outputs test results in exactly the same format as clang's lit tests
- Improve dosep script to output list of failing tests (output should look like clang test failure summaries)
- Cleanup lldb/test/Makefile to remove needless parameters and environment variables
- Switch makefile tests to use parsable-mode output; should make the buildbot results parsable
- Switch makefile tests to use dosep to log catch crashing tests (instead of halting the test suite)

llvm-svn: 175309

Daniel Malea caught an issue where calling dotest.py with an invalid directory would cause the progressbar init code to raise an exception
This commit fixes it

llvm-svn: 175229

The new progress bar mode was losing us information compared to the old dots mode in that we would have no way of knowing about test failures (short of peeking into the test result directory.. and you're not supposed to peek!)

Added a new line of information that reports the count of tests that pass, fail or have other things happen to them.

Again no flag to have the dots back. If you care, let us know!

llvm-svn: 174784

The LLDB test suite now shows a progress bar instead of dots when not in verbose mode
If you crave the dots, make your Terminal window smaller than 10 columns :-)
(or ask for a flag to have the dots come back on demand)

llvm-svn: 174777

Unset the environment variables as soon as possible when running the test suite. Also don't store the unset list into a global when they aren't needed in a global variable.

llvm-svn: 174750

- resolves errors in cases that run the lldb CLI utility

llvm-svn: 174522

- now prints the correct PYTHONPATH
- update dotest.py to use lldb -P result correctly
- resolves TestPublicAPIHeaders test failure (on Linux)

llvm-svn: 171558

Added an SBAPI to get the PythonPath (if the Host knows how to do that).  And a -P option to the Driver
to print it out.  Changed dotest.py to use that to find the PythonPath it should use given the lldb binary
it was told to run.

llvm-svn: 170932

./dotest.py  -C clang --arch x86_64 --arch i386  -v -t -f ObjCDataFormatterTestCase.test_appkit_with_dsym_and_run_command -f ObjCDataFormatterTestCase.test_appkit_with_dwarf_and_run_command -f TestObjCBuiltinTypes.test_with_dsym_and_python_api -f TestObjCBuiltinTypes.test_with_dwarf_and_python_api -f ObjCDataFormatterTestCase.test_appkit_with_dsym_and_run_command -f ObjCDataFormatterTestCase.test_appkit_with_dwarf_and_run_command -f TestObjCBuiltinTypes.test_with_dsym_and_python_api -f -TestObjCBuiltinTypes.test_with_dwarf_and_python_api

The previous implementation would only remember the last filter passed, and consequently break redo.py

llvm-svn: 170163

llvm-svn: 168370

explicitly pointed at an LLDB framework and
executable.

llvm-svn: 167250

to dotest.py so that the testsuite can run against
any LLDB.

<rdar://problem/12512268>

llvm-svn: 166635

Better error message for invalid argument to --category - silenced the printout of the raw args when not in verbose mode

llvm-svn: 166514

llvm-svn: 166453

testsuite.

llvm-svn: 164723

This feature allows us to group test cases into logical groups (categories), and to only run a subset of test cases based on these categories.

Each test-case can have a new method getCategories(self): which returns a list of strings that are the categories to which the test case belongs.
If a test-case does not provide its own categories, we will look for categories in the class that contains the test case.
If that fails too, the default implementation looks for a .category file, which contains a comma separated list of strings.
The test suite will recurse look for .categories up until the top level directory (which we guarantee will have an empty .category file).

The driver dotest.py has a new --category <foo> option, which can be repeated, and specifies which categories of tests you want to run.
(example: ./dotest.py --category objc --category expression)

All tests that do not belong to any specified category will be skipped. Other filtering options still exist and should not interfere with category filtering.
A few tests have been categorized. Feel free to categorize others, and to suggest new categories that we could want to use.

All categories need to be validly defined in dotest.py, or the test suite will refuse to run when you use them as arguments to --category.

In the end, failures will be reported on a per-category basis, as well as in the usual format.

This is the very first stage of this feature. Feel free to chime in with ideas for improvements!

llvm-svn: 164403

Patch from Filipe Cabecinhas that uses argparse in dotest.py instead of a hand coded option. I made a few modifications:

Changed the '-A' option to also have a long option of '--arch'. This is now specified multiple times to get multiple architectures.

Old: -A i386^x86_64
New: -A i386 -A x86_64
     --arch i386 --arch x86_64
     
Changed the '-C' option to also have a long option of '--compiler'. This is now specified multiple times to get multiple compiler.

Old: -C clang^gcc
New: -C clang -C gcc
     --compiler clang --compiler gcc
llvm-svn: 163141

llvm-svn: 162372

llvm-svn: 162040

llvm-svn: 161493

llvm-svn: 161492

    ./dotest.py

No progress bar appears.

llvm-svn: 157226

Enrico will follow this up with fixing the data formatter test cases that are failing.

llvm-svn: 155514

Add a '-R' option, which is similar to '-r', except that the relocated directory, if exists, will be removed entirely
before running the test suite.  A usage example looks like this:

test $ ./dotest.py -A x86_64 -R /tmp/x86_64 &
test $ ./dotest.py -A i386 -R /tmp/i386 &

where we would want to run the x86_64 and i386 archs concurrently but relocate the test suite to different directory
hierarchies in order not to stump on each other's intermediate files.

llvm-svn: 155491

rdar://problem/11283401

Example:

Collected 1 test

1: test_with_dwarf (TestCallStdStringFunction.ExprCommandCallFunctionTestCase)
   Test calling std::String member function. ... FAIL

======================================================================
FAIL: test_with_dwarf (TestCallStdStringFunction.ExprCommandCallFunctionTestCase)
   Test calling std::String member function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/ToT/test/lldbtest.py", line 427, in wrapper
    return func(self, *args, **kwargs)
  File "/Volumes/data/lldb/svn/ToT/test/expression_command/call-function/TestCallStdStringFunction.py", line 34, in test_with_dwarf
    self.call_function()
  File "/Volumes/data/lldb/svn/ToT/test/expression_command/call-function/TestCallStdStringFunction.py", line 48, in call_function
    substrs = ['Hello world'])
  File "/Volumes/data/lldb/svn/ToT/test/lldbtest.py", line 1235, in expect
    msg if msg else EXP_MSG(str, exe))
AssertionError: False is not True : 'Hello world' returns expected result
Config=i386-clang
----------------------------------------------------------------------
Ran 1 test in 1.148s

FAILED (failures=1)

llvm-svn: 155157

llvm-svn: 155152

the pre-flight code gets executed during setUp() after the debugger instance is available
and the post-flight code gets executed during tearDown() after the debugger instance has
done killing the inferior and deleting all the target programs.

Example:

[11:32:48] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight  functionalities/watchpoint/hello_watchpoint
config: {'pre_flight': <function pre_flight at 0x1098541b8>, 'post_flight': <function post_flight at 0x109854230>}
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-139
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154753
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154730
Last Changed Date: 2012-04-13 18:42:46 -0700 (Fri, 13 Apr 2012)


lldb.pre_flight: def pre_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning pre-flight function:"
    print "for test case:", test

lldb.post_flight: def post_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning post-flight function:"
    print "for test case:", test


Session logs for test failures/errors/unexpected successes will go into directory '2012-04-16-11_34_08'
Command invoked: python ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight functionalities/watchpoint/hello_watchpoint
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok

----------------------------------------------------------------------
Ran 2 tests in 1.584s

OK

llvm-svn: 154847

rdar://problem/11031264

llvm-svn: 154573

either @dsym_test or @dwarf_test to be executed during the testsuite run.  There are still lots of
Test*.py files which have not been decorated with the new decorator.

An example:

# From TestMyFirstWatchpoint.py ->
class HelloWatchpointTestCase(TestBase):

    mydir = os.path.join("functionalities", "watchpoint", "hello_watchpoint")

    @dsym_test
    def test_hello_watchpoint_with_dsym_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDsym(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()

    @dwarf_test
    def test_hello_watchpoint_with_dwarf_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDwarf(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()


# Invocation ->
[17:50:14] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-137
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154133
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154109
Last Changed Date: 2012-04-05 10:43:02 -0700 (Thu, 05 Apr 2012)



Session logs for test failures/errors/unexpected successes will go into directory '2012-04-05-17_50_49'
Command invoked: python ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... skipped 'dsym tests'
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... ok

----------------------------------------------------------------------
Ran 2 tests in 1.138s

OK (skipped=1)
Session logs for test failures/errors/unexpected successes can be found in directory '2012-04-05-17_50_49'
[17:50:50] johnny:/Volumes/data/lldb/svn/ToT/test $ 

llvm-svn: 154154

to pass to the toolchain in order to build the inferior programs to be run/debugged
duirng the test suite.  The architecture might dictate some special CFLAGS which are
more easily specified in a central place (like the command line) instead of inside
make rules.

For Example,

./dotest.py -v -r /shared/phone -A armv7 -E "-isysroot your_sdk_root" functionalities/watchpoint/hello_watchpoint

will relocate the particular test directory ('functionalities/watchpoint/hello_watchpoint' in this case) to a
new directory named '/shared/phone'.  The particular incarnation of the architecture-compiler combination of the
test support files are therefore to be found under:

/shared/phone.arch=armv7-compiler=clang/functionalities/watchpoint/hello_watchpoint

The building of the inferior programs under testing is now working.

The actual launching/debugging of the inferior programs are not yet working,
neither is the setting of a watchpoint on the phone.

llvm-svn: 153070

llvm-svn: 152568