Fixing issues in previous checkin - still figuring out how to make expectedFailureClang take the bugnumber

llvm-svn: 175945

The decorators @expectedFailure (plain and special-case like i386, clang, ...) are modified to optionally take a bugnumber argument
If such an argument is specified, the failure report (or unexpected success report) will include the information passed in as part of the message
This is mostly useful for associating failures to issue IDs in issue management systems (e.g. the LLVM bugzilla)

llvm-svn: 175942

- replace "catch-all" except clause with one that specifically catches what pexpect throws
- handle case where child is already terminated (or is terminating) by the time tear-down is run

llvm-svn: 175844

- rework the way SBDebugger.SetAsync() is used to avoid side effects (reset original value at TearDownHook)
- refactor expectedFailureClang (and add expectedFailureGcc decorator)
- mark TestChangeValueAPI.py as expectedFailureGcc due to PR-15039

llvm-svn: 175523

- fixed cleanup of Popen objects by pushing spawn logic into test Base and out of test cases
- connect subprocess stdin to PIPE (rather than the parent's STDIN) to fix silent terminal issue

Tested on Linux and Mac OS X

llvm-svn: 175301

- set auto-confirm to false when running TestExprs (avoid hang when using API)
- set prompt-on-quit to false in test helper (avoid timeout when using lldb CLI)

llvm-svn: 173485

- Add new decorator "@skipIfGcc" to lldbtest.py

llvm-svn: 173394

The skipOnLinux decorator wasn't calling the test method correctly (no need to pass in the "self") resulting
in errors on MacOS X for the tests so decorated.

llvm-svn: 168662

- use lldb 'settings' command to help testcase find shared library
- pull up dyldPath variable from TestLoadUnload.py to fixture base class (applicable in multiple cases)

llvm-svn: 168612

- add decorators @expectedFailLinux and @skipOnLinux
- skip/mark xfail cases due to open bugzillas # 14323, 14416, 14423, 14424, 14425, 14426

Patch by Ashok Thirumurthi!

llvm-svn: 168529

llvm-svn: 166628

llvm-svn: 166627

llvm-svn: 166626

llvm-svn: 166604

<rdar://problem/12481949> Fixing SBValue.GetValueAsSigned() to do the right thing when dealing with a 32-bit negative value

llvm-svn: 166603

Changed all relevant test cases to verify that MightHaveChildren() works correctly for objects of interest
Added a bunch of convenience methods for test cases to use: target(), process(), thread() and frame() which mimic the lldb.X convenience variables
As a bonus, edited the documentation on the website to describe the new method available for synthetic children providers writers to implement!

That's all folks!

llvm-svn: 166535

llvm-svn: 166453

command line to dotest.py, replace / with _ in
the logfile names that mention that compiler so
that we don't try to put log files in weird
places.

llvm-svn: 166038

Fix all the test case breakages caused by folks writing tests all over the place that depended explicitly 
on the output of "break set".  Please don't do this sort of thing!!!!!

llvm-svn: 164433

This feature allows us to group test cases into logical groups (categories), and to only run a subset of test cases based on these categories.

Each test-case can have a new method getCategories(self): which returns a list of strings that are the categories to which the test case belongs.
If a test-case does not provide its own categories, we will look for categories in the class that contains the test case.
If that fails too, the default implementation looks for a .category file, which contains a comma separated list of strings.
The test suite will recurse look for .categories up until the top level directory (which we guarantee will have an empty .category file).

The driver dotest.py has a new --category <foo> option, which can be repeated, and specifies which categories of tests you want to run.
(example: ./dotest.py --category objc --category expression)

All tests that do not belong to any specified category will be skipped. Other filtering options still exist and should not interfere with category filtering.
A few tests have been categorized. Feel free to categorize others, and to suggest new categories that we could want to use.

All categories need to be validly defined in dotest.py, or the test suite will refuse to run when you use them as arguments to --category.

In the end, failures will be reported on a per-category basis, as well as in the usual format.

This is the very first stage of this feature. Feel free to chime in with ideas for improvements!

llvm-svn: 164403

You run with trace mode by passing '-t' to the test driver.

llvm-svn: 161130

Adds a utility class method to TestBase that checks and removes a temp
file.
Removed every use of system() to execute rm -f.

llvm-svn: 158809

llvm-svn: 156943

to the already existing (test result, test id) to avoid collision and to
facilitate postmortem analysis.

llvm-svn: 155148

the pre-flight code gets executed during setUp() after the debugger instance is available
and the post-flight code gets executed during tearDown() after the debugger instance has
done killing the inferior and deleting all the target programs.

Example:

[11:32:48] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight  functionalities/watchpoint/hello_watchpoint
config: {'pre_flight': <function pre_flight at 0x1098541b8>, 'post_flight': <function post_flight at 0x109854230>}
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-139
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154753
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154730
Last Changed Date: 2012-04-13 18:42:46 -0700 (Fri, 13 Apr 2012)


lldb.pre_flight: def pre_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning pre-flight function:"
    print "for test case:", test

lldb.post_flight: def post_flight(test):
    __import__("lldb")
    __import__("lldbtest")
    print "\nRunning post-flight function:"
    print "for test case:", test


Session logs for test failures/errors/unexpected successes will go into directory '2012-04-16-11_34_08'
Command invoked: python ./dotest.py -A x86_64 -v -c ../examples/test/.lldb-pre-post-flight functionalities/watchpoint/hello_watchpoint
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... 
Running pre-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)

Running post-flight function:
for test case: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
ok

----------------------------------------------------------------------
Ran 2 tests in 1.584s

OK

llvm-svn: 154847

either @dsym_test or @dwarf_test to be executed during the testsuite run.  There are still lots of
Test*.py files which have not been decorated with the new decorator.

An example:

# From TestMyFirstWatchpoint.py ->
class HelloWatchpointTestCase(TestBase):

    mydir = os.path.join("functionalities", "watchpoint", "hello_watchpoint")

    @dsym_test
    def test_hello_watchpoint_with_dsym_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDsym(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()

    @dwarf_test
    def test_hello_watchpoint_with_dwarf_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDwarf(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()


# Invocation ->
[17:50:14] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-137
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154133
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154109
Last Changed Date: 2012-04-05 10:43:02 -0700 (Thu, 05 Apr 2012)



Session logs for test failures/errors/unexpected successes will go into directory '2012-04-05-17_50_49'
Command invoked: python ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... skipped 'dsym tests'
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... ok

----------------------------------------------------------------------
Ran 2 tests in 1.138s

OK (skipped=1)
Session logs for test failures/errors/unexpected successes can be found in directory '2012-04-05-17_50_49'
[17:50:50] johnny:/Volumes/data/lldb/svn/ToT/test $ 

llvm-svn: 154154

llvm-svn: 152941

llvm-svn: 152825

During the test case tearDown(), give it one final blow to make sure the child process spawned by pexpect is terminated.
Fix the issue of many '(lldb)' zombie processes observed by Jim.

llvm-svn: 151583

llvm-svn: 150133

llvm-svn: 149707

o Symbols.cpp:

  Emit a warning message when dSYM does not match the binary.

o warnings/uuid:

  Added regression test case.

o lldbtest.py:

  Modified to allow test case writer to demand that the build command does not begin
  with a clean first; required to make TestUUIDMismatchWanring.py work.

rdar://problem/10515708

llvm-svn: 149465

  Fix a bug where "settings set -r th" wouldn't complete.

o UserSettingsController.cpp:

  Fix a bug where "settings set target.process." wouldn't complete.

o test/functionalities/completion:

  Add various completion test cases related to 'settings set' command.

llvm-svn: 148596

Decorate the two test cases in TestReturnValue.py as i386 only expectedFailure, aka @expectedFailurei386.

llvm-svn: 147177

llvm-svn: 145732

Use this option with care as you would need to build the inferior(s) by hand
and build the executable(s) with the correct name(s).  This option can be used
with '-# n' to stress test certain test cases for n number of times.

An example:

[11:55:11] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ ls
Makefile		TestValueAPI.pyc	linked_list
TestValueAPI.py		change_values		main.c
[11:55:14] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ make EXE=test_with_dsym
clang -gdwarf-2 -O0  -arch x86_64   -c -o main.o main.c
clang -gdwarf-2 -O0  -arch x86_64   main.o -o "test_with_dsym"
/usr/bin/dsymutil  -o "test_with_dsym.dSYM" "test_with_dsym"
[11:55:20] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ cd ../..
[11:55:24] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
LLDB build dir: /Volumes/data/lldb/svn/trunk/build/Debug
LLDB-89
Path: /Volumes/data/lldb/svn/trunk
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 144914
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 144911
Last Changed Date: 2011-11-17 09:22:31 -0800 (Thu, 17 Nov 2011)



Session logs for test failures/errors/unexpected successes will go into directory '2011-11-17-11_55_29'
Command invoked: python ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
----------------------------------------------------------------------
Collected 1 test

1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.163s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.200s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.198s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.199s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.239s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.215s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.105s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.098s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.195s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.197s

OK
[11:55:34] johnny:/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 144919

llvm-svn: 144848

This helps track down possible zombie processes.

llvm-svn: 144846

    child=None, child_prompt=None, use_cmd_api=False

By default, expect a pexpect spawned child and child prompt to be
supplied (use_cmd_api=False).  If use_cmd_api is true, ignore the child
and child prompt and use self.runCmd() to run the hooks one by one.

Modify existing client to reflect the change.

llvm-svn: 142532

llvm-svn: 142469