Add an attribute __python_api_test__ (set to True) to the @python_api_test decorated
test method to distinguish them from the lldb command line tests.

llvm-svn: 121500

Example:

    @python_api_test
    def test_evaluate_expression_python(self):
        """Test SBFrame.EvaluateExpression() API for evaluating an expression."""
    ...

The opposite of Python APIs only test is an lldb command line test, which sends
commands to the lldb command interpreter.  Add a '-a' option to the test driver
to skip Python API only tests.

Modify TestExprs.py to mark a test as @python_api_test and remove an @expectedFailure
decorator as the bug has been fixed.

llvm-svn: 121442

as the args and the envs to the launched process.

o lldbtest.py:

  Forgot to check in some assertion messages changes for lldbtest.py.

o dotest.py:

  Also add "api" category to the default lldb log option list.

llvm-svn: 121220

As soon as an error or a failure is encountered, it stops the test suite.

llvm-svn: 120837

llvm-svn: 120751

log-categories, so it now becomes "event process expr state".

llvm-svn: 120745

o lldb => event process expr
o gdb-remote => packets process

llvm-svn: 120725

test classes or test cases to be excludued from the test suite.

Check in an example blacklist file: blacklist.py:

"""
'blacklist' is a Python dictionary, it stores the mapping of a string describing
either a testclass or a testcase, i.e, testclass.testmethod, to the reason (a
string) it is blacklisted.

Following is an example which states that test class IntegerTypesExprTestCase
should be skipped because 'This test class crashed' and the test case
FoundationTestCase.test_data_type_and_expr_with_dsym should be skipped because
it is 'Temporarily disabled'.

blacklist = {'IntegerTypesExprTestCase': 'This test class crashed',
             'FoundationTestCase.test_data_type_and_expr_with_dsym': 'Temporarily disabled'
             }
"""

blacklist = {}

An example of invoking the test driver and specifying a blacklist file:

./dotest.py -b blacklist.py -v types

This runs the tests under 'types' directory but excludes the tests specified in
balcklist.py.

llvm-svn: 120620

llvm-svn: 120295

ignore and not enforce the signleton pattern for the LLDBTestResult class.

llvm-svn: 120294

This is not to be used during normal test suite run, but to be used to stress test
specific test sequences repeatedly.

Example:

./dotest.py -# 3 -v breakpoint_conditions

will repeat the test suite 3 times for tests under the breakpoint_conditions directory.

llvm-svn: 119399

to invoke the test driver.

llvm-svn: 118847

llvm-svn: 118655

llvm-svn: 118650

llvm-svn: 118438

failed/errored tests at the start of the test run, in case the test run crashes
for any reason.  That way, it is easy to locate the session logs for accumulated
failures/errors.

llvm-svn: 118427

we want to run just the instance of testclass-testmethod combination and nothing else.

Specifying '-g' now will admit the whole .py test module if it does not contain a matching
testclass-testmethod combination at all.

This option arrangement adheres to the RISC principle of making the common cases fast. :-)

rdar://problem/8584914 Can the default for dotest.py's "-g" flag be switched?

llvm-svn: 118392

llvm-svn: 118279

idea to also dump the session information for expected failures in addition to
failures and errors (tests with unexpected exceptions).

Modify the LLDBTestResult class which derives from unittest2.TextTestResult to
intercept the addExpectedFailure() method in order to mark the test instance as
needing its session information dumped.

llvm-svn: 118185

llvm-svn: 117730

tests should be near the test summary message, instead of way up in the beginning.

llvm-svn: 117726

File name "2010-10-19-14:10:49.059609" is non-portable.

Do not use ':' in the pathname as it is not permitted on windows platforms.

llvm-svn: 117556

llvm-svn: 117129

llvm-svn: 117030

directory used to dump the session info for test failures/errors.

Example:

/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -s jason -v array_types

Session info for test errors or failures will go into directory jason
----------------------------------------------------------------------
Collected 4 tests

test_with_dsym_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dsym_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... ok
test_with_dwarf_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... FAIL

======================================================================
FAIL: test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 27, in test_with_dwarf_and_run_command
    self.array_types()
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 62, in array_types
    'stop reason = breakpoint'])
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 594, in expect
    self.runCmd(str, trace = (True if trace else False), check = not error)
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 564, in runCmd
    msg if msg else CMD_MSG(cmd, True))
AssertionError: False is not True : Command 'thread list' returns successfully

----------------------------------------------------------------------
Ran 4 tests in 3.086s

FAILED (failures=1)
/Volumes/data/lldb/svn/trunk/test $ ls jason
TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
/Volumes/data/lldb/svn/trunk/test $ head -10 jason/TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log 
Session info generated @ Thu Oct 21 09:54:15 2010

os command: [['/bin/sh', '-c', 'make clean; make MAKE_DSYM=NO']]
stdout: rm -rf "a.out" "a.out.dSYM"  main.o main.d
cc -arch x86_64 -gdwarf-2 -O0   -c -o main.o main.c
cc -arch x86_64 -gdwarf-2 -O0  main.o -o "a.out"

stderr: None
retcode: 0

/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 117028

session information files to.  This makes the directory name less intimidating.
Currently, the directory only gets created if there are failures/errors while
running the test suite.

llvm-svn: 116982

llvm-svn: 116980

now goes into a timestamp-specific directory instead of the previous .session-*
files.

[17:24:34] johnny:/Volumes/data/lldb/svn/trunk $ ls -l test/2010-10-18-16:56:12.935342
total 48
-rw-r--r--  1 johnny  admin  1695 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dsym_and_run_command.log
-rw-r--r--  1 johnny  admin  1652 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  2967 Oct 18 16:56 TestBreakpointCommand.BreakpointCommandTestCase.test_with_dsym.log
-rw-r--r--  1 johnny  admin  1648 Oct 18 16:56 TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  1665 Oct 18 16:56 TestClassTypesDisassembly.IterateFrameAndDisassembleTestCase.test_with_dsym_and_python_api.log
-rw-r--r--  1 johnny  admin  3873 Oct 18 16:58 TestFloatTypesExpr.FloatTypesTestCase.test_float_types_with_dsym.log
[17:24:37] johnny:/Volumes/data/lldb/svn/trunk $ 

Also, the dumping happens when a test errored in additioned to when it failed.

llvm-svn: 116778

llvm-svn: 116559

session info after a test case failure, allowing more direct inspection of
debugger session which leads to the test failure.

For a simple usage scenario:

[18:06:26] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log 

...

[18:14:43] johnny:/Volumes/data/lldb/svn/trunk/test $ ls -l .session-*
-rw-r--r--  1 johnny  admin  1359 Oct 14 18:06 .session-TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command
-rw-r--r--  1 johnny  admin  2054 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dsym_and_expr_parser
-rw-r--r--  1 johnny  admin  2055 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_expr_parser
-rw-r--r--  1 johnny  admin  1351 Oct 14 17:57 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command
[18:14:51] johnny:/Volumes/data/lldb/svn/trunk/test $ 

The test case which failed will have its recorded session info dumped to a
.session-* file in the current working directory.  For test suite using
relocated directory, expect to find the .session-* files there.

In this checkin, I also add @skip decorator to the two test methods in
test/foundation/TestObjCMethods.py as it looks like the test suite is
deadlocking when running the tests.  More investigations are needed.

llvm-svn: 116552

Inside the lldb module, there's no need (and as a matter of fact, incorrect) to specify the 'lldb'
module name.

Comment out the call to lldb.SBDebugger.Initialize() within the test driver itself, since it is
already done when we import the lldb.py module.

llvm-svn: 116485

llvm-svn: 116343

different configuration-based files using the config file.  For example:

    sys.stderr = open("/tmp/lldbtest-stderr", "w")
    sys.stdout = open("/tmp/lldbtest-stdout", "w")
    compilers = ["gcc", "llvm-gcc"]
    archs = ["x86_64", "i386"]
    split_stderr = True # This will split the stderr into configuration-specific file
    split_stdout = True # This will split the stdout into configuration-specific file

will produce:

/tmp/lldbtest-stderr
/tmp/lldbtest-stderr.arch=i386-compiler=gcc
/tmp/lldbtest-stderr.arch=i386-compiler=llvm-gcc
/tmp/lldbtest-stderr.arch=x86_64-compiler=gcc
/tmp/lldbtest-stderr.arch=x86_64-compiler=llvm-gcc
/tmp/lldbtest-stdout
/tmp/lldbtest-stdout.arch=i386-compiler=gcc
/tmp/lldbtest-stdout.arch=i386-compiler=llvm-gcc
/tmp/lldbtest-stdout.arch=x86_64-compiler=gcc
/tmp/lldbtest-stdout.arch=x86_64-compiler=llvm-gcc

as a result of splitting stderr and stdout.  In addition, each configuration can have
its individual top level relocated directory to house the test files as well as the
intermediate files by using '-r dir' to relocate the tests into a new relocated directory
instead of running the tests in place.

llvm-svn: 116341

llvm-svn: 116309

files to a different top level directory than those specified on the command line.

When relocated, the test clanups normally performed afterwards after each test method
and after each test class will not be exercised at all.  This allows for an easier
postmortem analysis of test failures.

Example:

./dotest.py -v -t -r /tmp/lldbtest types

will create a /tmp/lldbtest directory which houses the types directory and its supported
files.

Files modified:

o dotest.py, lldbtest.py:

  Add logic to process '-r dir' option to support relocating the tests to a different
  top level directory instead of exected in place.

o darwin.py, test/types/Makefile:

  The 'make clean' should only clean the minimum .o and .d files.

llvm-svn: 116255

llvm-svn: 116206

llvm-svn: 115916

  Temporarily commenting out the deprecated LaunchProcess() method.
  SWIG is not able to handle the overloaded functions.

o dotest.py/lldbtest.py:

  Add an '-w' option to insert some wait time between consecutive test cases.

o TestClassTypes.py:

  Make the breakpoint_creation_by_filespec_python() test method more robust and
  more descriptive by printing out a more insightful assert message.

o lldb.swig: Coaches swig to treat StateType as an int type, instead of a C++ class.

llvm-svn: 115899

testclass.testmethod to be run and with a '-g' option which instructs the test
driver to only admit the module which satisfy the filterspec condition to the
test suite.

Example:

# This only runs the test case under the array_types directory which has class
# name of 'ArrayTypesTestCase' and the test method name of 'test_with_dwarf_and_run_command'.

/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v -f 'ArrayTypesTestCase.test_with_dwarf_and_run_command' -g array_types
----------------------------------------------------------------------
Collected 1 test

test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.353s

OK

# And this runs the test cases under the array_types and the hello_world directories.
# If the module discovered has the 'ArrayTypesTestCase.test_with_dwarf_and_run_command'
# attribute, only the test case specified by the filterspec for the module will be run.
# If the module does not have the said attribute, e.g., the module under hello_world,
# the whole module is still admitted to the test suite.

/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v -f 'ArrayTypesTestCase.test_with_dwarf_and_run_command' array_types hello_world
----------------------------------------------------------------------
Collected 3 tests

test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... ok
test_with_dsym_and_run_command (TestHelloWorld.HelloWorldTestCase)
Create target, breakpoint, launch a process, and then kill it. ... ok
test_with_dwarf_and_process_launch_api (TestHelloWorld.HelloWorldTestCase)
Create target, breakpoint, launch a process, and then kill it. ... ok

----------------------------------------------------------------------
Ran 3 tests in 4.964s

OK

llvm-svn: 115832

  To not skip long running tests, pass '-l' to the test driver (dotest.py).

  An example:

    @unittest2.skipIf(TestBase.skipLongRunningTest(), "Skip this long running test")
    def test_foundation_disasm(self):
        ...

o Added a long running disassemble test to the foundation directory, which iterates
  the code symbols from Foundation.framework and kicks off a disassemble command for
  for the named function symbol.  Found a crasher: rdar://problem/8504895.

o Plus added/updated some comments for the TestBase class.

llvm-svn: 115368