cases when specifying '-w' to the test driver, by using an environment variable
LLDB_TIME_WAIT_BETWEEN_TEST_CASES.

llvm-svn: 120625

before iterating on it.

llvm-svn: 120623

test classes or test cases to be excludued from the test suite.

Check in an example blacklist file: blacklist.py:

"""
'blacklist' is a Python dictionary, it stores the mapping of a string describing
either a testclass or a testcase, i.e, testclass.testmethod, to the reason (a
string) it is blacklisted.

Following is an example which states that test class IntegerTypesExprTestCase
should be skipped because 'This test class crashed' and the test case
FoundationTestCase.test_data_type_and_expr_with_dsym should be skipped because
it is 'Temporarily disabled'.

blacklist = {'IntegerTypesExprTestCase': 'This test class crashed',
             'FoundationTestCase.test_data_type_and_expr_with_dsym': 'Temporarily disabled'
             }
"""

blacklist = {}

An example of invoking the test driver and specifying a blacklist file:

./dotest.py -b blacklist.py -v types

This runs the tests under 'types' directory but excludes the tests specified in
balcklist.py.

llvm-svn: 120620

the process status is indeed 'stopped' before invoking 'frame variable -t val'
command to inspect the frame variable.  When I run the test suite, it sometimes
emits an output like:

runCmd: frame variable -t val
runCmd failed!
error: you must be stopped in a valid stack frame to view frame variables.

The expect() statement inserted is to make sure that the process is stopped and
we have a valid frame.

llvm-svn: 120318

Increase the delta between successive test cases, enabled with the '-w' option to
the test driver, from 0.5 to 1.0 second, to arrive at a cleaner state before the
next test case.

llvm-svn: 120315

output from clang and llvm-gcc compiled program; both generate the correct debug
info with respect to the typedef scoped inside a namespace.

Add a TestBase.getCompiler(self) method which returns the compiler in effect the
test suite is now running with.  Subclasses (like TestNamespace) can use it to
distinguish among different compilers.

llvm-svn: 119445

the architecture and compiler specs.

llvm-svn: 118860

llvm-svn: 118739

test method when asserting:

    thread.GetStopReason() == lldb.eStopReasonBreakpoint

llvm-svn: 118711

invoke the test driver to rerun the very same test.  Example output:

/Volumes/data/lldb/svn/trunk/test $ tail 2010-11-09-14_51_34/ExpectedFailure-TestSettings.SettingsCommandTestCase.test_set_output_path.log 

Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/trunk/test/settings/TestSettings.py", line 136, in test_set_output_path
    "'stdout.txt' exists due to target.process.output-path.")
AssertionError: False is not True : 'stdout.txt' exists due to target.process.output-path.


To rerun this test, issue the following command from the 'test' directory:

./dotest.py -v -t -f SettingsCommandTestCase.test_set_output_path

llvm-svn: 118646

takes the form:

    "Command '%s' returns successfully" % str

and the latter takes the form:

    "'%s' returns expected result" % str
    or
    "'%s' matches expected result" % str

llvm-svn: 118599

identification of the test failures/errors by human beings as well as automatic
processings.

The prefix which identifies the type can be: Error, Failure, or ExpectedFailure.

llvm-svn: 118315

llvm-svn: 118285

local variable and an argument variable.

Add ValueTypeString() utility function into lldbutil.py which converts the enum
into a human readable string.

Modify TestBase.DebugSBValue() to also dump the value type of an SBValue object.

llvm-svn: 118197

idea to also dump the session information for expected failures in addition to
failures and errors (tests with unexpected exceptions).

Modify the LLDBTestResult class which derives from unittest2.TextTestResult to
intercept the addExpectedFailure() method in order to mark the test instance as
needing its session information dumped.

llvm-svn: 118185

llvm-svn: 118129

getsource_if_available(obj) which also handles the exception if it occurs.

llvm-svn: 117949

                    SBFrame print representation: idx: 0 -> #0

llvm-svn: 117924

llvm-svn: 117604

If TestBase.expect() is passed an assert message, pass it along when delegating to TestBase.runCmd()
to execute the debugger command.  This will result in a more semantic assert message than just the
command failed message like before:

AssertionError: False is not True : Command 'thread list' returns successfully

And now, we will have:

AssertionError: False is not True : Process state is stopped due to breakpoint
llvm-svn: 117569

llvm-svn: 117175

llvm-svn: 117116

directory used to dump the session info for test failures/errors.

Example:

/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -s jason -v array_types

Session info for test errors or failures will go into directory jason
----------------------------------------------------------------------
Collected 4 tests

test_with_dsym_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dsym_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... ok
test_with_dwarf_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... FAIL

======================================================================
FAIL: test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 27, in test_with_dwarf_and_run_command
    self.array_types()
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 62, in array_types
    'stop reason = breakpoint'])
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 594, in expect
    self.runCmd(str, trace = (True if trace else False), check = not error)
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 564, in runCmd
    msg if msg else CMD_MSG(cmd, True))
AssertionError: False is not True : Command 'thread list' returns successfully

----------------------------------------------------------------------
Ran 4 tests in 3.086s

FAILED (failures=1)
/Volumes/data/lldb/svn/trunk/test $ ls jason
TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
/Volumes/data/lldb/svn/trunk/test $ head -10 jason/TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log 
Session info generated @ Thu Oct 21 09:54:15 2010

os command: [['/bin/sh', '-c', 'make clean; make MAKE_DSYM=NO']]
stdout: rm -rf "a.out" "a.out.dSYM"  main.o main.d
cc -arch x86_64 -gdwarf-2 -O0   -c -o main.o main.c
cc -arch x86_64 -gdwarf-2 -O0  main.o -o "a.out"

stderr: None
retcode: 0

/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 117028

parent call frame information.  And comment out the check for stop reason for the
time being.  The stop reason disappeared from the "thread backtrace" output for
breakpoint stop with condition.

llvm-svn: 116939

invocations of them into session object.

Remove a debug statement.

llvm-svn: 116938

to be run during tearDown() to effect the restore action instead of executing it inline
during the test method, because the test may already fail and bailout before the inline
restore action.

Fix test_set_output_path() and pass_run_args_and_env_vars() to use this mechanism.

llvm-svn: 116881

llvm-svn: 116874

Add a little helper function SETTING_MSG(setting) to be used from TestSettings.test_set_output_path().

llvm-svn: 116847

Change the environment variable name to be more descriptive from LLDB_TIME_WAIT to LLDB_TIME_WAIT_NEXT_LAUNCH.

llvm-svn: 116809

now goes into a timestamp-specific directory instead of the previous .session-*
files.

[17:24:34] johnny:/Volumes/data/lldb/svn/trunk $ ls -l test/2010-10-18-16:56:12.935342
total 48
-rw-r--r--  1 johnny  admin  1695 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dsym_and_run_command.log
-rw-r--r--  1 johnny  admin  1652 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  2967 Oct 18 16:56 TestBreakpointCommand.BreakpointCommandTestCase.test_with_dsym.log
-rw-r--r--  1 johnny  admin  1648 Oct 18 16:56 TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  1665 Oct 18 16:56 TestClassTypesDisassembly.IterateFrameAndDisassembleTestCase.test_with_dsym_and_python_api.log
-rw-r--r--  1 johnny  admin  3873 Oct 18 16:58 TestFloatTypesExpr.FloatTypesTestCase.test_float_types_with_dsym.log
[17:24:37] johnny:/Volumes/data/lldb/svn/trunk $ 

Also, the dumping happens when a test errored in additioned to when it failed.

llvm-svn: 116778

facilitate session recording.  This happens inside an instance method where the
test instance is well known.

llvm-svn: 116649

pass in a 'sender' arg to the buildDefault(), buildDsym(), buildDwarf(), and
cleanup() functions.  The sender arg will be the test instance itself (i.e.,
an instance of TestBase).  This is so that the relevant command execution can be
recorded in the TestBase.session object if sender is available.

The lldbtest.system() command has been modified to pop the 'sender' arg out of
the keyword arguments dictionary and use it as the test instance to facilitate
seesion recordings.  An example is in test/types/AbstractBase.py:

    def generic_type_tester(self, atoms, quotedDisplay=False):
        """Test that variables with basic types are displayed correctly."""

        # First, capture the golden output emitted by the oracle, i.e., the
        # series of printf statements.
        go = system("./a.out", sender=self)

There are cases when sender is None.  This is the case when the @classmethod is
involved in the use of these APIs.  When this happens, there is no recording
into a session object, but printing on the sys.stderr is still honored if the
trace flag is ON.

An example is in test/settings/TestSettings.py:

    @classmethod
    def classCleanup(cls):
        system(["/bin/sh", "-c", "rm -f output.txt"])
        system(["/bin/sh", "-c", "rm -f stdout.txt"])

llvm-svn: 116648

    breakpoint modify -c 'val == 3' 1

after:

    breakpoint set -n c

which sets a breakpoint on function 'c'.  The breakpoint should only stop if
expression 'val == 3' evaluates to true.

llvm-svn: 116607

the session recordings.

llvm-svn: 116596

llvm-svn: 116583

is no need to check the return status of the command execution, and an error
status is not deemed a failure in the test.

llvm-svn: 116582

llvm-svn: 116559

session info after a test case failure, allowing more direct inspection of
debugger session which leads to the test failure.

For a simple usage scenario:

[18:06:26] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log 

...

[18:14:43] johnny:/Volumes/data/lldb/svn/trunk/test $ ls -l .session-*
-rw-r--r--  1 johnny  admin  1359 Oct 14 18:06 .session-TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command
-rw-r--r--  1 johnny  admin  2054 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dsym_and_expr_parser
-rw-r--r--  1 johnny  admin  2055 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_expr_parser
-rw-r--r--  1 johnny  admin  1351 Oct 14 17:57 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command
[18:14:51] johnny:/Volumes/data/lldb/svn/trunk/test $ 

The test case which failed will have its recorded session info dumped to a
.session-* file in the current working directory.  For test suite using
relocated directory, expect to find the .session-* files there.

In this checkin, I also add @skip decorator to the two test methods in
test/foundation/TestObjCMethods.py as it looks like the test suite is
deadlocking when running the tests.  More investigations are needed.

llvm-svn: 116552

and 'process handle'.  The test suite would like to control the asynch/sync
execution of the interpreter during the middle of the test method, so the
CommandInterpreter::SetSynchronous(bool value) is modified to allow the mode to
be changed more than once.

In practice, it would be advisable to control the process and to set the
async/sync mode from a single thread, too.

llvm-svn: 116467

function to get the line number to break on during setUp().

llvm-svn: 116275