llvm-svn: 117116

directory used to dump the session info for test failures/errors.

Example:

/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -s jason -v array_types

Session info for test errors or failures will go into directory jason
----------------------------------------------------------------------
Collected 4 tests

test_with_dsym_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dsym_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... ok
test_with_dwarf_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... FAIL

======================================================================
FAIL: test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 27, in test_with_dwarf_and_run_command
    self.array_types()
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 62, in array_types
    'stop reason = breakpoint'])
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 594, in expect
    self.runCmd(str, trace = (True if trace else False), check = not error)
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 564, in runCmd
    msg if msg else CMD_MSG(cmd, True))
AssertionError: False is not True : Command 'thread list' returns successfully

----------------------------------------------------------------------
Ran 4 tests in 3.086s

FAILED (failures=1)
/Volumes/data/lldb/svn/trunk/test $ ls jason
TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
/Volumes/data/lldb/svn/trunk/test $ head -10 jason/TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log 
Session info generated @ Thu Oct 21 09:54:15 2010

os command: [['/bin/sh', '-c', 'make clean; make MAKE_DSYM=NO']]
stdout: rm -rf "a.out" "a.out.dSYM"  main.o main.d
cc -arch x86_64 -gdwarf-2 -O0   -c -o main.o main.c
cc -arch x86_64 -gdwarf-2 -O0  main.o -o "a.out"

stderr: None
retcode: 0

/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 117028

parent call frame information.  And comment out the check for stop reason for the
time being.  The stop reason disappeared from the "thread backtrace" output for
breakpoint stop with condition.

llvm-svn: 116939

invocations of them into session object.

Remove a debug statement.

llvm-svn: 116938

to be run during tearDown() to effect the restore action instead of executing it inline
during the test method, because the test may already fail and bailout before the inline
restore action.

Fix test_set_output_path() and pass_run_args_and_env_vars() to use this mechanism.

llvm-svn: 116881

llvm-svn: 116874

Add a little helper function SETTING_MSG(setting) to be used from TestSettings.test_set_output_path().

llvm-svn: 116847

Change the environment variable name to be more descriptive from LLDB_TIME_WAIT to LLDB_TIME_WAIT_NEXT_LAUNCH.

llvm-svn: 116809

now goes into a timestamp-specific directory instead of the previous .session-*
files.

[17:24:34] johnny:/Volumes/data/lldb/svn/trunk $ ls -l test/2010-10-18-16:56:12.935342
total 48
-rw-r--r--  1 johnny  admin  1695 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dsym_and_run_command.log
-rw-r--r--  1 johnny  admin  1652 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  2967 Oct 18 16:56 TestBreakpointCommand.BreakpointCommandTestCase.test_with_dsym.log
-rw-r--r--  1 johnny  admin  1648 Oct 18 16:56 TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  1665 Oct 18 16:56 TestClassTypesDisassembly.IterateFrameAndDisassembleTestCase.test_with_dsym_and_python_api.log
-rw-r--r--  1 johnny  admin  3873 Oct 18 16:58 TestFloatTypesExpr.FloatTypesTestCase.test_float_types_with_dsym.log
[17:24:37] johnny:/Volumes/data/lldb/svn/trunk $ 

Also, the dumping happens when a test errored in additioned to when it failed.

llvm-svn: 116778

facilitate session recording.  This happens inside an instance method where the
test instance is well known.

llvm-svn: 116649

pass in a 'sender' arg to the buildDefault(), buildDsym(), buildDwarf(), and
cleanup() functions.  The sender arg will be the test instance itself (i.e.,
an instance of TestBase).  This is so that the relevant command execution can be
recorded in the TestBase.session object if sender is available.

The lldbtest.system() command has been modified to pop the 'sender' arg out of
the keyword arguments dictionary and use it as the test instance to facilitate
seesion recordings.  An example is in test/types/AbstractBase.py:

    def generic_type_tester(self, atoms, quotedDisplay=False):
        """Test that variables with basic types are displayed correctly."""

        # First, capture the golden output emitted by the oracle, i.e., the
        # series of printf statements.
        go = system("./a.out", sender=self)

There are cases when sender is None.  This is the case when the @classmethod is
involved in the use of these APIs.  When this happens, there is no recording
into a session object, but printing on the sys.stderr is still honored if the
trace flag is ON.

An example is in test/settings/TestSettings.py:

    @classmethod
    def classCleanup(cls):
        system(["/bin/sh", "-c", "rm -f output.txt"])
        system(["/bin/sh", "-c", "rm -f stdout.txt"])

llvm-svn: 116648

    breakpoint modify -c 'val == 3' 1

after:

    breakpoint set -n c

which sets a breakpoint on function 'c'.  The breakpoint should only stop if
expression 'val == 3' evaluates to true.

llvm-svn: 116607

the session recordings.

llvm-svn: 116596

llvm-svn: 116583

is no need to check the return status of the command execution, and an error
status is not deemed a failure in the test.

llvm-svn: 116582

llvm-svn: 116559

session info after a test case failure, allowing more direct inspection of
debugger session which leads to the test failure.

For a simple usage scenario:

[18:06:26] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log 

...

[18:14:43] johnny:/Volumes/data/lldb/svn/trunk/test $ ls -l .session-*
-rw-r--r--  1 johnny  admin  1359 Oct 14 18:06 .session-TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command
-rw-r--r--  1 johnny  admin  2054 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dsym_and_expr_parser
-rw-r--r--  1 johnny  admin  2055 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_expr_parser
-rw-r--r--  1 johnny  admin  1351 Oct 14 17:57 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command
[18:14:51] johnny:/Volumes/data/lldb/svn/trunk/test $ 

The test case which failed will have its recorded session info dumped to a
.session-* file in the current working directory.  For test suite using
relocated directory, expect to find the .session-* files there.

In this checkin, I also add @skip decorator to the two test methods in
test/foundation/TestObjCMethods.py as it looks like the test suite is
deadlocking when running the tests.  More investigations are needed.

llvm-svn: 116552

and 'process handle'.  The test suite would like to control the asynch/sync
execution of the interpreter during the middle of the test method, so the
CommandInterpreter::SetSynchronous(bool value) is modified to allow the mode to
be changed more than once.

In practice, it would be advisable to control the process and to set the
async/sync mode from a single thread, too.

llvm-svn: 116467

function to get the line number to break on during setUp().

llvm-svn: 116275

within a file.  This is to be used within the test case to avoid hardcoded line number.

array_types/TestArrayTypes.py is modified first to use this pattern.  Other test modules
to follow.

rdar://problem/8537816
Testsuite: don't set breakpoints by exact file & line number

llvm-svn: 116270

files to a different top level directory than those specified on the command line.

When relocated, the test clanups normally performed afterwards after each test method
and after each test class will not be exercised at all.  This allows for an easier
postmortem analysis of test failures.

Example:

./dotest.py -v -t -r /tmp/lldbtest types

will create a /tmp/lldbtest directory which houses the types directory and its supported
files.

Files modified:

o dotest.py, lldbtest.py:

  Add logic to process '-r dir' option to support relocating the tests to a different
  top level directory instead of exected in place.

o darwin.py, test/types/Makefile:

  The 'make clean' should only clean the minimum .o and .d files.

llvm-svn: 116255

  Update the expected match string.

o lldbtest.py:

  Indicate when a command fails, even if there is nothing in the error stream.

o TestHelp.py:

  Add a regression test case for 'help image dump symtab'.

o CommandObjectHelp.cpp:

  Some of the logic branches with successful help command results were not tagged
  with a Success Status.  They are fixed now.  This is important for Python
  interaction.

llvm-svn: 116062

update the affected API clients.

llvm-svn: 115995

llvm-svn: 115922

  Temporarily commenting out the deprecated LaunchProcess() method.
  SWIG is not able to handle the overloaded functions.

o dotest.py/lldbtest.py:

  Add an '-w' option to insert some wait time between consecutive test cases.

o TestClassTypes.py:

  Make the breakpoint_creation_by_filespec_python() test method more robust and
  more descriptive by printing out a more insightful assert message.

o lldb.swig: Coaches swig to treat StateType as an int type, instead of a C++ class.

llvm-svn: 115899

Add a utility function to lldbtest.py to get the native pointer size of the host os.

llvm-svn: 115652

  To not skip long running tests, pass '-l' to the test driver (dotest.py).

  An example:

    @unittest2.skipIf(TestBase.skipLongRunningTest(), "Skip this long running test")
    def test_foundation_disasm(self):
        ...

o Added a long running disassemble test to the foundation directory, which iterates
  the code symbols from Foundation.framework and kicks off a disassemble command for
  for the named function symbol.  Found a crasher: rdar://problem/8504895.

o Plus added/updated some comments for the TestBase class.

llvm-svn: 115368

llvm-svn: 115166

testing various combinations of displaying variales of basic types.

The generic_type_tester() works by first capturing the golden output produced
by the printf stmts of ./a.out, creating a list of (var, value) pairs, and then
running the a.out to a stop point, and comparing the 'frame variable var' output
against the list of (var, value) pairs.

Modified int_type() and added long_type() to use generic_type_tester().

Also modified TestBase.expect() such that substring matching also return ok if
the substring starts at the 0-th position.

llvm-svn: 114708

actually test-and-compare anything yet.  The lldbtest.TestBase has an added
method setTearDownCleanup(dictionary=None) to facilitate running the cleanup
right after each data type test is run.  The test case can pass a dictionary
object when registering the test case cleanup.

There is currently only int_type test in the repository.

llvm-svn: 114600

llvm-svn: 114505

themselves.  Right now, it tests a breakpoint both before and after it has been
resolved.

Updated lldbtest.TestBase.expect() with an additional keyword argument 'exe' (
default to True), which if set to False, will treat the mandatory first argument
as just the string to be matched/or not-matched against the golden input.

llvm-svn: 114501

And removed the informational output from the conditional_break test case.

llvm-svn: 114493

of 'breakpoint command add/list/remove' commands to set breakpoint callbacks,
list them, and then remove one.

Modified the lldbtest.TestBase.expect() method to add two additional keyword
arguments:

o matching (default to True), which, if set to False, reverses the semantics of
  'expect' to 'expect not'

o patterns (default to None), which specifies a list of regexp patterns to match
  against the output from running the command

TestBreakpointCommand.py uses the matching=False and the patterns=[...] expect()
API.

llvm-svn: 114480

llvm-svn: 114232

to evaluate expressions.  Marked with @expectedFailure decorators for the time
being.

Enhanced the lldbtest.TestBase.expect() API to allow an additional keyword arg
named "error".  If the client passes "error=True", it signifies that an error
while running the command is expected.  The golden input is then compared
against the return object's error output.

llvm-svn: 114228

checks that the launched process writes its standard output there.

llvm-svn: 114102

This will remove the confusion experienced when previous test runs left some
files (both intermediate or by-product as a result of the test).

lldbtest.TestBase defines a classmethod tearDownClass(cls) which invokes the
platform-specific cleanup() function as defined by the plugin; after that, it
invokes a subclass-specific function classCleanup(cls) if defined; and, finally,
it restores the old working directory.

An example of classCleanup(cls) is in settings/TestSettings.py:

    @classmethod
    def classCleanup(cls):
        system(["/bin/sh", "-c", "rm output.txt"])

where it deletes the by-product "output.txt" as a result of running a.out.

llvm-svn: 114058

llvm-svn: 113989

has been fixed by r113947.  But just in case it still fails, emits a prominent
message always.

llvm-svn: 113986