This provides a way to potentially provide conversational interactions with 'lldb' in the test suite.

llvm-svn: 127479

on the command line.  For example, use '-A x86_64^i386' to launch the inferior use both x86_64
and i386.

This is an example of building the debuggee using both clang and gcc compiers:

[17:30:46] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -C clang^gcc -v -f SourceManagerTestCase.test_modify_source_file_while_debugging

Session logs for test failures/errors will go into directory '2011-03-03-17_31_39'
Command invoked: python ./dotest.py -C clang^gcc -v -f SourceManagerTestCase.test_modify_source_file_while_debugging

Configuration:  compiler=clang
----------------------------------------------------------------------
Collected 1 test

1: test_modify_source_file_while_debugging (TestSourceManager.SourceManagerTestCase)
   Modify a source file while debugging the executable. ... Command 'run' failed!

original content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

new content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello lldb.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after writing new content: 1299202305.0
content restored to: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after restore: 1299202307.0
ok

----------------------------------------------------------------------
Ran 1 test in 8.259s

OK

Configuration:  compiler=gcc
----------------------------------------------------------------------
Collected 1 test

1: test_modify_source_file_while_debugging (TestSourceManager.SourceManagerTestCase)
   Modify a source file while debugging the executable. ... original content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

new content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello lldb.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after writing new content: 1299202307.0
content restored to: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after restore: 1299202309.0
ok

----------------------------------------------------------------------
Ran 1 test in 2.301s

OK
[17:31:49] johnny:/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 126979

of Stephen Wilson's idea (thanks for the input Stephen!). What I ended up
doing was:
- Got rid of ArchSpec::CPU (which was a generic CPU enumeration that mimics
  the contents of llvm::Triple::ArchType). We now rely upon the llvm::Triple 
  to give us the machine type from llvm::Triple::ArchType.
- There is a new ArchSpec::Core definition which further qualifies the CPU
  core we are dealing with into a single enumeration. If you need support for
  a new Core and want to debug it in LLDB, it must be added to this list. In
  the future we can allow for dynamic core registration, but for now it is
  hard coded.
- The ArchSpec can now be initialized with a llvm::Triple or with a C string
  that represents the triple (it can just be an arch still like "i386").
- The ArchSpec can still initialize itself with a architecture type -- mach-o
  with cpu type and subtype, or ELF with e_machine + e_flags -- and this will
  then get translated into the internal llvm::Triple::ArchSpec + ArchSpec::Core.
  The mach-o cpu type and subtype can be accessed using the getter functions:
  
  uint32_t
  ArchSpec::GetMachOCPUType () const;

  uint32_t
  ArchSpec::GetMachOCPUSubType () const;
  
  But these functions are just converting out internal llvm::Triple::ArchSpec 
  + ArchSpec::Core back into mach-o. Same goes for ELF.

All code has been updated to deal with the changes.

This should abstract us until later when the llvm::TargetSpec stuff gets
finalized and we can then adopt it.

llvm-svn: 126278

llvm-svn: 126260

llvm-svn: 126257

different build configurations.

llvm-svn: 125584

llvm-svn: 125522

llvm-svn: 124517

to be used for the test run.  Could be useful for debugging the setup of the
test environment.

llvm-svn: 124516

$ ./dotest.py -A i386 -C clang -v -w -t -p TestObjCMethods.py
$ ./dotest.py -A x86_64 -C gcc -v -w types

llvm-svn: 124303

since different options can affect the run time.

llvm-svn: 123843

module.

On my MBP running SnowLeopard:

$ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/Volumes/data/lldb/svn/trunk/test /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/cProfile.py -o my.profile ./dotest.py -v -w 2> ~/Developer/Log/lldbtest.log

After that, I used the pstats.py module to browse the statistics recorded in the my.profile file. 

llvm-svn: 123807

previously added ordinal number of the currently running test case.

llvm-svn: 122922

describing the ordinal number of the currently running test case.

llvm-svn: 122901

llvm-svn: 121501

Add an attribute __python_api_test__ (set to True) to the @python_api_test decorated
test method to distinguish them from the lldb command line tests.

llvm-svn: 121500

Example:

    @python_api_test
    def test_evaluate_expression_python(self):
        """Test SBFrame.EvaluateExpression() API for evaluating an expression."""
    ...

The opposite of Python APIs only test is an lldb command line test, which sends
commands to the lldb command interpreter.  Add a '-a' option to the test driver
to skip Python API only tests.

Modify TestExprs.py to mark a test as @python_api_test and remove an @expectedFailure
decorator as the bug has been fixed.

llvm-svn: 121442

as the args and the envs to the launched process.

o lldbtest.py:

  Forgot to check in some assertion messages changes for lldbtest.py.

o dotest.py:

  Also add "api" category to the default lldb log option list.

llvm-svn: 121220

As soon as an error or a failure is encountered, it stops the test suite.

llvm-svn: 120837

llvm-svn: 120751

log-categories, so it now becomes "event process expr state".

llvm-svn: 120745

o lldb => event process expr
o gdb-remote => packets process

llvm-svn: 120725

test classes or test cases to be excludued from the test suite.

Check in an example blacklist file: blacklist.py:

"""
'blacklist' is a Python dictionary, it stores the mapping of a string describing
either a testclass or a testcase, i.e, testclass.testmethod, to the reason (a
string) it is blacklisted.

Following is an example which states that test class IntegerTypesExprTestCase
should be skipped because 'This test class crashed' and the test case
FoundationTestCase.test_data_type_and_expr_with_dsym should be skipped because
it is 'Temporarily disabled'.

blacklist = {'IntegerTypesExprTestCase': 'This test class crashed',
             'FoundationTestCase.test_data_type_and_expr_with_dsym': 'Temporarily disabled'
             }
"""

blacklist = {}

An example of invoking the test driver and specifying a blacklist file:

./dotest.py -b blacklist.py -v types

This runs the tests under 'types' directory but excludes the tests specified in
balcklist.py.

llvm-svn: 120620

llvm-svn: 120295

ignore and not enforce the signleton pattern for the LLDBTestResult class.

llvm-svn: 120294

This is not to be used during normal test suite run, but to be used to stress test
specific test sequences repeatedly.

Example:

./dotest.py -# 3 -v breakpoint_conditions

will repeat the test suite 3 times for tests under the breakpoint_conditions directory.

llvm-svn: 119399

to invoke the test driver.

llvm-svn: 118847

llvm-svn: 118655

llvm-svn: 118650

llvm-svn: 118438

failed/errored tests at the start of the test run, in case the test run crashes
for any reason.  That way, it is easy to locate the session logs for accumulated
failures/errors.

llvm-svn: 118427

we want to run just the instance of testclass-testmethod combination and nothing else.

Specifying '-g' now will admit the whole .py test module if it does not contain a matching
testclass-testmethod combination at all.

This option arrangement adheres to the RISC principle of making the common cases fast. :-)

rdar://problem/8584914 Can the default for dotest.py's "-g" flag be switched?

llvm-svn: 118392

llvm-svn: 118279

idea to also dump the session information for expected failures in addition to
failures and errors (tests with unexpected exceptions).

Modify the LLDBTestResult class which derives from unittest2.TextTestResult to
intercept the addExpectedFailure() method in order to mark the test instance as
needing its session information dumped.

llvm-svn: 118185

llvm-svn: 117730

tests should be near the test summary message, instead of way up in the beginning.

llvm-svn: 117726

File name "2010-10-19-14:10:49.059609" is non-portable.

Do not use ':' in the pathname as it is not permitted on windows platforms.

llvm-svn: 117556

llvm-svn: 117129

llvm-svn: 117030

directory used to dump the session info for test failures/errors.

Example:

/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -s jason -v array_types

Session info for test errors or failures will go into directory jason
----------------------------------------------------------------------
Collected 4 tests

test_with_dsym_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dsym_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... ok
test_with_dwarf_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... FAIL

======================================================================
FAIL: test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 27, in test_with_dwarf_and_run_command
    self.array_types()
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 62, in array_types
    'stop reason = breakpoint'])
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 594, in expect
    self.runCmd(str, trace = (True if trace else False), check = not error)
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 564, in runCmd
    msg if msg else CMD_MSG(cmd, True))
AssertionError: False is not True : Command 'thread list' returns successfully

----------------------------------------------------------------------
Ran 4 tests in 3.086s

FAILED (failures=1)
/Volumes/data/lldb/svn/trunk/test $ ls jason
TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
/Volumes/data/lldb/svn/trunk/test $ head -10 jason/TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log 
Session info generated @ Thu Oct 21 09:54:15 2010

os command: [['/bin/sh', '-c', 'make clean; make MAKE_DSYM=NO']]
stdout: rm -rf "a.out" "a.out.dSYM"  main.o main.d
cc -arch x86_64 -gdwarf-2 -O0   -c -o main.o main.c
cc -arch x86_64 -gdwarf-2 -O0  main.o -o "a.out"

stderr: None
retcode: 0

/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 117028