[Checkers] Testing Approach Proposal

[Michael Ernst]

Mahmood-

Thanks for the proposal on testing.

Just to follow up, I think we agreed that the first proposal is worthwhile.

Also, the second proposal, or a version of the first proposal that put the
type in a comment to reduce the potential for error, might be nice to have,
but aren't really worth doing based on all the other things that are on our
critical path.

Furthermore, while I support testing of components that we have reason to
believe are buggy, or are complex and bug-prone, I do not think that we
should dedicate substantial resources to unit testing all portions of the
system, or even to end-to-end tests of corner cases.  It's more important
to perform case studies to find out what problems are of real importance to
users.  (If there are 5 bugs in the system but only 1 prevents typical
developers from using it, then fixing the others is likely to have negative
benefits if it delays us from discovering and fixing the others.)  Once we
do find a bug, then it's usually worth doing a bit of code inspection,
static analysis, or testing to make sure that that bug appears nowhere else
in the codebase.

                    -Mike


> From: Mahmood Ali <mahmood at MIT.EDU>
> To: checkers at lists.csail.mit.edu
> Subject: [Checkers] Testing Approach Proposal
> Date: Thu, 3 Apr 2008 10:11:22 -0400
> 
> Hi guys,
> 
> Currently we cannot do any unit testing due to the way the compiler  
> works. I have been thinking about this lately, and came up with two  
> approaches.  Both approaches require an input java file
> (consider one specified as 'class Test { void test()  
> { Lists.asList("m", "n"); } }').  We can do one of the following:
> 
> 1. Using textual expected type.
> The checker tester would specify in a different file the expected type  
> for some trees and their corresponding types.  To test the Interned  
> Checker, one would specify
> test():Lists.asList("m", "n")		List<@Interned String>		// only check  
> Lists.asList() within method test
> *:List.asList("m", "n")			List<@Interned String>		// check  
> List.asList() within all methods
> 
> This is a very limiting test, as it would only test  
> AnnotatedTypeFactory.getAnnotatedType().  However, implementing this  
> approach is somewhat straight-forward and takes care of a lot of the  
> cases.
> 
> 2. A specialized visitor in code
> We could have a specialized test suite.  To test the previous case one  
> would write something along the lines:
> class MyTest extends SpecializedTest {
>    AnnotatedTypeFactory createFactory(ProcessingEnvironment env,  
> CompilationUnitTree root) {
>      return new InternedAnnotatedTypeFactory(env, root);
>    }
> 
>    @TestForTree("*:List.asList("m", "n")")
>    void testTypeInference(Tree tree) {
>      AnnotatedTypeMirror type = factory.getAnnotatedType(tree);
>      assertEquals(type.toString(), "List<@Interned String>");
>    }
> }
> 
> SpecializedTest class would have a specialized tree visitor that  
> invoke the test method for tree found in TestForTree.  A method would  
> only be called on the tree specified in @TestForTree.
> 
> This allows for greater testing as one can "unit" test any they like,  
> without too much boilerplate code.
> 
> 
> What do you think?
> 
> - Mahmood
> 
> _______________________________________________
> checkers mailing list
> checkers at lists.csail.mit.edu
> https://lists.csail.mit.edu/mailman/listinfo/checkers