[Checkers] Short JavaCOP evaluation

[Mahmood Ali]

Hi guys,

I have been playing with the distributed JavaCOP tool and the nonnull  
rules.  Here is my evaluation of the system as user (I haven't looked  
at the implementation nor the complexity of the rules).

Website:
- The JavaCOP page, unfortunately the second page in Google, provides  
simple information for downloading the tool and getting it to run.
- The documentation of the page is quite limited, without any  
reference.  The user is faced with four binary deliverables without  
much explanation.
- I could not find any promises nor explanation of what the examples  
actual do.
- There is a limited API to the trees, without any documentations.
- The paper serves as a guide to how one can write rules for JavaCOP.   
I have not read it.
- The page shows no progress whatsoever.  The only dating I can see is  
a message to the mailing list in the end of October.

Installation:
- Installation has been quite smooth more me.  I simply downloaded the  
executables as the tutorial indicated, and launched JavaCOP NonNull  
checker.
- I particularly liked the ability to not compile the source, and  
being able to use -cp.

JavaCOP system:
- The compiler is based on barebone java 1.7, without support for  
JSR308.  Thus one can only annotate JSR175.  In other words, one  
cannot annotate method parameters, receivers, nor generics.
- While there might be some way to specify annotations on such things  
(through comments or remote annotations), it is not documented.
- The system relies on the Javac internal API rather than the public  
API.  It made some issues easier and more efficient.
- The JavaCop rule file can link to Java code that actually do anything.

NonNull Checker:
- The NonNull checker supports three annotations: Nonnull, Raw,  
RawThis.  However, I honestly don't what Raw and RawThis semantics  
are.  They are not documented, nor are there any annotated examples.
- NonNull here is not a full type system, nor does it provide inter- 
procedural analysis.  It does no provide any method invocation tests,  
method overriding tests, etc.
- The nonnull inference is probably as as good as ours.  It inferred  
nullability properly in complex conditions relying on short-circuiting  
in Java (e.g. (v == null || v.toString() == null) or (v == null &&  
v.toString() == null).
- Generic support is limited if existent.  Not only cannot you  
annotate generics, it does not try to infer anything regarding it. For  
example, it does it do anything regarding: if (ls.get(0) != null)  
{ ls.get(0).toString() }
- It didn't check that the expression in enhanced for loop is nonnull  
(i.e. didn't check that 'list' is nonnull in 'for (Object o : ls)  
{ }').  When browsing the rule file for NonNull, I noticed a rule to  
address for-each statements, but I guess it's buggy.
- Doesn't warn against redundant checks (that could easily be done  
though I assume).
I will write some tests for nonnull checker regarding the for loop and  
the catch clause.

Needless to say that we are way ahead of JavaCOP in terms of  
capabilities.

Regards,
Mahmood