[JSR308] Suggestion (and example) for annotations that take any annotation or for annotation inheritance

[Joe Darcy]

Greetings.

Further comments interspersed below.

Jonathan Aldrich wrote:
> Hello,
>
> I appreciate Joe sharing his perspective on this issue.  I work with 
> Ciera and Nels at CMU; they and others in my group have found that 
> annotation inheritance and/or annotations that take another annotation 
> are useful for a wide range of analyses, and so I wanted to respond to 
> some of the points Joe made and better explain why we think this 
> extension would be worth its weight.
>
> Regarding JSR 175, I definitely appreciate its focus on the needs of 
> "Enterprise Edition" Java.  On the other hand, JSR 308 is at least in 
> part aimed at adapting annotations, in minor ways, to make them more 
> suitable for program analysis.  Whether annotation-based program 
> analysis becomes mainstream is unknown--but if it does, as I and many 
> other believe it will, it will benefit all programmers, not just 
> Enterprise programmers.  Thus JSR 308 could have a wider impact than 
> JSR 175.  Furthermore, although I'm not an Enterprise edition expert, 
> there are good reasons for thinking that Enterprise users might 
> benefit from annotations that take another annotation as well (see 
> below).

The features of JSR 308 would definitely expand the scope of annotation 
usage somewhat, but the EE developers are not shy in putting in their 
own requests!  Related to 308, they have asked for repeated annotations 
on an element, a request that dates back to JSR 175; unrelated to 308, 
they have requested a way to systematically retrieve the names of method 
and constructor parameters.  Such names are needed when interfacing with 
languages having nominal rather than positional parameter passing 
conventions.  Of course, in language design as in other areas, it is 
important to determine the problem that should be solved rather than 
just evaluating the solution being offered.

> I believe allowing annotations to take arbitrary annotations as 
> arguments is a smaller change than allowing annotations wherever types 
> go (the primary focus of the proposal at present).  Furthermore, it 
> will eventually be needed by almost all of the program analyses/type 
> systems targeted by JSR 308.  Finally, this one change, by itself, can 
> express much of the the JSR 308 existing proposal (I'm not arguing we 
> should throw the rest out, only that this piece carries its weight 
> well).  Let me explain these one by one.

Alex has written up some thoughts on "Complexity in language design":
http://blogs.sun.com/abuckley/entry/complexity_in_language_design

>
>
> FIRST: annotations taking other annotations as parameters is a 
> (comparatively) small change.  The existing JSR 308 proposal changes 
> the Java language syntax.  Merely permitting annotations to extend 
> other annotations would not change the language syntax.

Allowing annotations in more syntactic places isn't that big of a 
language change.  Syntax changes are big when positive compatibility is 
affected; that is, when existing valid program are no longer valid or 
when existing valid programs have their semantics changed.  When only 
negative compatibility is involved, meaning the string of some 
previously invalid program is now valid, that is in and of itself fairly 
painless.  JSR 308 involves more than just syntax changes of course, but 
the new classfile attributes, etc. don't fundamentally change many other 
parts of the system.

See also "Kinds of Compatibility: Source, Binary, and Behavioral:"
http://blogs.sun.com/darcy/entry/kinds_of_compatibility


>   The existing JSR 308 proposal changes the class file format.  Joe 
> raised a great question about whether annotations taking other 
> annotations as parameters would require changing the class file format 
> as well.  As far as I can tell, from the specification of the Java 5 
> classfile format, it would not:
>
> http://docs.oracle.com/javase/specs/jvms/se5.0/ClassFileFormat-Java5.pdf 
>
>
> Specifically, annotations are stored in an "annotation" structure in 
> the classfile.  This structure contains a type index (the type of the 
> annotation) and a list of element_value_pairs.  The value in the 
> element_value_pair is an element_value, which is a tagged union, and 
> if the tag is "@" for annotation, then the union stores an 
> "annotation" structure.  That's the annotation structure above--which 
> in fact contains a type index, that in the case of "annotations taking 
> arbitrary annotations" tells us what type the nested annotation is.  
> Ironically, in the current classfile format, this is wasted space, as 
> it could be inferred from the surrounding annotation type, but it 
> nicely supports this extension.  Thus, "annotations containing 
> arbitrary annotations" need change neither the syntax of the language 
> nor the classfile format.  It could be that there are other issues 
> lurking, and Joe is right that we need to investigate this in detail, 
> but to me it initially looks like a simpler change than the other 
> elements of JSR 308.

Whether or not the format has to change in a deep way also depends on 
the code which reads the format.  In Sun's JDK, this is primarily done 
in the sun.reflect.annotation package.

>
> SECOND: such an extension would be of very wide utility.  In the last 
> week, four people have suggested this.  Two of them are my students, 
> who are developing entirely different type systems, with independent 
> and somewhat different needs for annotations within annotations.  I 
> have at least one more example that I'll give

Two students at the same university working with the same professor on 
the same general topic advocating the same language doesn't sound like 
two fully independent events to me ;-)

In terms of judging utility of a language change, certainly the direct 
utility to programmers should count; a change like the for-each loop 
from Java SE 5 is more useful to more people than, say, hexadecimal 
floating-point literals.  Indirect utility should count too; although 
not as much.  That is, if adding feature XYZ allows a small group of N 
people to build a framework that is usable to 10*N times people, the 
utility of the feature to developers should be scored somewhere between 
N and 10*N.  However, there are millions and millions of Java developers 
and as Spock says, "The needs of the many outweigh the needs of the few, 
or the one."  For example, we've long had a vocal contingent of  
developers desiring design by contract support over and above the assert 
facility, but most people and satisfied with assert and we have no plans 
to add design by contract.

> below.  This, and that two others unrelated to me have proposed 
> variants of the same idea, suggests we're on to something general.
>
> In fact, the most important use case for "annotations taking arbitrary 
> annotations" is annotation parameters that don't correspond to a type 
> parameter.  Imagine that you have developed some kind of container 
> class that is designed to hold things of a specific type, not of an 
> arbitrary type.  This may be realistic in cases where the container 
> needs to interact with the things it contains in domain-specific 
> ways.  In this case, there is no reason to have a type parameter on 
> the container. However, it may still make lots of sense to have 
> annotations on the contained thing.
>
> As a concrete example, another one of my students is using annotations 
> to implement an ownership type system.  Ownership types can be used 
> for lots of things, including finding design errors, eliminating race 
> conditions, etc.  Imagine a MySubject object that has a list of 
> MyObserver objects.  The MySubject knows the MyObserver type, and 
> doesn't need a type parameter for the particular kind of observer, but 
> it does need to know who owns its observers.  Right now, we have to 
> express this in an ugly string-based syntax:
>
> @OwnedBy("owned<shared>") MySubject subject
>
> Here "owned" is the owner of MySubject, while "shared" is the owner of 
> the observers stored in MySubject.  We could add a dummy parameter to 
> MySubject and use JSR 308 as follows:
>
> @Owned
> MySubject< @Shared MyObserver> subject
>
> First, note the benefits of JSR308: I was able to avoid parsing a 
> string, and I can use the more structured @Owned and @Shared instead 
> of "owned" and "shared."  But it is very unpleasant, to the point of 
> being unmaintainable in large systems where there might be multiple 
> such parameters, to add unneeded type parameters just to add an 
> annotation parameter.  A much better solution is to use annotations:
>
> @Owned
> @OwnershipArgs({@Shared})
> MySubject subject
>
> Almost any analysis is likely to come up against this issue.  To pick 
> a few from the JSR 308 spec, containers without type parameters could 
> benefit in:
>  * Titanium, where contained elements are @local
>  * dependent types, where the contained elements are arrays of a @Size
>  * javari/immutability, where the contained elements are immutable
>  * non-nullness, where the contained elements are non-null
>  * taintedness, where the contained elements are tainted
>  * etc. for just about any type qualifier
>
> Another great example in the JSR 308 spec, which is independent of the 
> "annotation parameterization without type parameterization" is units 
> of measurement.  Instead of @Units("m/s") it might be nice to expose 
> the structure of the units with annotations: @Units(value={@Meters}, 
> dividedBy={@Seconds})  This won't work without "annotations taking 
> arbitrary annotations."
>
> As a further (conceptual) motivation, JSR 175 was originally developed 
> for "Enterprise Edition" Java--but when Enterprise users aren't using 
> annotations, they're using XML configuration files.  Interestingly, 
> the most important semantic difference between JSR 175 annotations and 
> XML files is that XML files allow recursive structure--XML elements 
> containing arbitrary XML elements--whereas JSR 175 does not allow 
> annotations to contain arbitrary annotations.  I am not an Enterprise 
> Java user, but I suspect that there would be a lot of use cases in 
> Enterprise Java for recursively structured annotations--in particular 
> I imagine complex issues like database persistence and security could 
> probably benefit.
>
> So if JSR 308 is designed to facilitate (at least in part) static 
> checkers, nearly all of them would benefit from "annotations taking 
> arbitrary annotations," and although I don't have examples, it's 
> reasonable to think that other uses of annotations would benefit as well.
>
>
>
> THIRD, "annotations taking arbitrary annotations" can express much of 
> what's in JSR 308--not as cleanly, to be sure, but still allowing 
> users to express annotations on most uses of a type.  Instead of the 
> current proposal:
>
> Map<@NonNull String, @NonEmpty List<@Readonly Document>> files;
>
> we could write:
>
> @Params({@NonNull, @AnnotAndParams(@NonEmpty, {@Readonly})})
> Map<String, List<Document>> files;
>
>
> Similar ideas would work for arrays, type bounds, wildcards, class 
> inheritance, throws clauses, and method receivers.  It would not work 
> for new statements, casts, or generic type arguments; syntax 
> extensions a la JSR 308 are needed there.
>
> Now, the JSR 308 syntax is *definitely* more readable, and I am not 
> for a moment suggesting that we give it up.  But the fact that 
> "annotations taking arbitrary annotations" is by itself, likely 
> simpler than the rest of JSR 308 to implement, is broadly applicable, 
> and can express much of what the rest of JSR 308 does, seems to me 
> like a good argument that it pulls its own weight.  It arguably may 
> fall higher than the rest of JSR 308 on the "worse is better" scale.
>
>
>
> Joe also said that work needs to be done in ensuring this can be 
> feasibly implemented, beyond the short analysis above; I agree, and I 
> hope we can contribute to that.  But first I'd like to get a broader 
> sense for if the JSR 308 community sees this as a valuable proposal.

Yes, in general I'm more likely to be persuaded by the style of 
investigation written up here:
http://www.cs.rice.edu/~mgricken/research/xajavac/index.shtml

>
> Thanks,
>
> Jonathan
>
>
>
> P.S. Regarding "anything that requires instanceof is suspicious" - 
> functional programmers who use pattern matching would disagree, and I 
> suspect instanceof is harmless in many of the use cases for this 
> extension.  But

Yes, but since Java doesn't have pattern matching, using instanceof as a 
substitute is not very idiomatic Java.

Cheers,

-Joe


> from an OO extensibility perspective, the "right change" is actually 
> not "annotations that take arbitrary annotations" but rather 
> "annotations that inherit from other annotations."  In the latter 
> case, no instanceof is necessary, as ordinary dispatch or the visitor 
> pattern can be used.  But "annotations that inherit from other 
> annotations" is more complex than "annotations that take arbitrary 
> annotations", so a "worse is better" perspective suggests that we 
> might prefer "annotations that take arbitrary annotations" despite the 
> instanceof issue.  If there's a groundswell of support for annotations 
> that inherit from other annotations, and the technical issues are 
> solvable, I'd be delighted.