JSR 308 langtools: README for maintainers

This document explains processes for maintaners of the JSR 308 (type annotations) tools.

Contents:

Updating jsr308-langtools from OpenJDK changes

jsr308-langtools and basic-jsr308-langtools are the same, if one removes all edits from mernst, mali, mahmood, notnoop. That means that they should be updated in the same way. I presume one should update basic-jsr308-langtools from OpenJDK, then jsr308-langtools from basic-jsr308-langtools.

# First, make sure Sun has not broken the OpenJDK version
# (Sun's repository is at http://hg.openjdk.java.net/jdk7/jdk7/langtools/ ).
cd ~/research/types/openjdk-langtools
hg fetch
cd make; unset jsr308_imports; ant -e -Dboot.java.home=$HOME/java/jdk -Dtarget.java.home=$HOME/java/jdk -Djtreg.home=$HOME/research/types/jtreg clean build-all-tools jtreg

# Next, make sure that we have not broken our version:
cd ~/research/types/basic-jsr308-langtools
hg fetch
cd make; unset jsr308_imports; ant -e -Dboot.java.home=$HOME/java/jdk -Dtarget.java.home=$HOME/java/jdk -Djtreg.home=$HOME/research/types/jtreg clean build-all-tools jtreg

# Incorporate OpenJDK changes into basic.
# There are tags like "jdk7-b108" that I should take advantage of rather
# than just blind "hg fetch" to whatever is current.
cd ~/research/types/basic-jsr308-langtools
## Alternately:  hg fetch http://hg.openjdk.java.net/jdk7/tl/langtools/
## which gets patches faster (jdk7/jdk7 version is synced every 3 weeks),
## which can be good or bad.
hg fetch http://hg.openjdk.java.net/jdk7/jdk7/langtools/
cd make; unset jsr308_imports; ant -e -Dboot.java.home=$HOME/java/jdk -Dtarget.java.home=$HOME/java/jdk -Djtreg.home=$HOME/research/types/jtreg clean build-all-tools jtreg

# Incorporate basic changes into jsr308-langtools
cd ~/research/types/jsr308-langtools
hg fetch https://code.google.com/p/basic.jsr308-langtools/
cd make; unset jsr308_imports; ant -e -Dboot.java.home=$HOME/java/jdk -Dtarget.java.home=$HOME/java/jdk -Djtreg.home=$HOME/research/types/jtreg clean build-javac build-javap build-javadoc build-doclets javadoc-javac jtreg

# Do not commit any changes to jsr308-langtools yet!
(make -C $qn) && (make -C $pl/java check-nullness check-interning) && (make -C $inv check-nullness check-interning)

# If the above passes, commit changes to jsr308-langtools.

In addition, run

diff -u jsr308-langtools/dist/bin/javac checker-framework/checkers/binary/javac

and incorporate any recent Oracle javac changes into the Checker Framework's copy.

Releasing the JSR 308 langtools and the Checker Framework

This section explains how to make a public release of the JSR 308 langtools and checkers.

You should first update jsr308-langtools from recent OpenJDK changes.

The release process is mostly automated, but you should expect to spend 30 to 60 minutes for each release you make. You will actually make the release twice — first to a testing site, and then to the live site.

[I'm not sure whether this is a good or a bad idea.] You may wish to make the release as a fresh user that has no customizations.

Requirements

To make a release of the JSR 308 langtools and checkers, you need:

You should have all of the above if you use a Washington CSE machine (e.g., jicama or godwit).

Initial setup

  1. Set environment variables (in your .bashrc file).
      export anno=/scratch/$USER/jsr308-release
      export PATH=${PATH}:$anno/plume-lib/bin
      export PLUME_LIB=$anno/plume-lib
      export TEXINPUTS=.:/homes/gws/mernst/tex/sty:/homes/gws/mernst/tex:/uns/lib/hevea:..:
      export EDITOR=emacs
      export PERLLIB=${PERLLIB}:/homes/gws/mernst/bin/src/perl/lib/perl5/site_perl/5.10.0
      export PERL5LIB=${PERL5LIB}:/homes/gws/mernst/bin/src/perl/lib/perl5/site_perl/5.10.0
      # If dia and hevea are not on your path, add them, from either /uns or Michael Ernst's directory:
      export PATH=${PATH}:/homes/gws/mernst/bin/install/dia-0.97.1-install/bin:/homes/gws/mernst/bin/install/hevea-1.10/bin
    
  2. Make sure that JAVA_HOME is set to a JDK 6 and that JDK 6 is first on the PATH.
  3. Directory /uns/lib/hevea doesn't exist on Fedora 15 machines. Use /projects/uns/F11/lib/hevea/ instead.
  4. If you have never made a release before or no longer have a prepared release directory, set up a release directory as follows:
    1. Create a new directory. Place the directory in a non-NFS location such as a scratch partition, because a local disk is faster than NFS.

          mkdir -p $anno
          cd $anno
      
    2. Check out the jsr308-langtools and checker-framework repositories:

          hg clone https://code.google.com/p/jsr308-langtools/ jsr308-langtools
          hg clone https://code.google.com/p/checker-framework/ checker-framework
          hg clone https://code.google.com/p/plume-lib/ plume-lib
          hg clone https://code.google.com/p/annotation-tools/ annotation-tools
      
    3. Copy some files from the JSR 308 site as the basis for a testing site:

          cd $anno/checker-framework/release
          ant -f release.xml site-copy
      

      and answer the prompt accordingly. The site will be created in ~/www/jsr308test by default. [TODO: make it possible to be created elsewhere, for users with limited disk space. Or, just link from that directory to elsewhere.]

Making a release

  1. [This is actually done by the Python script, so it is not strictly necessary. Does the Python script also check "hg outgoing"? Does it need to?] Ensure that your checkouts are up to date and you have no uncommitted changes. Each of these commands should succeed, printing "no changes found":

      cd $anno/plume-lib && hg fetch && hg outgoing
      cd $anno/jsr308-langtools && hg fetch && hg outgoing
      cd $anno/annotation-tools && hg fetch && hg outgoing
      cd $anno/checker-framework && hg fetch && hg outgoing
    
    If any "fetch" fails, you can see the differences with:
      cd $anno/plume-lib && hg diff
      cd $anno/jsr308-langtools && hg diff
      cd $anno/annotation-tools && hg diff
      cd $anno/checker-framework && hg diff
    
    If you want to revert the have to start over and have changed, then do:
      cd $anno/jsr308-langtools && hg revert --all
      cd $anno/checker-framework && hg revert --all
    
    and then redo the above.
  2. Ensure that all regression tests pass.

  3. Run:

      # Only necessary if you did not run the regression tests in this directory.
      cd $anno/jsr308-langtools/make
      ant
    
      # Depends on the JSR 308 javac.
      cd $anno/annotation-tools/annotation-file-utilities
      ant zipfile
    

    TODO: merge this into the main script: in python or, better, in the Ant file.

    Make a release of the Annotation File Utilities before doing the JSR308/Checker Framework release. This should ensure that the file date for the AFU links is correctly updated. After the JSR308/Checker Framework release, ensure that the AFU release dates are correct.

  4. Edit the changelogs ($anno/jsr308-langtools/doc/changelog-jsr308.txt and $anno/checker-framework/checkers/changelog-checkers.txt) to indicate what has changed in this version. (Hopefully, this has already been done by each person who made a recent change.) The changelogs should focus on user-visible changes, or very significant implementation changes that may affect users.

    Every release, update file $anno/jsr308-langtools/doc/changelog-jsr308.txt, the changelog for the JSR 308 (Type Annotations) reference implementation; the release scripts run make -C $anno/jsr308-langtools/doc web which publishes the implementation.

    If there were changes in the specification, update its changelog in file jsr308-changes.html. To release the specification, run: make -C $anno/jsr308-langtools/doc web-spec

    Note: The release script below auto-populate the changelog files with changeset description since last release, and prompts to clean up the info.

    [TODO: Update the the instructions here to use tags instead]

    To double-check that the changelogs are adequate, run the following commands, and read the files in ~/tmp that they create:

    Determine the revision number for the previous release (TODO: we should tag releases instead) by examining the output of

      (cd $anno/checker-framework; hg log .hgtags)
      (cd $anno/jsr308-langtools; hg log .hgtags)
    

    and setting environment variables accordingly:

      export CF_PREV_REVISION=78cb6185e063
      export JL_PREV_REVISION=f448d3583dff
    
        mkdir -p ~/tmp
        # Diff the manuals against the previous versions:
        cd $anno/checker-framework/checkers/manual
        hg diff -w -r$CF_PREV_REVISION . > ~/tmp/checkers-manual-diff
        cd $anno/jsr308-langtools
        hg diff -w -r$JL_PREV_REVISION README-jsr308.html > ~/tmp/langtools-manual-diff
    
        # Use the version control logs since the last release:
        # In Emacs, it's useful to do:  (delete-matching-paragraphs "automated merge with " nil nil t)
        cd $anno/checker-framework
        hg log -v -r$CF_PREV_REVISION:tip > ~/tmp/checkers-log
        cd $anno/jsr308-langtools
        hg log -r$JL_PREV_REVISION:tip > ~/tmp/langtools-log
        hg log -r$JL_PREV_REVISION:tip --style changelog > ~/tmp/langtools-log-with-filenames
    
  5. Execute:

        cd $anno/checker-framework/release
        python -m release
    

    and follow the instructions it provides.

    If you are running on a system without access to the /cse/www2/types directory, then when the sanity check fails, you can do:

    (cd $HOME/www; scp -r jsr308test mernst@godwit.cs.washington.edu:./www/)
    

    (which puts the Checker Framework distribution at http://homes.cs.washington.edu/~mernst/jsr308test/checker-framework/current/checkers-manual.html), and then run a second time with

        python -m release -Dsanitycheck.dry.url=http://homes.cs.washington.edu/~mernst/jsr308test/checker-framework/releases/1.1.1
    

    (It would be possible to just run the sanitycheck target directly:

    ant -f release.xml -Dsanitycheck.dry.url=http://homes.cs.washington.edu/~mernst/jsr308test/checker-framework/releases/1.1.1 -Drelease.ver=1.1.1 sanitycheck
    

    but that fails to check for broken links.)

  6. Update the Checker Plugin for Eclipse.

    The basic instructions are contained in the README file. The project is hosted at eclipselabs.org.

Making a manual release

These instructions are deprecated in favor of the python release script. The material will be deleted in due time. But, for now there is information here that is not captured in the python release script.

  1. Set the anno and CFVERSION environment variables. CFVERSION is for the version being released. For example,
      export anno=$HOME/research/types
      export CFVERSION=1.0.3
    
  2. Set the anno and CFVERSION environment variables. CFVERSION is for the version being released. For example,
      export anno=$HOME/research/types
      export CFVERSION=1.0.3
    
  3. Ensure that all regression tests pass.

  4. Edit the changelogs ($anno/jsr308-langtools/doc/changelog-jsr308.txt and $anno/checker-framework/checkers/changelog-checkers.txt) to indicate what has changed in this version. (Hopefully, this has already been done by each person who made a recent change.) The changelogs should focus on user-visible changes, or very significant implementation changes that may affect users.

    To double-check that the changelogs are adequate, run the following commands, and read the files in ~/tmp that they create:

    Determine the revision number for the previous release (TODO: we should tag releases instead) by examining the output of

      (cd $anno/checker-framework; hg log checkers/changelog-checkers.txt)
      (cd $anno/jsr308-langtools; hg log /doc/changelog-jsr308.txt)
    

    and setting environment variables accordingly:

      export CF_PREV_REVISION=78cb6185e063
      export JL_PREV_REVISION=f448d3583dff
    
        mkdir -p ~/tmp
        # Diff the manuals against the previous versions:
        cd $anno/checker-framework/checkers/manual
        hg diff -w -r$CF_PREV_REVISION . > ~/tmp/checkers-manual-diff
        cd $anno/jsr308-langtools
        hg diff -w -r$JL_PREV_REVISION README-jsr308.html > ~/tmp/langtools-manual-diff
    
        # Use the version control logs since the last release:
        # In Emacs, it's useful to do:  (delete-matching-paragraphs "automated merge with " nil nil t)
        cd $anno/checker-framework
        hg log -v -r$CF_PREV_REVISION:tip > ~/tmp/checkers-log
        cd $anno/jsr308-langtools
        hg log -r$JL_PREV_REVISION:tip > ~/tmp/langtools-log
        hg log -r$JL_PREV_REVISION:tip --style changelog > ~/tmp/langtools-log-with-filenames
    
  5. Ensure that your checkouts are up to date and you have no uncommitted changes. Each one of these hg commands should print "no changes found":

      cd $anno/checker-framework
      hg fetch
      hg outgoing
      cd $anno/jsr308-langtools
      hg fetch
      hg outgoing
    
  6. Update your release checkout:

        cd $anno/checker-framework
        hg fetch
        cd ../jsr308-langtools
        hg fetch
    

    There shouldn't be any conflicts, since the release scripts shouldn't make any serious local modifications (aside from version number changes, which you'll check in later in the process).

  7. Ensure that the properties for the dry-run release is set appropriately in checker-framework/release/release.properties, set as follows:

    (You'll do a dry-run of the release to your site copy, and will only update the live site after you've sanity-checked the release.)

  8. Do a dry-run of the release:

        cd $anno/checker-framework/release
        ant -f release.xml -Drelease.ver=$CFVERSION clean web sanitycheck
    

    The web step will run ant clean build-javac build-javap for langtools, and ant clean dists all-tests for checkers, so if the build is broken or any checker tests fail, you won't be able to continue. (If this is the case, you should fix the issue, or contact someone who can via checker-framework-dev@googlegroups.com.)

    The sanitycheck step unzips the releases from the website, then rebuilds both projects and runs tests on the source and binary versions from the release.

    If all steps pass and you see BUILD SUCCESSFUL, then proceed. Otherwise, fix the problems and repeat the ant command.

  9. Sanity-check the release. If there are any problems, correct them in the repository and re-make the release.

  10. Make the final release:
        cd $anno/checker-framework/release
        ant -f release.xml -Drelease.ver=$CFVERSION -Drelease.is.real=true clean web checklinks sanitycheck
    

    The same process as in the dry-run occurs; chances are, if the dry-run passed successfully, this step should pass as well.

  11. Again test the simple installation instructions guideline:
    cd
    wget -nv -N http://types.cs.washington.edu/jsr308/Makefile-jsr308-install
    make -f Makefile-jsr308-install
    
  12. Check in any changes made to version numbers and dates as part of the release process. (Should this be done before the release, so that its data is consistent?)

        cd $anno/jsr308-langtools
        hg diff
        # As always, review the changes output by "hg diff" before committing.
        # Typical commit message:
        hg commit -m "Update version number to $CFVERSION"
        hg push
    
        cd $anno/checker-framework
        hg diff
        # As always, review the changes output by "hg diff" before committing.
        # Typical commit message:
        hg commit -m "Update version number to $CFVERSION"
        hg push
    
  13. Copy checkers-quals.jar to some libraries that use it:
      cp -pf .../checkers-quals.jar ${inv}/java/lib
      cvs commit ${inv}/java/lib/checkers-quals.jar -m "checkers-quals.jar version $CFVERSION"
      cp -pf .../checkers-quals.jar ${pl}/java/lib
      hg commit ${pl}/java/lib/checkers-quals.jar -m "checkers-quals.jar version $CFVERSION"
    
  14. Send a release announcement to jsr308-discuss@groups.google.com. The announcement should indicate what is new in the release, and should include the changelog entries from

      checker-framework/checkers/changelog-checkers.txt
      jsr308-langtools/doc/changelog-jsr308.txt