diff --git a/FORMAT.md b/FORMAT.md
new file mode 100644
index 0000000000000000000000000000000000000000..efe73b95cfde90c21e568fbe94b396bc5fc54ec7
--- /dev/null
+++ b/FORMAT.md
@@ -0,0 +1,503 @@
+# Signature Formats
+
+This document describes the signature file format created and used by metalava,
+doclava, apicheck, etc.
+
+There are currently 3 versions of this format:
+
+1. The format emitted by doclava, and used for Android's signature files up
+ through Android P. Note that this isn't actually a single format; it evolved
+ over time, so older signature files vary a bit (many of these changes were
+ due to bugs getting fixed, such as type parameters missing from classes
+ and methods until they start appearing), and some were deliberate changes,
+ such as dropping the "final" modifier in front of every member if the
+ containing class is final.
+
+2. The "new" format, which is described below, and is used in Android Q. This
+ format adds new information, such as annotations, parameter names and default
+ values, as well as cleans up a number of things (such as dropping
+ java.lang. prefixes on types, etc)
+
+3. This is format v2, but will all nullness annotations replaced by a
+ Kotlin-syntax, e.g. "?" for nullable types, "!" for unknown/platform types,
+ and no suffix for non-nullable types. The initial plan was to include this
+ in format v2, but it was deferred since type-use annotations introduces
+ somple complexities in the implementation.
+
+
+## Motivation
+
+Why did we change from the historical doclava signature format (v1)
+to a new format?
+
+In order to support Kotlin better (though this will also benefit Java
+developers), we'd like to have nullness annotations (as well as some other
+annotations) be a formal part of the SDK.
+
+That means the annotations should be part of the signature files too -- such
+that we can not just record explicitly what the API contract is, but also
+enforce that changes are not only deliberate changes but also compatible
+changes. (For example, you can change the return value of a final method from
+nullable to non null, but not the other way around.)
+
+And if we were going to change the signature format, we might as well make some
+other changes too.
+
+
+### Comments
+
+In v2, line comments (starting with //) are allowed. This allows us to leave
+reminders and other issues with the signature source (though the update-api task
+will generally blow these away, so use sparingly.)
+
+### Header
+
+New signature files (v2+) generally include a file header comment which states
+the version number. This makes it possible for tools to more safely interpret
+signature files. For example, in v3 the type "String" means "@NonNull String",
+but in v2 "String" means "String with unknown nullness".
+
+The header looks like this:
+
+```
+// Signature format: 2.0
+```
+
+Here "2" is the major format version; the .0 allows for compatible minor
+variations of the format.
+
+### Include Annotations
+
+The new signature format now includes annotations; not all annotations (such as
+@Override etc); only those which are significant for the API, such as nullness
+annotations, etc.
+
+Annotations are included on the same line as the class/field/method, right
+before the modifiers.
+
+Here's how this looks:
+
+
+```
+ method @Nullable public static Integer compute1(@Nullable java.util.List<java.lang.String>);
+```
+
+
+(Notice how the annotations are not using fully qualified name; that's discussed
+below.)
+
+The annotations to be included are annotations for annotation types that are not
+hidden, and have class file or runtime retention.
+
+The annotations should be sorted alphabetically by fully qualified name.
+
+
+### Use Special Syntax or Nullness Annotations
+
+(Note: Only in version format 3+)
+
+As a special optimization, since we eventually want **all** APIs to have
+explicit nullness, use Kotlin's syntax for nullness. That means that for
+nullable elements, we add "?" after the type, for unknown nullness we add "!",
+and otherwise there's no suffix. In other words:
+
+
+<table>
+ <tr>
+ <td>
+ </td>
+ <td>Java Type
+ </td>
+ <td>Signature File Type
+ </td>
+ </tr>
+ <tr>
+ <td>Nullable
+ </td>
+ <td>@Nullable String
+ </td>
+ <td>String?
+ </td>
+ </tr>
+ <tr>
+ <td>Not nullable
+ </td>
+ <td>@NonNull String
+ </td>
+ <td>String
+ </td>
+ </tr>
+ <tr>
+ <td>Unknown nullability
+ </td>
+ <td>String
+ </td>
+ <td>String!
+ </td>
+ </tr>
+</table>
+
+
+The above signature line is turned into
+
+
+```
+ method public Integer? compute1(java.util.List<java.lang.String!>?);
+```
+
+
+### Clean Up Terminology
+
+Format v2 also cleans up some of the terminology used to describe the class
+structure in the signature file. For example, in v1, an interface is called an
+"abstract interface"; an interface extending another interface is said to
+"implement" it instead of "extend"-ing it, etc; enums and annotations are just
+referred to as classes that extend java.lang.Enum, or java.lang.Annotation etc.
+
+With these changes, these lines from v1 signature files:
+
+
+```
+ public abstract interface List<E> implements java.util.Collection { ... }
+ public class TimeUnit extends java.lang.Enum { ... }
+ public abstract class SuppressLint implements java.lang.annotation.Annotation { ... }
+```
+
+
+are replaced by
+
+
+```
+ public interface List<E> extends java.util.Collection<E> { ... }
+ public enum TimeUnit { ... }
+ public @interface SuppressLint { ... }
+```
+
+
+
+### Use Generics Everywhere
+
+The v1 signature files uses raw types in some places but not others. Note that
+in the above it was missing from super interface Collection:
+
+
+```
+ public abstract interface List<E> implements java.util.Collection { ... }
+```
+
+
+ whereas in the v2 format it's included:
+
+
+```
+ public interface List<E> extends java.util.Collection<E> { ... }
+```
+
+
+Similarly, v1 used erasure in throws clauses. For example, for this method:
+
+
+```
+ public <X extends Throwable> T orElseThrow(Supplier<? extends X> exceptionSupplier) throws X
+```
+
+v1 used this signature:
+
+
+```
+ method public <X extends java.lang.Throwable> T orElseThrow(java.util.function.Supplier<? extends X>) throws java.lang.Throwable;
+```
+
+Note how that's "throws Throwable" instead of "throws X". This results in b/110302703.
+
+In the v2 format we instead use the correct throws type:
+
+```
+ method public <X extends java.lang.Throwable> T orElseThrow(java.util.function.Supplier<? extends X>) throws X;
+```
+
+
+### Support Annotations
+
+The old format was completely missing annotation type methods:
+
+```
+ public static abstract class ViewDebug.ExportedProperty implements java.lang.annotation.Annotation {
+ }
+```
+
+We need to include annotation member methods, as well as their default values
+since those are API-significant. Here's how this looks in the v2 file format
+(also applying the @interface terminology change described above) :
+
+
+```
+ public static @interface ViewDebug.ExportedProperty {
+ method public abstract String category() default "";
+ method public abstract boolean deepExport() default false;
+ method public abstract android.view.ViewDebug.FlagToString[] flagMapping() default {};
+ method public abstract boolean formatToHexString() default false;
+ method public abstract boolean hasAdjacentMapping() default false;
+ method public abstract android.view.ViewDebug.IntToString[] indexMapping() default {};
+ method public abstract android.view.ViewDebug.IntToString[] mapping() default {};
+ method public abstract String prefix() default "";
+ method public abstract boolean resolveId() default false;
+ }
+```
+
+
+### Support Kotlin Modifiers
+
+This doesn't currently apply to the SDK, but the signature files are also used
+in the support library, and some of these are written in Kotlin and exposes
+Kotlin-specific APIs.
+
+That means the v2 format can express API-significant aspects of Kotlin. This
+includes special modifiers, such as sealed, inline, operator, infix, etc:
+
+```
+ method public static operator int get(android.graphics.Bitmap, int x, int y);
+ method public static infix android.graphics.Rect and(android.graphics.Rect, android.graphics.Rect r);
+```
+
+### Support Kotlin Properties
+
+Kotlin's Java support means that it wil take a Kotlin property and compile it
+into getters and setters which you can call from Java. But you cannot calls
+these getters and setters from Kotlin; you **must** use the property
+syntax. Therefore, we need to also capture properties in the signature files. If
+you have this Kotlin code:
+
+
+```
+ var property2: String? = "initial"
+```
+
+it will get recorded in the signature files like this:
+
+```
+ property public java.lang.String? property2 = "initial";
+ method public java.lang.String? getProperty2();
+ method public void setProperty2(java.lang.String? p);
+```
+
+The last two elements are "redundant"; they could be computed from the property
+name (and included if the property declaration uses special annotations to name
+the getters and setters away from the defaults), but it's helpful to be explicit
+(and this allows us to specify the default value).
+
+### Support Named Parameters
+
+Kotlin supports default values for parameters, and these are a part of the API
+contract, so we need to include them in the signature format.
+
+Here's an example:
+
+```
+ method public static void edit(android.content.SharedPreferences, boolean commit);
+```
+
+In v1 files we only list type names, but in v2 we allow an optional parameter
+name to be specified; "commit" in the above.
+
+Note that this isn't just for Kotlin. Just like there are special nullness
+annotations to mark up the null contract for an element, we will also have a
+special annotation to explicitly name a Java parameter:
+@android.annotation.ParameterName (which is hidden). This obviously isn't usable
+from Java, but Kotlin client code can now reference the parameter.
+
+Therefore, the following Java code (not signature code) will also produce
+exactly the same signature as the above:
+
+```
+ public static void edit(SharedPreferences prefs, @ParameterName("commit") boolean ct) {…}
+```
+
+(Note how the implementation parameter doesn't have to match the public, API
+name of the parameter.)
+
+### Support Default Values
+
+In addition to named parameters, Kotlin also supports default values. These are
+also be part of the v2 signature since (as an example) removing a default value
+is a compile-incompatible change.
+
+Therefore, the v2 format allows default values to be specified after the type
+and/or parameter name:
+
+```
+ method public static void edit(SharedPreferences, boolean commit = false);
+```
+
+For Kotlin code, the default parameter values are extracted automatically, and
+for Java, just as with parameter names, you can specify a special annotation to
+record the default value for usage from languages that support default parameter
+values:
+
+```
+ public static void edit(SharedPreferences prefs, @DefaultValue("false") boolean ct) {…}
+```
+
+
+### Include Inherited Methods
+
+Consider a scenario where a public class extends a hidden class, and that hidden
+class defines a public method.
+
+Doclava did not include these methods in the signature files, but they **were**
+present in the stub files (and therefore part of the API). In the v2 signature
+file format, we include these.
+
+An example of this is StringBuilder#setLength. According to the old signature
+files, that method does not exist, but clearly it's there in the SDK. The reason
+this happens is that StringBuilder is a public class which extends hidden class
+AbstractStringBuilder, which defines the public method setLength.
+
+
+### No Hardcoded Enum Methods
+
+Doclava always inserted two special methods in the signature files for every
+enum: values() and valueOf():
+
+```
+ public static final class CursorJoiner.Result extends java.lang.Enum {
+ method public static android.database.CursorJoiner.Result valueOf(java.lang.String);
+ method public static final android.database.CursorJoiner.Result[] values();
+ enum_constant public static final android.database.CursorJoiner.Result BOTH;
+ enum_constant public static final android.database.CursorJoiner.Result LEFT;
+ enum_constant public static final android.database.CursorJoiner.Result RIGHT;
+ }
+```
+
+It didn't do that in stubs, because you can't: those are special methods
+generated by the compiler. There's no reason to list these in the signature
+files since they're entirely implied by the enum, you can't change them, and
+it's just extra noise.
+
+In the new v2 format these are no longer present:
+
+```
+ public static enum CursorJoiner.Result {
+ enum_constant public static final android.database.CursorJoiner.Result BOTH;
+ enum_constant public static final android.database.CursorJoiner.Result LEFT;
+ enum_constant public static final android.database.CursorJoiner.Result RIGHT;
+ }
+```
+
+### Remove "deprecated" Modifier
+
+The old signature file format used "deprecated" as if it was a modifier. In the
+new format, we instead list these using annotations, @Deprecated.
+
+### Standard Modifier Order
+
+Doclava had a "random" (but stable) order of modifiers.
+
+In the new signature format, we're using the standard modifier order for Java
+and Kotlin, wihch more closely mirrors what is done in the source code.
+
+Version format 1 order:
+
+```
+public/protected/private default static final abstract synchronized transient volatile
+```
+
+Version format 2 order:
+
+```
+public/protected/internal/private abstract default static final transient volatile synchronized
+```
+
+The above list doesn't include the Kotlin modifiers, which are inserted
+according to the Kotlin language style guide:
+https://kotlinlang.org/docs/reference/coding-conventions.html#modifiers
+
+### Sort Classes By Fully Qualified Names
+
+In "extends" lists, the signature file can list a comma separated list of
+classes. The classes are listed by fully qualified name, but in v1 it was sorted
+by simple name. In the v2 format, we sort by fully qualified name instead.
+
+### Use Wildcards Consistently
+
+Doclava (v1) would sometimes use the type bound <?> and other times use <?
+extends Object>. These are equivalent. In the v2 format, <? extends Object> is
+always written as <?>.
+
+### Annotation Simple Names
+
+We have a number of annotations which are significant for the API -- not just
+the nullness as deprecation ones (which are specially supported in v3 via the
+?/! Kotlin syntax and the deprecated "modifier"), but annotations for permission
+requirements, range constraints, valid constant values for an integer, and so
+on.
+
+In the codebase, these are typically in the android.annotation. package,
+referencing annotation classes that are generally **not** part of the API. When
+we generate the SDK, we translate these into publicly known annotations,
+androidx.annotation, such that Studio, lint, the Kotlin compiler and others can
+recognize the metadata.
+
+That begs the question: which fully qualified name should we put in the
+signature file? The one that appeared in the source (which is hidden, or in the
+case of Kotlin code, a special JetBrains nullness annotation), or the one that
+it gets translated into?
+
+In v2 we do neither: We use only the simple name of the annotations in the
+signature file, for annotations that are in the well known packages. In other
+words, instead of any of these alternative declarations:
+
+```
+ method public void setTitleTextColor(@android.annotation.ColorInt int);
+ method public void setTitleTextColor(@android.support.annotation.ColorInt int);
+ method public void setTitleTextColor(@androidx.annotation.ColorInt int);
+```
+
+in v2 we have simply
+
+```
+ method public void setTitleTextColor(@ColorInt int);
+```
+
+### Simple Names in Java.lang
+
+In Java files, you can implicitly reference classes in java.lang without
+importing them. In v2 offer the same thing in signature files. There are several
+classes from java.lang that are used in lots of places in the signature file
+(java.lang.String alone is present in over 11,000 lines of the API file), and
+other common occurrences are java.lang.Class, java.lang.Integer,
+java.lang.Runtime, etc.
+
+This basically builds on the same idea from having an implicit package for
+annotations, and doing the same thing for java.lang: Omitting it when writing
+signature files, and implicitly adding it back when reading in signature files.
+
+This only applies to the java.lang package, not any subpackages, so for example
+java.lang.reflect.Method will **not** be shortened to reflect.Method.
+
+### Type Use Annotations
+
+In v3, "type use annotations" are supported which means annotations can appear
+within types.
+
+### Miscellaneous
+
+Some other minor tweaks in v2:
+
+* Fix formatting for package private elements. These had two spaces of
+ indentation; this is probably just a bug. The new format aligns their
+ indentation with all other elements.
+* Don't add spaces in type bounds lists (e.g. Map<X,Y>, not Map<X, Y>.)
+
+## Historical API Files
+
+Metalava can read and write these formats. To switch output formats, invoke it
+with for example --format=v2.
+
+The Android source tree also has checked in versions of the signatures for all
+the previous API levels. Metalava can regenerate these for a new format.
+For example, to update all the signature files to v3, run this command:
+
+```
+$ metalava --write-android-jar-signatures *<android source dir>* --format=v3
+```
diff --git a/README.md b/README.md
index ff1253b75bf077b363e6b325221c07c622d2ffed..608e7a9e5efaf035b29be46cc56479e7117ed104 100644
--- a/README.md
+++ b/README.md
@@ -1,23 +1,24 @@
# Metalava
-(Also known as "doclava2", but deliberately not named doclava2 since crucially it
-does not generate docs; it's intended only for **meta**data extraction and generation.)
+(Also known as "doclava2", but deliberately not named doclava2 since crucially
+it does not generate docs; it's intended only for **meta**data extraction and
+generation.)
Metalava is a metadata generator intended for the Android source tree, used for
a number of purposes:
-* Allow extracting the API (into signature text files, into stub API files (which
- in turn get compiled into android.jar, the Android SDK library)
- and more importantly to hide code intended to be implementation only, driven
- by javadoc comments like @hide, @$doconly, @removed, etc, as well as various
+* Allow extracting the API (into signature text files, into stub API files
+ (which in turn get compiled into android.jar, the Android SDK library) and
+ more importantly to hide code intended to be implementation only, driven by
+ javadoc comments like @hide, @$doconly, @removed, etc, as well as various
annotations.
* Extracting source level annotations into external annotations file (such as
the typedef annotations, which cannot be stored in the SDK as .class level
annotations).
-* Diffing versions of the API and determining whether a newer version is compatible
- with the older version.
+* Diffing versions of the API and determining whether a newer version is
+ compatible with the older version.
## Building and running
@@ -48,11 +49,12 @@ To run metalava:
--verbose Include extra diagnostic output
...
+
(*output truncated*)
Metalava has a new command line syntax, but it also understands the doclava1
-flags and translates them on the fly. Flags that are ignored are listed on
-the command line. If metalava is dropped into an Android framework build for
+flags and translates them on the fly. Flags that are ignored are listed on the
+command line. If metalava is dropped into an Android framework build for
example, you'll see something like this (unless running with --quiet) :
metalava: Ignoring javadoc-related doclava1 flag -J-Xmx1600m
@@ -69,40 +71,40 @@ example, you'll see something like this (unless running with --quiet) :
* Compatibility with doclava1: in compat mode, metalava spits out the same
signature files for the framework as doclava1.
-* Ability to read in an existing android.jar file instead of from source, which means
- we can regenerate signature files etc for older versions according to new formats
- (e.g. to fix past errors in doclava, such as annotation instance methods which were
- accidentally not included.)
+* Ability to read in an existing android.jar file instead of from source, which
+ means we can regenerate signature files etc for older versions according to
+ new formats (e.g. to fix past errors in doclava, such as annotation instance
+ methods which were accidentally not included.)
* Ability to merge in data (annotations etc) from external sources, such as
IntelliJ external annotations data as well as signature files containing
annotations. This isn't just merged at export time, it's merged at codebase
load time such that it can be part of the API analysis.
-* Support for an updated signature file format:
+* Support for an updated signature file format (which is described in FORMAT.md)
- * Address errors in the doclava1 format which for example was missing annotation
- class instance methods
+ * Address errors in the doclava1 format which for example was missing
+ annotation class instance methods
* Improve the signature format such that it for example labels enums "enum"
- instead of "abstract class extends java.lang.Enum", annotations as "@interface"
- instead of "abstract class extends java.lang.Annotation", sorts modifiers in
- the canonical modifier order, using "extends" instead of "implements" for
- the superclass of an interface, and many other similar tweaks outlined
- in the `Compatibility` class. (Metalava also allows (and ignores) block
- comments in the signature files.)
+ instead of "abstract class extends java.lang.Enum", annotations as
+ "@interface" instead of "abstract class extends java.lang.Annotation", sorts
+ modifiers in the canonical modifier order, using "extends" instead of
+ "implements" for the superclass of an interface, and many other similar
+ tweaks outlined in the `Compatibility` class. (Metalava also allows (and
+ ignores) block comments in the signature files.)
* Add support for writing (and reading) annotations into the signature
- files. This is vital now that some of these annotations become part of
- the API contract (in particular nullness contracts, as well as parameter
- names and default values.)
+ files. This is vital now that some of these annotations become part of the
+ API contract (in particular nullness contracts, as well as parameter names
+ and default values.)
- * Support for a "compact" nullness format -- one based on Kotlin's syntax. Since
- the goal is to have **all** API elements explicitly state their nullness
- contract, the signature files would very quickly become bloated with
- @NonNull and @Nullable annotations everywhere. So instead, the signature
- format now uses a suffix of `?` for nullable, `!` for not yet annotated, and
- nothing for non-null.
+ * Support for a "compact" nullness format -- one based on Kotlin's
+ syntax. Since the goal is to have **all** API elements explicitly state
+ their nullness contract, the signature files would very quickly become
+ bloated with @NonNull and @Nullable annotations everywhere. So instead, the
+ signature format now uses a suffix of `?` for nullable, `!` for not yet
+ annotated, and nothing for non-null.
Instead of
@@ -114,10 +116,9 @@ example, you'll see something like this (unless running with --quiet) :
method public java.lang.Double! convert0(java.lang.Float!);
method public java.lang.Double? convert1(java.lang.Float);
-
- * Other compactness improvements: Skip packages in some cases both for
- export and reinsert during import. Specifically, drop "java.lang."
- from package names such that you have
+ * Other compactness improvements: Skip packages in some cases both for export
+ and reinsert during import. Specifically, drop "java.lang." from package
+ names such that you have
method public void onUpdate(int, String);
@@ -126,35 +127,34 @@ example, you'll see something like this (unless running with --quiet) :
method public void onUpdate(int, java.lang.String);
Similarly, annotations (the ones considered part of the API; unknown
- annotations are not included in signature files) use just the simple
- name instead of the full package name, e.g. `@UiThread` instead of
+ annotations are not included in signature files) use just the simple name
+ instead of the full package name, e.g. `@UiThread` instead of
`@android.annotation.UiThread`.
- * Misc documentation handling; for example, it attempts to fix sentences
- that javadoc will mistreat, such as sentences that "end" with "e.g. ".
- It also looks for various common typos and fixes those; here's a sample
- error message running metalava on master:
- Enhancing docs:
+ * Misc documentation handling; for example, it attempts to fix sentences that
+ javadoc will mistreat, such as sentences that "end" with "e.g. ". It also
+ looks for various common typos and fixes those; here's a sample error
+ message running metalava on master: Enhancing docs:
frameworks/base/core/java/android/content/res/AssetManager.java:166: error: Replaced Kitkat with KitKat in documentation for Method android.content.res.AssetManager.getLocales() [Typo]
frameworks/base/core/java/android/print/PrinterCapabilitiesInfo.java:122: error: Replaced Kitkat with KitKat in documentation for Method android.print.PrinterCapabilitiesInfo.Builder.setColorModes(int, int) [Typo]
* Built-in support for injecting new annotations for use by the Kotlin compiler,
not just nullness annotations found in the source code and annotations merged
- in from external sources, but also inferring whether nullness annotations
- have recently changed and if so marking them as @Migrate (which lets the
- Kotlin compiler treat errors in the user code as warnings instead of errors.)
+ in from external sources, but also inferring whether nullness annotations have
+ recently changed and if so marking them as @Migrate (which lets the Kotlin
+ compiler treat errors in the user code as warnings instead of errors.)
-* Support for generating documentation into the stubs files (so we can run javadoc or
- [Dokka](https://github.com/Kotlin/dokka) on the stubs files instead of the source
- code). This means that the documentation tool itself does not need to be able to
- figure out which parts of the source code is included in the API and which one is
- implementation; it is simply handed the filtered API stub sources that include
- documentation.
+* Support for generating documentation into the stubs files (so we can run
+ javadoc or [Dokka](https://github.com/Kotlin/dokka) on the stubs files instead
+ of the source code). This means that the documentation tool itself does not
+ need to be able to figure out which parts of the source code is included in
+ the API and which one is implementation; it is simply handed the filtered API
+ stub sources that include documentation.
* Support for parsing Kotlin files. API files can now be implemented in Kotlin
- as well and metalava will parse and extract API information from them just
- as is done for Java files.
+ as well and metalava will parse and extract API information from them just as
+ is done for Java files.
* Like doclava1, metalava can diff two APIs and warn about API compatibility
problems such as removing API elements. Metalava adds new warnings around
@@ -163,15 +163,15 @@ example, you'll see something like this (unless running with --quiet) :
but not versa). It also lets you diff directly on a source tree; it does not
require you to create two signature files to diff.
-* Consistent stubs: In doclava1, the code which iterated over the API and generated
- the signature files and generated the stubs had diverged, so there was some
- inconsistency. In metalava the stub files contain **exactly** the same signatures
- as in the signature files.
+* Consistent stubs: In doclava1, the code which iterated over the API and
+ generated the signature files and generated the stubs had diverged, so there
+ was some inconsistency. In metalava the stub files contain **exactly** the
+ same signatures as in the signature files.
(This turned out to be incredibly important; this revealed for example that
- StringBuilder.setLength(int) was missing from the API signatures since it is
- a public method inherited from a package protected super class, which the
- API extraction code in doclava1 missed, but accidentally included in the SDK
+ StringBuilder.setLength(int) was missing from the API signatures since it is a
+ public method inherited from a package protected super class, which the API
+ extraction code in doclava1 missed, but accidentally included in the SDK
anyway since it packages package private classes. Metalava strictly applies
the exact same API as is listed in the signature files, and once this was
hooked up to the build it immediately became apparent that it was missing
@@ -187,19 +187,19 @@ example, you'll see something like this (unless running with --quiet) :
2770 out of 47492 parameters were annotated (5%)
More importantly, you can also point it to some existing compiled applications
- (.class or .jar files) and it will then measure the annotation coverage of
- the APIs used by those applications. This lets us target the most important
- APIs that are currently used by a corpus of apps and target our annotation
- efforts in a targeted way. For example, running the analysis on the current
- version of framework, and pointing it to the
+ (.class or .jar files) and it will then measure the annotation coverage of the
+ APIs used by those applications. This lets us target the most important APIs
+ that are currently used by a corpus of apps and target our annotation efforts
+ in a targeted way. For example, running the analysis on the current version of
+ framework, and pointing it to the
[Plaid](https://github.com/nickbutcher/plaid) app's compiled output with
... --annotation-coverage-of ~/plaid/app/build/intermediates/classes/debug
This produces the following output:
- 324 methods and fields were missing nullness annotations out of 650 total API references.
- API nullness coverage is 50%
+ 324 methods and fields were missing nullness annotations out of 650 total
+ API references. API nullness coverage is 50%
```
| Qualified Class Name | Usage Count |
@@ -222,7 +222,8 @@ example, you'll see something like this (unless running with --quiet) :
| android.view.ViewGroup.MarginLayoutParams | 21 |
| ... (99 more items | |
```
- Top referenced un-annotated members:
+
+Top referenced un-annotated members:
```
| Member | Usage Count |
@@ -246,30 +247,33 @@ example, you'll see something like this (unless running with --quiet) :
| ... (309 more items | |
```
- From this it's clear that it would be useful to start annotating android.os.Parcel
- and android.view.View for example where there are unannotated APIs that are
- frequently used, at least by this app.
-
-* Built on top of a full, type-resolved AST. Doclava1 was integrated with javadoc,
- which meant that most of the source tree was opaque. Therefore, as just one example,
- the code which generated documentation for typedef constants had to require the
- constants to all share a single prefix it could look for. However, in metalava,
- annotation references are available at the AST level, so it can resolve references
- and map them back to the original field references and include those directly.
-
-* Support for extracting annotations. Metalava can also generate the external annotation
- files needed by Studio and lint in Gradle, which captures the typedefs (@IntDef and
- @StringDef classes) in the source code. Prior to this this was generated manually
- via the development/tools/extract code. This also merges in manually curated data;
- some of this is in the manual/ folder in this project.
-
-* Support for extracting API levels (api-versions.xml). This was generated by separate
- code (tools/base/misc/api-generator), invoked during the build. This functionality
- is now rolled into metalava, which has one very important attribute: metalava
- will use this information when recording API levels for API usage. (Prior to this,
- this was based on signature file parsing in doclava, which sometimes generated
- incorrect results. Metalava uses the android.jar files themselves to ensure that
- it computes the exact available SDK data for each API level.)
+ From this it's clear that it would be useful to start annotating
+ android.os.Parcel and android.view.View for example where there are
+ unannotated APIs that are frequently used, at least by this app.
+
+* Built on top of a full, type-resolved AST. Doclava1 was integrated with
+ javadoc, which meant that most of the source tree was opaque. Therefore, as
+ just one example, the code which generated documentation for typedef constants
+ had to require the constants to all share a single prefix it could look
+ for. However, in metalava, annotation references are available at the AST
+ level, so it can resolve references and map them back to the original field
+ references and include those directly.
+
+* Support for extracting annotations. Metalava can also generate the external
+ annotation files needed by Studio and lint in Gradle, which captures the
+ typedefs (@IntDef and @StringDef classes) in the source code. Prior to this
+ this was generated manually via the development/tools/extract code. This also
+ merges in manually curated data; some of this is in the manual/ folder in this
+ project.
+
+* Support for extracting API levels (api-versions.xml). This was generated by
+ separate code (tools/base/misc/api-generator), invoked during the build. This
+ functionality is now rolled into metalava, which has one very important
+ attribute: metalava will use this information when recording API levels for
+ API usage. (Prior to this, this was based on signature file parsing in
+ doclava, which sometimes generated incorrect results. Metalava uses the
+ android.jar files themselves to ensure that it computes the exact available
+ SDK data for each API level.)
## Architecture & Implementation
@@ -280,33 +284,33 @@ modifier lists (including annotations).
This is done for multiple reasons:
-(1) It allows us to have multiple "back-ends": for example, metalava can read
- in a model not just from parsing source code, but from reading older SDK
+(1) It allows us to have multiple "back-ends": for example, metalava can read in
+ a model not just from parsing source code, but from reading older SDK
android.jar files (e.g. backed by bytecode) or reading previous signature
- files. Reading in multiple versions of an API lets doclava perform "diffing",
- such as warning if an API is changing in an incompatible way. It can also
- generate signature files in the new format (including data that was missing
- in older signature files, such as annotation methods) without having to
- parse older source code which may no longer be easy to parse.
+ files. Reading in multiple versions of an API lets doclava perform
+ "diffing", such as warning if an API is changing in an incompatible way. It
+ can also generate signature files in the new format (including data that was
+ missing in older signature files, such as annotation methods) without having
+ to parse older source code which may no longer be easy to parse.
(2) There's a lot of logic for deciding whether code found in the source tree
should be included in the API. With the model approach we can build up an
- API and for example mark a subset of its methods as included. By having
- a separate hierarchy we can easily perform this work once and pass around
- our filtered model instead of passing around PsiClass and PsiMethod instances
+ API and for example mark a subset of its methods as included. By having a
+ separate hierarchy we can easily perform this work once and pass around our
+ filtered model instead of passing around PsiClass and PsiMethod instances
and having to keep the filtered data separately and remembering to always
consult the filter, not the PSI elements directly.
-The basic API element class is "Item". (In doclava1 this was called a "DocInfo".)
-There are several sub interfaces of Item: PackageItem, ClassItem, MemberItem,
-MethodItem, FieldItem, ParameterItem, etc. And then there are several
-implementation hierarchies: One is PSI based, where you point metalava to a
-source tree or a .jar file, and it constructs Items built on top of PSI:
-PsiPackageItem, PsiClassItem, PsiMethodItem, etc. Another is textual, based
-on signature files: TextPackageItem, TextClassItem, and so on.
+The basic API element class is "Item". (In doclava1 this was called a
+"DocInfo".) There are several sub interfaces of Item: PackageItem, ClassItem,
+MemberItem, MethodItem, FieldItem, ParameterItem, etc. And then there are
+several implementation hierarchies: One is PSI based, where you point metalava
+to a source tree or a .jar file, and it constructs Items built on top of PSI:
+PsiPackageItem, PsiClassItem, PsiMethodItem, etc. Another is textual, based on
+signature files: TextPackageItem, TextClassItem, and so on.
-The "Codebase" class captures a complete API snapshot (including classes
-that are hidden, which is why it's called a "Codebase" rather than an "API").
+The "Codebase" class captures a complete API snapshot (including classes that
+are hidden, which is why it's called a "Codebase" rather than an "API").
There are methods to load codebases - from source folders, from a .jar file,
from a signature file. That's how API diffing is performed: you load two
@@ -327,17 +331,16 @@ For example, here's how you can visit every class:
})
Similarly you can visit all items (regardless of type) by overriding
-`visitItem`, or to specifically visit methods, fields and so on
-overriding `visitPackage`, `visitClass`, `visitMethod`, etc.
+`visitItem`, or to specifically visit methods, fields and so on overriding
+`visitPackage`, `visitClass`, `visitMethod`, etc.
-There is also an `ApiVisitor`. This is a subclass of the `ItemVisitor`,
-but which limits itself to visiting code elements that are part of the
-API.
+There is also an `ApiVisitor`. This is a subclass of the `ItemVisitor`, but
+which limits itself to visiting code elements that are part of the API.
This is how for example the SignatureWriter and the StubWriter are both
-implemented: they simply extend `ApiVisitor`, which means they'll
-only export the API items in the codebase, and then in each relevant
-method they emit the signature or stub data:
+implemented: they simply extend `ApiVisitor`, which means they'll only export
+the API items in the codebase, and then in each relevant method they emit the
+signature or stub data:
class SignatureWriter(
private val writer: PrintWriter,
@@ -360,12 +363,12 @@ method they emit the signature or stub data:
### Visiting Types
-There is a `TypeVisitor` similar to `ItemVisitor` which you can use
-to visit all types in the codebase.
+There is a `TypeVisitor` similar to `ItemVisitor` which you can use to visit all
+types in the codebase.
When computing the API, all types that are included in the API should be
-included (e.g. if `List<Foo>` is part of the API then `Foo` must be too).
-This is easy to do with the `TypeVisitor`.
+included (e.g. if `List<Foo>` is part of the API then `Foo` must be too). This
+is easy to do with the `TypeVisitor`.
### Diffing Codebases
@@ -397,11 +400,11 @@ Another visitor which helps with implementation is the ComparisonVisitor:
This makes it easy to perform API comparison operations.
-For example, metalava has a feature to mark "newly annotated" nullness annotations
-as migrated. To do this, it just extends `ComparisonVisitor`, overrides the
-`compare(old: Item, new: Item)` method, and checks whether the old item
-has no nullness annotations and the new one does, and if so, also marks
-the new annotations as @Migrate.
+For example, metalava has a feature to mark "newly annotated" nullness
+annotations as migrated. To do this, it just extends `ComparisonVisitor`,
+overrides the `compare(old: Item, new: Item)` method, and checks whether the old
+item has no nullness annotations and the new one does, and if so, also marks the
+new annotations as @Migrate.
Similarly, the API Check can simply override
@@ -417,21 +420,20 @@ in which case the deletion is allowed.))
### Documentation Generation
As mentioned above, metalava generates documentation directly into the stubs
-files, which can then be processed by Dokka and Javadoc to generate the
-same docs as before.
+files, which can then be processed by Dokka and Javadoc to generate the same
+docs as before.
-Doclava1 was integrated with javadoc directly, so the way it generated
-metadata docs (such as documenting permissions, ranges and typedefs from
-annotations) was to insert auxiliary tags (`@range`, `@permission`, etc) and
-then this would get converted into English docs later via `macros_override.cs`.
+Doclava1 was integrated with javadoc directly, so the way it generated metadata
+docs (such as documenting permissions, ranges and typedefs from annotations) was
+to insert auxiliary tags (`@range`, `@permission`, etc) and then this would get
+converted into English docs later via `macros_override.cs`.
This it not how metalava does it; it generates the English documentation
directly. This was not just convenient for the implementation (since metalava
-does not use javadoc data structures to pass maps like the arguments for
-the typedef macro), but should also help Dokka -- and arguably the Kotlin
-code which generates the documentation is easier to reason about and to
-update when it's handling loop conditionals. (As a result I for example
-improved some of the grammar, e.g. when it's listing a number of possible
-constants the conjunction is usually "or", but if it's a flag, the sentence
-begins with "a combination of " and then the conjunction at the end should
-be "and").
+does not use javadoc data structures to pass maps like the arguments for the
+typedef macro), but should also help Dokka -- and arguably the Kotlin code which
+generates the documentation is easier to reason about and to update when it's
+handling loop conditionals. (As a result I for example improved some of the
+grammar, e.g. when it's listing a number of possible constants the conjunction
+is usually "or", but if it's a flag, the sentence begins with "a combination of
+" and then the conjunction at the end should be "and").
diff --git a/src/main/java/com/android/tools/metalava/ConvertJarsToSignatureFiles.kt b/src/main/java/com/android/tools/metalava/ConvertJarsToSignatureFiles.kt
index 662d7b0ee5649ad35610374f476cd061e7a21c95..5625f881a8618e51b03efa20a1100322e341a33b 100644
--- a/src/main/java/com/android/tools/metalava/ConvertJarsToSignatureFiles.kt
+++ b/src/main/java/com/android/tools/metalava/ConvertJarsToSignatureFiles.kt
@@ -54,7 +54,9 @@ class ConvertJarsToSignatureFiles {
val signatureFile = "prebuilts/sdk/$api/public/api/android.txt"
val oldApiFile = File(root, "prebuilts/sdk/$api/public/api/android.txt")
val newApiFile =
- File(root, "prebuilts/sdk/$api/public/api/android.${if (options.compatOutput) "txt" else "sig"}")
+ // Place new-style signature files in separate files?
+ // File(root, "prebuilts/sdk/$api/public/api/android.${if (options.compatOutput) "txt" else "v2.txt"}")
+ File(root, "prebuilts/sdk/$api/public/api/android.txt")
progress("\nWriting signature files $signatureFile for $apiJar")
@@ -107,6 +109,12 @@ class ConvertJarsToSignatureFiles {
SignatureWriter(printWriter, apiEmit, apiReference, jarCodebase.preFiltered)
}
+ // Delete older redundant .xml files
+ val xmlFile = File(newApiFile.parentFile, "android.xml")
+ if (xmlFile.isFile) {
+ xmlFile.delete()
+ }
+
api++
}
}