JUnit initialization failures are reported like test methods

One of my Gradle plugins processes JUnit results XML files and needs to examine the method in the class file corresponding to each test case. Loading the class and searching for the method specified in the XML usually worked fine, but recently I found out about a crash when looking for the method. Apparently if JUnit 4 is unable to initialize a test class (e.g. the static initializer fails or tests are improperly declared), it still emits a results file with failure information recorded in a test case named classMethod or initializationError. Nothing else seems to distinguish them from real test methods, so my plugin tried and failed to find a method with that name. I adjusted the plugin to watch for those names specifically - providing a useful notice - and gracefully ignore other unknown methods.

AdviceAdapter's exit advice may trigger multiple times

I recently tried to use AdviceAdapter from ASM Commons to insert a bit of code that would run before a method exited. It was important that the new code run exactly once no matter what happened inside the original function. Unfortunately the adapter seems to call onMethodExit right before every return or throw instruction. Advising on returns is fine, not so much on throws because the exception could be caught inside the method. Checking for containment in a try block would not help without some serious rewriting because a called function could be capable of throwing multiple types of exceptions, only one of which might be caught by the function being instrumented. So in AdviceAdapter's defense, its job is essentially impossible. I ended up renaming the original method and generating a "bridge" method with a simple try-finally around the call to the original.

Java monitorenter instructions cannot be interrupted

A Java application I'm helping with sometimes needs to terminate untrusted threads that got out of hand. Our sandbox seemed to be working great until I tried running two tasks that tried to synchronize on the same object and spin indefinitely once the lock was acquired. Trying to terminate the locked-out thread failed until the thread that acquired the lock was stopped. Apparently the monitorexit JVM instruction used to enter a synchronized block cannot be interrupted by stopping the thread, much less with the normal interrupt mechanism. We were already doing some bytecode editing for our sandbox, so I extended that to rewrite uses of synchronization-related instructions or methods to call static sandbox functions that simulate the originals using ReentrantLock, locking functions of which can be halted by thread shutdown.

Infinite try-catch loops can crash the JVM

Previously I discovered that the Java compiler sometimes emits exception table entries set to "handle" an exception by jumping back to the start of the covered region. This led to deadlock if an exception was indeed thrown in that region, usually due to bytecode rewriting inserting instructions. It was manageable to fix during bytecode rewriting with ASM by dropping exception table entries (not calling super.visitTryCatchBlock) that had the same start and handler label. Unfortunately there are other circumstances, particularly with while loops inside synchronized blocks, in which the compiler emits an entry with the handler in the middle of the covered region. I saw this crash the JVM entirely, citing an "invalid class file" but providing no additional information with -Xverify:all. Based on the access-violation error code, it seemed to be a stack overflow in the JVM itself. Avoiding this problem is more difficult because ASM doesn't provide a way to know the order of labels until all the exception table entries have been visited. I had to make the ClassReader first accept a visitor that recorded the indexes of problematic try-catch blocks, then use that information during the real transformation that dropped the offending entries.

Exposing a Kotlin function only to Java code

I'm involved in a Kotlin project that's intended to be used from both Kotlin and Java code. Because Java doesn't have optional/default-value parameters, we use @JvmOverloads to generate overloads with subsequences of parameters, but we also end up writing a lot of overloads manually to remove optional parameters in the middle. These pollute Kotlin autocomplete lists, so I wanted to hide from from Kotlin code.

The @SinceKotlin annotation is meant to hide a member from sufficiently old versions of Kotlin. Specifying a bogus, extremely large version number hides the member from Kotlin in general. I defined a JAVA_ONLY string constant with a large version like 999.0, so the hiding annotation looks like @SinceKotlin(JAVA_ONLY), which is nice enough. This annotation doesn't mean anything to the Java compiler, so it's perfectly happy to use the function.

When Java exceptions don't include a stack trace or message

I was helping someone investigate a crash in their Android Studio project's tests today. One exception totally lacked all context: it was just the exception class name, no message, no stack trace. It turns out Java has a performance feature that skips generating the stack trace in certain cases for common exceptions. The feature can be disabled by adding -XX:-OmitStackTraceInFastThrow to the JVM command-line arguments. That brought the stack trace back, which was very helpful for finding the bug.

Avoiding infinite throw-catch loops

Previously I encountered a problem when doing bytecode instrumentation of try-catch blocks. The Java compiler sometimes emits exception table entries that register short regions as their own exception handlers. When code that can cause exceptions is inserted into the middle of these regions, an infinite loop results.

To avoid this with the ASM bytecode library, I just added a check in visitTryCatchBlock to see if the handler label was the same as the start label. If so, the function returns without passing the event along, removing the exception table entry. This can't break anything because if the handler was ever used under normal circumstances, it would trigger a thread-termination-resistant infinite loop, which I don't observe.

Java exception tables can allow infinite loops

Today I worked with an application that does some Java bytecode instrumentation to intercept exception handling. The exception table of a method specifies which regions to detect exceptions in and where to jump if an exception occurs. In one class, an exception table entry's target ended up being the start of the covered region. If an exception occurred inside the region, an infinite loop would begin. That would even prevent the thread from being terminated because the ThreadDeath exception couldn't propagate out of the function.

Peril of reentrant defineClass: ClassCircularityError

Today I tried to make a Java application faster by eagerly loading a lot of classes into a classloader at once when the first of a set was requested. This did not work well. When the JVM loads a class in response to a defineClass call, it makes sure that there are no cycles in the inheritance hierarchy. This check is done immediately, requesting all parent classes from the classloader. If those haven't been loaded yet, my classloader would be called again (with the first defineClass call still on the call stack) and try to eagerly load a bunch of additional stuff. Apparently the JVM doesn't expect unrelated classes to be defined in this fallout from a defineClass call: if the next class defined isn't the next one up the inheritance hierarchy, a ClassCircularityError will be thrown even in the absence of actual circularity.

Android Gradle 3.5.3 doesn't clean the desugared local JARs directory

One of my Gradle plugins helps swap out a handful of JAR dependencies in Android projects. I noticed that after using it for a while on an Android Gradle 3.5.3 project, building an APK would eventually start failing with a "program type already present" error.

I determined that the problem occurred when two copies of the same JAR dependency ended up in the external_file_lib_dex_archives intermediates directory. Files there are named with a combination of a number (increasing within one run) and the original filename. I observed the same JAR with different numbers, presumably from different builds. That directory is populated by DexFileDependenciesTask, which from reading the code does not seem to ever clean its output. So if the order of JARs changes or if one is removed, extra JARs will remain and clog up the dexer.

I worked around this in my plugin by registering a doFirst action on all DexFileDependenciesTasks that deletes and recreates the task's output directory.

Building Gradle plugins with Maven

Today I started writing a Gradle plugin module for a project that uses Maven as its build system. I know it's possible, with some fiddling, to involve a Gradle build in a Maven setup, but this project is managed by someone else and I wanted to stay with their preferred technology as much as possible.

Since Gradle plugins are ultimately just JARs, it's possible to compile them with Maven. Of course, this loses Gradle's convenient facilitation of writing Gradle plugins. Specifically, you need to add various dependencies yourself rather than just using gradleApi(). These <dependency>s generally look like:

<groupId>org.gradle</groupId>
<artifactId>gradle-something</artifactId>
<version>5.6.4</version>
<scope>provided</scope>

I'm currently using version 5.6.4 because it's the latest version of the Gradle 5 series, which supported by Android Studio and used by a lot of projects. The "provided" scope makes it used only for compilation, never bundled with the plugin. You may also need a reference to Groovy.

The various types are spread across quite a few artifacts. For a pretty straightforward plugin (doing some extra stuff with the results of Java compilation), I needed dependencies on gradle-core, gradle-core-api, gradle-base-services, gradle-model-core, gradle-language-jvm, gradle-language-java, gradle-platform-jvm, gradle-plugins, and gradle-tooling-api. To determine which artifact provides a class the compiler can't find, you can search the Gradle monorepository.

For adding the plugin metadata, the properties file can go in the same place under resources/META-INF that it would with a Gradle build.

Jackson KotlinModule signature changed between versions

I'm working with several Gradle plugins that use Jackson for JSON and/or YAML parsing. They're written in Kotlin, so they take advantage of the Jackson Kotlin module. I found after trying to upgrade some plugins' dependencies that Gradle sync always crashed with a NoSuchMethodError regarding the KotlinModule constructor. It appears that that constructor gained a new optional parameter - which changes its JVM method signature - in multiple versions, which is a binary-incompatible change. That makes differences in dependency version very likely to cause crashes. Updating some but not all plugins to 2.10.x causes a problem; 2.9.8 and 2.9.9 seem to be fine.

Constant pool entries are reused for different kinds of constants

Names of class members in Java class files as indexes into the file's constant pool. Changing (using something like BCEL) the value of the UTF8 constant used by a member's name renames the member. However, that constant might also be used for something that's not a reference to that member, like a string literal in code or a reference to a different class's member of the same name, so that rename strategy could break things. Instead, new UTF8 and name-and-type constants should be added and the member reference constants should be adjusted to refer to those. Alternatively, a higher-level library like ASM may be more convenient.

Name collisions due to similarly obfuscated libraries

I'm currently working with a project that has several Android AAR library dependencies, independently built and obfuscated with R8. This setup initially caused crashes with NoSuchMethodErrors for the constructor of classes with obfuscated names like a.a.a.a.a.a. Apparently the obfuscation of multiple dependencies remapped different classes to the same name, only one of which was loaded into the application. Since the classes didn't have the same shape, the uses from one dependency required members that weren't there. I worked around the problem by directing R8 to -keep one of the classes that was mapped to the collided name according to mapping.txt.

Compiling one additional file with an Android Gradle JavaCompile task

I recently needed to create a Gradle task to compile a single Java file in an Android project after the normal compilation of a variant's source set was complete. I found some inspiration from this Gradle forum thread, but things are more complicated when using the Android Gradle plugin rather than the plain Java one. I'm not sure how to get the appropriate paths from the project configuration, so I grab them from an existing Java compile task for the variant. task is the new JavaCompile task; javaTask is the existing one for the variant (e.g. compileDebugUnitTestJavaWithJavac):

task.options.sourcepath = project.fileTree("src/test/java")
task.source = task.options.sourcepath!!.asFileTree
task.include("com/example/package/TheFile.java")
task.classpath = javaTask.classpath + javaTask.outputs.files
task.options.bootstrapClasspath = javaTask.options.bootstrapClasspath
task.destinationDir = javaTask.destinationDir
task.sourceCompatibility = javaTask.sourceCompatibility
task.targetCompatibility = javaTask.targetCompatibility

(That's Kotlin code; Groovy will probably be similar minus the !!.) The file can be excluded from the original compilation task's inputs like so:

javaTask.excludes.add("com/example/package/TheFile.java")

PowerMock stubbing can help deal with subclasses of private classes

Today I needed to mock an inner class that was a subclass of a different, private inner class. Creating a mock or spy failed in the class initializer because the generated mock class didn't have access to the private superclass. However, using PowerMockito.stub on a PowerMockito.method representation of the method to replace worked.

NullPointerExceptions from PowerMock can indicate a non-mocked static class

Today I was writing some PowerMock-based test code and encountered a NullPointerException in the middle of normal-looking and previously-working mock configuration for a static method. Another test in the same test suite was failing with an UnfinishedStubbingException in something that also looked reasonable. It turned out that I had forgotten to PowerMockito.mockStatic the class in that test.

Finalizers could be problems for sandboxing

The finalize method of any Java object will run after garbage collection when all references to the object have been lost. This occurs on a different "finalizer" thread at an unspecified time. So if an untrusted task is run in a sandboxed thread/ThreadGroup and terminated, there may still be objects waiting to be finalized. If sandboxing is based on ThreadGroup, the untrusted code there will no longer be sandboxed. Even if the sandboxing is based on class loader, an infinite loop could hang the finalizer thread, interfering with finalization of other objects. To prevent these problems, my organization uses bytecode manipulation to drop finalize methods from untrusted classes before running them in a sandbox.

A timed out thread pool task isn't necessarily canceled

I'm involved in a JVM application that uses an Executor to run a task and calls get with a timeout. This is done as part of a "dry run" to make sure a component is reasonably well behaved and doesn't take ages/forever to run. It was hoped that get would cancel the task if it failed to complete in time, but it does not. Rather, the calling thread is allowed to proceed, but the task thread is still going in the background, possibly in an infinite loop. As a workaround, the Callable we run on the other thread records its Thread so that the main thread can stop it if the task times out. Of course, this doesn't work if the code under test catches ThreadDeath, but ours is known to be only possibly-mistaken, not malicious.

PowerMock indirectly allows setAccessible on any method

The CS class I'm on staff for is looking into ways to grade Java Android app-based assignments securely. Restricting student code without breaking Robolectric is tricky - we need to allow e.g. reflection when done by Robolectric or the test suites but not when done by student code. Some of our test suites use PowerMock, which of course internally exercises all kinds of permissions. Unfortunately it could be used by student code to override access modifiers and call arbitrary methods: WhiteboxImpl provides the public function getAllMethods, which makes all a class's methods accessible and returns them.

So checking whether trusted frameworks are using the dangerous permissions is insufficient. Fortunately, we make PowerMock a testImplementation dependency, so it's not on the compile classpath for the student sources. While students could access it at runtime via reflection (the problematic method is public), our SecurityManager can check whether PowerMock is being invoked reflectively and deny permission if so.