Kotlin Gradle buildscripts can't easily apply arbitrary plugins from other repositories

Previously I published a Gradle plugin to Maven Central. It had been working fine in Groovy buildscripts, but using it from a Kotlin buildscript was tricky. It registers an extension, so using the apply function to apply it didn't work; the extension needed the plugin already applied to even compile. The plugins block didn't immediately work because it looked for a plugin in the Gradle Plugins Portal, or through some kind of placeholder/marker artifact of a different name in other repositories.

The way to deal with this is to add a pluginManagement block to settings.gradle to set which artifact coordinate Gradle will use to look for the plugin of a given ID. repositories should specify the repository containing the plugin plus gradlePluginPortal() for any other plugins. resolutionStrategy → eachPlugin is run for every requested plugin and can intercept and change the artifact coordinate. The requested plugin information can be checked to see if it's the tricky one; if so the useModule function should be called to set the artifact coordinate.

The phpBB 2 to 3 converter might blank a few posts with non-ASCII characters

Today I received a report from a user on a forum I host that a couple old posts appeared blank. These were made before I migrated the forum from phpBB 2 to phpBB 3, a transition that was made tricky by a problem with non-ASCII character encoding. Examining the phpbb_posts table showed that the affected posts' post_text had been replaced with <?xml version="1.0"?> plus a linefeed. Consulting a backup of the phpBB 2 database, I saw that both posts originally contained non-ASCII characters that get encoded into multiple bytes by UTF-8.

Detecting whether an Android ID is defined

I'm writing some test suites for student Android code. I'm trying to minimize the tests' compile-time dependencies on student code so that students can work on one part at a time without having compilable versions of everything. Some assignments ask them to set up layouts, in which view IDs create constants on R.id. The integer value of an ID constant can be looked up by name using the getIdentifier method of Resources. It can be called with all information in the first parameter...

resources.getIdentifier("com.example.app:id/theName", null, null)

...or with the resource type and package in separate parameters...

resources.getIdentifier("theName", "id", "com.example.app")

The return value is zero if the ID doesn't exist. That can be checked and used to throw a nice error message at runtime when that test is run rather than a compile error.

PowerMock stubbing can help deal with subclasses of private classes

Today I needed to mock an inner class that was a subclass of a different, private inner class. Creating a mock or spy failed in the class initializer because the generated mock class didn't have access to the private superclass. However, using PowerMockito.stub on a PowerMockito.method representation of the method to replace worked.

Ionic needs a full restart for some changes

The Ionic web server has a slick feature that detects changes to the source files, recompiles the appropriate components, and delivers the changes to the browser automatically. However, this does not appear to work for all changes. In particular, I've seen adding new TypeScript files cause strange behavior that was resolved by stopping and restarting ionic serve.

Working with support library AlertDialog in Robolectric

Robolectric provides a ShadowAlertDialog class with a getLatestAlertDialog function to get the last Android AlertDialog. However, in my project that uses a support library (AndroidX), that function returned null. To get the last shown dialog, I had to use ShadowDialog.getLatestDialog and cast that Dialog to the AlertDialog type from the support library.

NullPointerExceptions from PowerMock can indicate a non-mocked static class

Today I was writing some PowerMock-based test code and encountered a NullPointerException in the middle of normal-looking and previously-working mock configuration for a static method. Another test in the same test suite was failing with an UnfinishedStubbingException in something that also looked reasonable. It turned out that I had forgotten to PowerMockito.mockStatic the class in that test.

Exotherm - Need a new host

Mercuric lost connection with FICS some time today. To see if it crashed from a Python exception, I looked at the log, but the file was empty. Previously I had suspected that the OS was terminating it to reclaim memory, but then I wondered if some management service was terminating long-running processes since this kind of server is usually just used for web hosting. I did some research and discovered that the hosting company does not like standalone long-running processes, which is fair since their plans are designed for web sites. I shut the bot down for now and will need to find a new home for it - so far PythonAnywhere looks promising.

Exotherm - Can't investigate disconnection when it doesn't disconnect

Mercuric usually disconnects from FICS after about a day, but it's somehow still up after I added logging yesterday. Looks like I'll have to wait a little longer before I can determine whether the random disconnects are due to crashes.

Exotherm - Investigating disconnection

For quite a while, I've been planning to look more into why Exotherm (deployed as "mercuric") disconnects from the FICS after a few hours. Today I finally did something about it: in case the disconnection is due to a crash, I added a bit of code to log any errors to a file before exiting. The bot will probably have fallen over by morning - then I can check the log to see if a bug is causing it to crash.

Robolectric shadows can help mock final classes

PowerMock can mock final classes, but this requires rewriting them into a different classloader. In some cases, like getting a SupportMapFragment from the Google Maps SDK, PowerMock's reload leads to errors saying a something can't be cast to its own type. (The class gets loaded twice and instances are not convertible between the two.) Adding the class to @PowerMockIgnore fixes that, but then PowerMock can no longer work with it.

Fortunately, Robolectric does its own bytecode manipulation. Shadows can replace methods from Android classes. In the process of installing a shadow for a class, Robolectric seems to make it non-final. So declaring an empty shadow is sufficient to allow Mockito.mock to create mocks of it.

@Implements(GoogleMap.class)
public class ShadowGoogleMap { }

The shadow class needs to be registered either in robolectric.properties or in the shadows property of the @Config annotation on the test suite.

Finalizers could be problems for sandboxing

The finalize method of any Java object will run after garbage collection when all references to the object have been lost. This occurs on a different "finalizer" thread at an unspecified time. So if an untrusted task is run in a sandboxed thread/ThreadGroup and terminated, there may still be objects waiting to be finalized. If sandboxing is based on ThreadGroup, the untrusted code there will no longer be sandboxed. Even if the sandboxing is based on class loader, an infinite loop could hang the finalizer thread, interfering with finalization of other objects. To prevent these problems, my organization uses bytecode manipulation to drop finalize methods from untrusted classes before running them in a sandbox.

A timed out thread pool task isn't necessarily canceled

I'm involved in a JVM application that uses an Executor to run a task and calls get with a timeout. This is done as part of a "dry run" to make sure a component is reasonably well behaved and doesn't take ages/forever to run. It was hoped that get would cancel the task if it failed to complete in time, but it does not. Rather, the calling thread is allowed to proceed, but the task thread is still going in the background, possibly in an infinite loop. As a workaround, the Callable we run on the other thread records its Thread so that the main thread can stop it if the task times out. Of course, this doesn't work if the code under test catches ThreadDeath, but ours is known to be only possibly-mistaken, not malicious.

Policy Plus - Dependencies cleanup

I recently noticed that Policy Plus had dependencies on a few .NET assemblies from which it never used anything. In the interest of cleanliness I removed those.

I also investigated a report of an "an item with the same key has already been added" error when loading policy definitions. I was unable to reproduce it; IDs are qualified by the target namespace of the ADMX file that defines them and defining a duplicate target namespace produces a different error. I'll remain on the lookout for problems related to that system.

PowerMock indirectly allows setAccessible on any method

The CS class I'm on staff for is looking into ways to grade Java Android app-based assignments securely. Restricting student code without breaking Robolectric is tricky - we need to allow e.g. reflection when done by Robolectric or the test suites but not when done by student code. Some of our test suites use PowerMock, which of course internally exercises all kinds of permissions. Unfortunately it could be used by student code to override access modifiers and call arbitrary methods: WhiteboxImpl provides the public function getAllMethods, which makes all a class's methods accessible and returns them.

So checking whether trusted frameworks are using the dangerous permissions is insufficient. Fortunately, we make PowerMock a testImplementation dependency, so it's not on the compile classpath for the student sources. While students could access it at runtime via reflection (the problematic method is public), our SecurityManager can check whether PowerMock is being invoked reflectively and deny permission if so.

Java MethodHandles can work like reflection

I'm helping test a Java execution sandbox. We want to allow untrusted code to use streams, so we can't deny the suppressAccessChecks and accessDeclaredMembers permissions, so instead we made the classloader for untrusted code reject anything in the java.lang.reflect package. Today I found that the MethodHandles API allows essentially the same capabilities as reflection. A lookup object obtained through privateLookupIn on a class belonging to a classloader that is an ancestor of the sandbox classloader can bypass the class lookup restrictions. Handles can then be obtained to normal reflection API methods, which can be used to override access modifiers.

Blocking the entire java.lang.invoke package isn't viable because streams/lambdas need parts of it, but blocking MethodHandles and MethodHandles.Lookup should make it impossible to dynamically invoke arbitrary methods.

When Kotlin tests fail with "class not found" in Android Studio

Today I was setting up a new Android Studio project. For reasons, the app had to be in Java, but I wanted to write the test suite in Kotlin. The test class and test methods had the run buttons in the margin, and the build appeared to go smoothly, but trying to run them produced a "class not found" error instead of test results. It turns out I had forgotten to apply the kotlin-android plugin. (That plugin comes from the usual Kotlin buildscript dependency.) It's also important to add the Kotlin standard library as a normal dependency, as testImplementation in my case.

Calling MethodHandle#invoke from Kotlin

Java has a class called MethodHandle which represents a dynamic but strongly typed operation. It provides invoke and invokeExact methods that are "signature polymorphic" and handled specially: rather than treating their invocations as the use of a normal varargs method, the compiler emits bytecode involving the invokedynamic instruction.

Since calls to these methods require compiler support, other JVM language compilers may not handle them properly. Kotlin 1.3.x doesn't generate the special bytecode by default and therefore cannot call signature-polymorphic methods correctly. However, the correct behavior can be enabled with the -XXLanguage:+PolymorphicSignature argument to the compiler. Alternatively, the similar invokeWithArguments method can be called normally, but may be a little slower due to the extra work of transforming the normal varargs call into a dynamic invocation.

Fixing the signing plugin on Gradle 5.1 and later

I previously found that using the maven-publish and signing Gradle plugins to publish to Maven Central did not work on Gradle versions 5.1 and newer: the signArchives task failed with a "duplicate key" error. So for a while I had to use the Gradle 5.0 wrapper to deploy. More recently I decided to investigate more thoroughly, and came up with a fix. Conveniently, the list of signatures is available from the signing task. Rather than this simple configuration block...

signing {
    sign configurations.archives
}

...I remove duplicates...

signing {
    def signTasks = sign configurations.archives
    signTasks.each { t ->
        def signatures = t.getSignatures()
        signatures.removeAll { oneSig ->
            signatures.count { anotherSig -> oneSig.getClassifier() == anotherSig.getClassifier() } > 1
        }
    }
}

...allowing the signing and upload to complete successfully.

The Robolectric RuntimeEnvironment isn't accessible from a different classloader

Today I worked on a SecurityManager that oversees the execution of Robolectric tests. In one check, I needed to consult the RuntimeEnvironment of the test. Trying to simply use members of that class gave wrong results; it appeared that no tests had been started yet. The problem was a difference of classloader. Robolectric loads reloads tests into a "sandbox" class loader to provide SDK isolation. Each loader got its own RuntimeEnvironment with its own set of static members. Since the SecurityManager was loaded in the system loader, it always referred to the copy in the system loader, which isn't used for tests. The solution was to go through the class context to get the test class loader, then use reflection on that to get the desired properties.

Robolectric's sandbox classloader doesn't preserve the ProtectionDomain

Today I investigated why my organization had been having trouble applying a ProGrade security policy to Robolectric tests. The policy we used for previous projects granted and denied rights to the untrusted code using codeBase directives, but they seemed to have no effect. It turns out that Robolectric "sandboxes" code by SDK version by reloading classes into different classloaders. The sandbox classloader doesn't seem to preserve the ProtectionDomain/CodeSource of the sandboxed classes, so the information about their origin in an untrusted location is lost. Since we control which directory structures (and therefore packages) can be compiled, I'm working on writing a SecurityManager that discriminates by class/package.