Allowing standard users to run a privileged scheduled task

A Super User asker wanted to allow all users to perform one specific action that usually requires administrative privileges. Application developers wanting to do this should create a service with an RPC mechanism, but a quick way for a scripter to do it is to take advantage of the Task Scheduler service. First, a scheduled task that runs something as SYSTEM (no triggers necessary) can be created as usual.

Then the trick is to adjust the task's access control list. Neither the Task Scheduler MMC snap-in nor the relevant PowerShell module provide a way to change tasks' security descriptors, but the IRegisteredTask COM interface implements IDispatch, so it can be used in PowerShell. To obtain a task object:

$ts = New-Object -ComObject 'Schedule.Service'
$ts.Connect('localhost')
$task = $ts.GetFolder('\').GetTask($taskName)

The GetSecurityDescriptor method takes a flags argument specifying which parts of the security descriptor to get as a SDDL string; 4 gets just the DACL. Appending (A;;FRFX;;;BU) adds an extra access control entry that allows (A) read (FR) and execute (FX) access to the Users group (BU). The adjusted SDDL can be applied with the SetSecurityDescriptor method. Standard users will then be able to see and run the task, but not alter it to do something else or delete it. 

Applying shims to standard Windows programs

While browsing for information about the Windows application compatibility shim engine, I found someone remarking that they had trouble applying shims to Notepad for demonstration purposes.

It is possible to shim standard Windows programs like Notepad, but an extra step is required. Like Windows DLLs, EXEs under System32 are excluded from hooking by the default inex policy. The shim definition must therefore have an INCLUDE directive for the executable, such as in this ShimDBC XML fragment:

<SHIM NAME="CustomShim" FILE="AcRes.dll" RUNTIME_PLATFORM="X86_ANY,AMD64">
    <INCLUDE MODULE="notepad.exe"/>
</SHIM>

The shim can then be applied as usual:

<APP NAME="Notepad">
    <EXE NAME="notepad.exe" RUNTIME_PLATFORM="AMD64" FILE_DESCRIPTION="Notepad">
        <SHIM NAME="CustomShim"/>
    </EXE>
</APP>

A few standard shims - specifically RedirectEXE, InjectDll, and TerminateExe - refuse to perform their function on system EXEs as determined by a function called ShimLib::IsSystemExeFromSelf, which appears to check whether the EXE file is owned by TrustedInstaller. That's not a concern for custom shims, though.

64-bit support in the custom shim development kit

The initial release of my custom Windows appcompat shim development kit only supported 32-bit shim modules. Today I downloaded IDA 7, which became freely available recentlyish and supports 64-bit disassembly. I looked at the 64-bit versions of a couple of the official shim modules and the shim engine interface looked very familiar, keeping in mind that x64 has only one calling convention, reminiscent of fastcall. The 64-bit version of apphelp.dll seemed to have all the exports I used in my 32-bit shims.

So I optimistically compiled my apphelp import library in the x64 configuration, configured the CustomShim project to use it in the x64 configuration as well, and compiled the shim module as 64-bit. I addressed a few compiler warnings that resulted from assuming size_t is the same as int, then went to register my 64-bit shim module. Playing around with ShimDBC, I found that RUNTIME_PLATFORM attributes accept comma-separated lists of platforms, so I can indicate that a shim exists in both the 32-bit and 64-bit modules with X86_ANY,AMD64. It turns out that one SDB can apply fixes to both 32-bit and 64-bit executables - just leave off the -op switch when compiling in Custom mode to make one big SDB - and Windows accepts this, so it's not clear why there's an AppPatch64 directory or why the Compatibility Administrator comes in two versions.

After installing my new dual-bitness SDB and placing the 64-bit version of my shim module in System32, I observed my example shims applied to Sysinternals AccessChk64 and Autoruns64. The updates to the Visual Studio projects and documentation are live on GitHub.

Custom Windows application compatibility shim development kit

As might be guessed from my recent posts, I've been poking at the Windows application compatibility infrastructure for a while. I am now pleased to present a custom shim development kit that can be used to create real 32-bit shim modules that provide real shims applied with the real shim engine by the same mechanism as for the standard shims you see in the Compatibility Administrator. Custom shims are of course not supported by Microsoft, but my approach appears to work on Windows 10 version 2004. I am aware that Detours exists; I did this just for fun, though the shim engine's support for COM hooks turned out to be more convenient than Detours' in my opinion. WinRT hooks are apparently also a thing the shim engine can do, but I don't know how they work and don't currently facilitate them.

You can read about how to build custom shims in the shim module's README and how to use/apply/debug them in the overview README, so I'll just use this post for extra details and asides. 

But first, a demonstration. The example shim module includes a shim called FakeSchTask that hooks (i.e. intercepts) the Task Scheduler 2 COM API to make the application see a scheduled task in the root folder that isn't actually there. Like the other example shim, this is pretty silly and is implemented in a way targeted specifically at a Sysinternals utility I had laying around (Autoruns in this case), but it's a fine example of how to write a shim. The fake scheduled task's name defaults to "Fake Task" but can be specified as the shim command line. Placing the compiled shim module as AcRes.dll in SysWOW64, compiling the XML in the project's README using ShimDBC, installing the SDB, and running Autoruns:

Notice the top Task Scheduler entry in Autoruns and its absence from the real list

The first step in making this happen, once it's established that the process being started needs to be shimmed, is that the shim engine in apphelp.dll loads each needed shim by calling the GetHookAPIs function exported from the shim's DLL. It's supposed to return a pointer to an array of HOOKAPI structures specifying imported functions to redirect through the shim. At some point this involves comparing the requested shim name to the name of each shim implemented in the DLL and getting the right one's hook array. Microsoft's shim modules have, per shim, a namespace containing one major function that handles various callbacks including installation and sometimes a couple other functions, apparently implemented with a library called ShimLib that probably provides some macros in addition to the functions common to the several shim modules. I went for a more object-oriented approach, implementing each shim as a subclass of the abstract Shim class. My GetHookAPIs scans the list of shim instances for the one with the requested name, invokes the command line processing then hook-list-populating functions of that shim object, and adds it to the list of active shims for later notification.

Throughout the process's life, the other shim module exported function NotifyShims is called many times with different reason codes. After shims are installed with GetHookAPIs, the shim engine notifies shim modules about every DLL the executable is linked with (and presumably their dependencies), providing a pointed to the loader data table entry. (I wouldn't have been able to identify that structure myself. Thanks to the ReactOS project for having some of this reverse-engineered.) After all those, it notifies shim modules that everything is initialized. This appears to be a good time to install COM hooks. As DLLs are delay-loaded or unloaded, there may be more DLL-related notifications.

Shims are loaded very early in the process's life, so you have to be very careful about what features are available when. Loading the CLR in an attempt to write shims in a managed language would not go well. Using C++'s new operator in GetHookAPIs originally crashed with an access violation, probably because something important hadn't been loaded yet, so I replaced the standard memory management operators with implementations that wrap LocalAlloc and LocalFree, which are available. Even so, declaring static/global variables with constructors that allocate (e.g. std::vector) crashed the process with exit code 0xC0000142. This is why my knownShims and activeShims variables are pointers that are initialized inside functions called by GetHookAPIs at the earliest. They are never destructed, but there's only one leak of each type per process, so it's not really a leak.

While writing up a Stack Overflow answer that used my kit, I tried to use SprintDLL to quickly check that a Win32 function was being hooked. By default .NET applications' PInvoke calls won't be hooked because clr.dll is excluded by the standard "inex policy", which I learned by reading the shim engine logs after seeing SprintDLL fail to reflect the hook. If your shim is getting installed but isn't working, reading the ApphelpDebug log is a good way to see what's going on. 

Unfortunately I also noticed that SprintDLL sometimes failed to start when being shimmed, hanging for multiple seconds then exiting with a breakpoint-related NTSTATUS. The shim engine log said it "failed to lock engine", which is an error-level log event, so it brought the process down when no debugger was around to handle the DebugBreak. Running under WinDbg, a thread did indeed fail to RtlEnterCriticalSection on the engine lock, while the thread holding that lock was in the ASL-log-writing code waiting to acquire some other lock: classic deadlock. (ASL is the shims logging system, as opposed to the shim engine log. Maybe it stands for AppCompat shim log or layer.) Disabling the shims log fixed the problem. Along the way I noticed that shim engine logs also get sent to debug output and that a bunch of shim-related stuff happens before the debugger breaks into a new process, which I'm sure would make more complex issues interesting to debug.

Speaking of ways to crash the process, it is critical to get the calling convention right for hook functions and pointers to original functions. If a convention is not specified, it defaults to fastcall, which is not right for any Win32 function I know of, leading to stack imbalance then an access violation or stack cookie fast fail. Similarly, getting the parameter sizes wrong, either directly in the signature or by associating a hook function with a real function you didn't mean, will ruin the stack.

To reduce accidental mismatches as much as possible, I did some macro and template tricks. The shim engine provides the original/next function in a field of the HOOKAPI structure that was given to it by GetHookAPIs, so calling it from inside a hook function involves indexing that array, which requires keeping track of the order hooks were declared and using the right index in each hook function - easy to get wrong if done manually. Instead of having you build the array yourself, I provided an ADD_HOOK macro for use in RegisterHooks overrides that appends the hook information to a vector maintained by the Shim superclass and records its index in a struct templated by the hook function, effectively stapling the index to the function. The index is looked up in that struct by the DEFINE_NEXT macro (for use inside hook functions) and passed to a superclass function that extracts the next function pointer from the hooks array. By using decltype in a cast it spares you from repeating, and possibly mismatching, the function signature. Hook functions have to be static; the macro gets access to the shim instance through a static variable set in the constructor defined by the SHIM_INSTANCE macro. I would have liked to parameterize by hook function name to remove the need to pass even the current hook function to DEFINE_NEXT, but the Microsoft C++ compiler doesn't seem to fully support string literal template parameters, and I didn't want to use a map because of the runtime cost.

COM original function lookup is simpler because the SE_COM_Lookup function provided by the shim engine already takes a "this" pointer and hook function, both of which are easily accessible and not prone to accidental mismatches. It is pretty easy, though, to mismatch the vtable index during member function hook registration. Unfortunately I can't do anything about this unless CINTERFACE is defined, which would disable the nice C++ interfaces, so you'll just have to very carefully count the indexes of fields in the headers' C Vtbl structs yourself.

Hooking member functions of registered COM classes is pretty straightforward: you tell SE_COM_AddHook the class ID, interface ID, function index, and your replacement function and it intercepts the COM instantiation machinery (used by e.g. CoCreateInstance) to hook the function when an object of that type is created. The shim engine cannot, however, detect or interfere with COM objects allocated directly, like those created and returned by existing objects. You can still register hooks by IID and vtable index, but such hooks can't be applied automatically. Instead, once you get ahold of an object implementing that interface (in a hook function that was applied automatically), you can call SE_COM_HookObject to apply the hooks to it. Despite the function name, that generally applies the hook to the entire class since all instances of a class usually all point to the same vtable. The shim engine can't copy the vtable because it doesn't know the size. If you only want your hooks to interfere with some instances of the internally instantiated class, you will need some way for them to distinguish interesting instances. My FakeSchTask shim's hooks of ITaskFolder use get_Name to check if the folder is the root folder. Before I noticed that helpful property, I noted the pointer to the root folder in the ITaskService::GetFolder hook and checked the "this" pointer against it in the ITaskFolder hooks, but this was a mess because objects can be freed, allowing the address to be reused. The shim engine doesn't seem to support multiple hooks of the same COM function, and since it already hooked all the IUnknown functions, I couldn't instrument Release. Instead of juggling pointers, for IRegisteredTaskCollection objects produced by ITaskFolder::GetTasks, I wrapped the real objects of interest in an instance of my own FakeTaskCollection class before returning from my GetTasks hook. That way I didn't have to hook any IRegisteredTaskCollection functions. To save you some boilerplate I put the base interface IUnknown implementation and IDispatch stub, which could be useful for other wrapper classes, in a separate file. Implementing IDispatch in plain C++ is not my idea of a good time, so my objects don't actually support scripting - which is conveniently not necessary for shimming Autoruns - but I hear that generating type libraries with MIDL can make this easier.

Interestingly, the Task Scheduler 2 header taskschd.h doesn't define the CLSID_TaskScheduler GUID the way most COM component headers do. It just declares the variable extern, requiring linking with taskschd.lib, which was not in the default linker input for me. If you get "unresolved external symbol" errors about class or interface IDs and have already included initguid.h, you probably need to reference another LIB.

Well! I think that's all the adventure notes I have. Please don't rely on any part of this for anything too important, but if you have any questions or suggestions, I'd be happy to take a look via the GitHub issue tracker.

Policy Plus - Explicit checkbox elements and policy state detection

This morning I received a report from a Policy Plus user that its assessment of a policy state was inconsistent with that of the real Local Group Policy Editor. The policy had been set to Disabled in the official editor, but Policy Plus reported it as Unknown, which indicates that some Registry evidence favors it being Enabled while other evidence suggest it's Disabled. Indeed, the policy's main Registry value was deleted, but the value corresponding to a checkbox element was present and set to 0. Zero is that element's "off" value, but an unchecked checkbox is different than the whole policy being Disabled.

Experimenting with several different policies involving checkbox elements, I found that the Local Group Policy Editor writes checkbox elements' "off" values to the Registry when the policy is Disabled if the element has data explicitly defined. If the element's data is left implicit/default, disabling the policy deletes its Registry value like I expected. I still think it would be better if things worked the way I had previously assumed, since the way it actually works allows cases (in real policies!) where "policy Enabled but checkbox unchecked" is literally indistinguishable from "policy Disabled", but Policy Plus needs to reflect reality rather than my preferred design.

When judging a policy's state, Policy Plus now considers the presence of a checkbox element's "off" value as weak evidence for the policy being Disabled. When saving the state of Disabled policies, it writes checkbox elements' "off" values, if specified, instead of deleting them. While I was at it, I adjusted the setting editor window to avoid crashing when editing a mixed-state policy, not that there should be any anymore. The changes are live on GitHub.

Turning the Compatibility Administrator into ShimDBC

Microsoft creates the application compatibility database (SDB) files from XML using an internal utility apparently named ShimDBC. Geoff Chappell found that this utility is embedded within Compatadmin.exe, the Compatibility Administrator utility that comes with the freely available Windows Assessment and Deployment Kit. To access ShimDBC functionality, we need to patch Compatadmin to pass command-line arguments of our choosing through to it. The linked article provides a patch, but for an older version. I'll provide a general approach for producing such a patch and provide the specific changes for the 32-bit Windows 10 2004 edition (file version 10.1.19041.1).

We will need two utilities: a disassembler and a hex editor. I'll use IDA 5 (though apparently version 7 is free now) and XVI32. Fire up IDA and open the PE executable Compatadmin.exe using the default settings. Tell IDA to download the PDB when prompted, wait for the initial autoanalysis to complete, then use the Functions tab to navigate to the WinMain function, the starting point of the application. We can't just write our whole patch over the beginning of this function, though, because the loader might decide to load the module at a non-preferred address, in which case it would add offsets to addresses of called functions to compensate, so we'll need to use an existing fixup site. Look down for the first call instruction and click it to see its address in the status bar.

In the above screenshot, this call instruction is at 0x3A467 in the file and at an offset of 0x29 from the start of the function. We need this to calculate a jump offset from the first instruction. We're going to replace the first instruction with a short jump to get to the call site, and a short jump takes 2 bytes, so the jump offset will be 2 less: 0x27. Click the first instruction (a mov) and record its address from the left eight-hex-digit display: 0x3A43E for me. Open XVI32 to a copy of the Compatadmin executable and use Address | Goto to go to that absolute address. The highlighted byte should be hex 8B. Type over that byte with EB (short jump), then the next byte with the offset to the call site. To double-check the jump offset, use Goto again to go relative down by your offset. The byte you land on should be FF (call absolute indirect).

We now need to find the import address table entry for GetCommandLineW so we can pass the command line to the database compiler. Start typing that function name in IDA's Imports window and it should take you to the entry. We will need the Address value in little endian, so the 004B60D0 I see becomes D0 60 4B 00. Back in XVI32, leave the FF 15 intact and overwrite the next four bytes with your little-endian address. The ShimDBC invoker was compiled as a fastcall function, so it takes its first/only argument in the ECX register, so we need to move the result from EAX to ECX: overwrite the next two bytes with 89 C1.

At this point we are getting unfortunately close to another call instruction, whose address could be adjusted by the loader, thereby ruining our code. We will need to jump over it. Back in the main viewer on WinMain, click the first instruction after the second call.

From its file offset (0x3A477 in the screenshot), subtract the current XVI32 cursor position (0x3A46F for me), then subtract 2 more for the size of the short jump: I get 6. Like before, overwrite the next byte with EB, then the next with the jump offset.

Now we need to make a near call to the ShimDBC invoker, named ShimdbcExecute in the current version. Use IDA's Functions tab to navigate to it, click its first instruction (a push for me), and note its file offset: I get 0x5630F. From that, subtract the file offset of the first post-call instruction we just used, and further subtract 5 to account for the length of the call instruction: I get 0x1BE93, or 93 BE 01 00 as a little-endian double-word. In XVI32, go to the absolute address of that first post-call instruction. Overwrite the byte you land on with E8, then overwrite the following four bytes with the offset.

Finally, we just need to exit the function restoring the stack appropriately for the sixteen bytes' worth of parameters that were passed. Overwrite the next three bytes with C2 10 00.

The EXE is still declared as a GUI program, so to get console output from it you would need to redirect it to a file. That can be fixed by changing the subsystem ID in the PE header. Near the very beginning of the file, you will see the letters "PE": select the P, then go relative down 0x5C bytes. You should land on an 02 (GUI) - change it to 03 (CUI).

In summary, my changes are:

• At 0x3A467: EB 27 (replacing 8B FF)
• At 0x3A46A: D0 60 4B 00 (replacing D8 60 4B 00)
• Following at 0x3A46D: 89 C1 (replacing 6A 04)
• Following at 0x3A46F: EB 06 (replacing 8B FE)
• At 0x3A477: E8 93 BE 01 00 (replacing 33 C0 66 89 84)
• Following at 0x3A47C: C2 10 00 (replacing 24 80 01)
• At 0x154: 03 (replacing 02)

For additional convenience, you can remove the program's desire to run as administrator by searching for "highestAvailable" and typing over it in the right/ASCII pane with "asInvoker" plus spaces after the closing quote to cover the extra length.

When LoadLibrary fails with ERROR_MOD_NOT_FOUND for an existent library

Today I wrote a small DLL in C++ and tried to use it on another machine. Strangely, the host process didn't load it, nor could a manual LoadLibraryW call in SprintDLL. I did have the path right and Process Monitor confirmed that the file was being opened (by both the host process and CSRSS for some reason), but LoadLibraryW returned null with ERROR_MOD_NOT_FOUND. To get more details, I used the GFlags utility to enable "show loader snaps" for SprintDLL, attached WinDbg to the SprintDLL process, and tried the load call again. Output in the debugger showed that loading my DLL also triggered the loading of vcruntime140d.dll, which did not exist on my other machine. So a module was not being found - it was "just" a dependency instead of my library itself. Using a Release build, which links against the release version of that redistributable which is actually present on the other machine, fixed the problem.

Windows compatibility shim logs

One of the ways Windows provides backward compatibility with applications targeted to older OS versions is by loading compatibility shims into their processes to intercept and alter various Windows API calls. apphelp.dll provides most of the machinery for this, at least for user mode. Interestingly, that DLL and the shim DLLs it can load are capable of logging some information about what they're doing or errors they encounter, though those log entries don't go anywhere by default. Out of curiosity I poked around in IDA to figure out how to enable them.

The SepApplyDebugPolicy function, called early in SE_InitializeEngine, is responsible for initializing some global variables related to logging. It reads two environment variables, SHIM_DEBUG_LEVEL and SHIMENG_DEBUG_LEVEL, into global variables controlling the verbosity of shims' logs and AppHelp itself's logs respectively. Level 5 produces messages of all detail levels including "trace." These logs seem to be designed for consumption via ETW Trace Logging, but I was unable to record any relevant events with WPR. 

They can also, however, output to files. For shim logs, AppHelp also consults a SHIM_FILE_LOG environment variable which specifies a filename under %WINDOWS%\Temp to save the log as. The code tries to take just the filename - the segment after the last backslash - but ../ sequences can be used to traverse up the directory tree and place the log anywhere you have write access to. For AppHelp (the shim engine) itself, there is no environment variable to set filenames; logs are always saved in that temp directory with a name starting with AslLog_ApphelpDebug. The shim engine state (?) is saved as an XML document in an adjacent file with a name starting with AslLog_shimengstate. These logs have to be enabled with a LogFlags DWord value in HKLM\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags in the Registry. Setting that to 0xFF (255) does the job nicely.

For completeness' sake: SepApplyDebugPolicy also looks for a DWord value named ShowDebugInfo in that key, but the effect is not clear to me from the disassembly and I did not notice any difference in behavior.

Uninstalling old Windows Assessment and Deployment Kit versions

I recently went to use the Compatibility Administrator from the Windows Assessment and Deployment Kit, but was blocked by the Program Compatibility Assistant telling me it doesn't support my version of Windows. Indeed, I had installed it a long time ago on a previous build of Windows 10, 10586. So I downloaded the new ADK installer, but it refused to install unless I removed the old version first. That's reasonable, so I tried to uninstall the 10586 version from the Apps settings page, but it failed with "setup cannot locate WimMountAdkSetupAmd64.exe" and prompted me to "locate content." This is probably related to build 10586 no longer being supported and the relevant ADK downloads being removed from Microsoft's web site. I found a couple copies of that file laying around the disk, but when I selected them, the process failed at 33% with "unable to verify the integrity of downloaded content." Trying a "repair" installation, even with a newly downloaded copy of the old version's installer, failed with the same errors.

I could, however, use the "change" setup feature to remove all ADK components except Deployment Tools. Going back to a WimMountAdkSetupAmd64.exe file I found (Explorer search does a fine job of locating a copy under C:\ProgramData\Package Cache), I ran it from an administrative command prompt with the /uninstall switch. That completed without incident, after which using "change" in the ADK setup to remove the Deployment Tools component worked! Since that was the last component, the ADK vanished from the apps list and I could successfully install the new version.

Policy Plus - Fix list element REG export

I received a Policy Plus issue report on GitHub stating that the Export REG feature failed. After experimenting with the GPOs on a couple machines, I found what was necessary for the crash: list policy elements that clear the list key before adding values. When the PolFile representing the GPO tried to apply the key clearance, it requested the list of existing values in the target RegFile, which crashed because the REG file implementation of PolicySource is write-only. Changing ApplyDifference to use the ClearKey interface method instead fixed the problem.

But in the course of hunting down the issue and testing the fix, I found multiple other problems related to list elements. The POL editor window had an off-by-one error that caused a crash if adding a "clear the key before adding values" directive to the last key in the list - an easy fix. The list editor window (launched from the setting editor dialog) showed the **delvals. pseudo-value on subsequent launches of the setting editor for list elements that clear the key first. That was caused by the PolFile implementation of GetValueNames returning raw POL value names, which is important for the POL editor window but problematic here. For consistency with the RegistryPolicyProxy implementation, I changed the interface method implementation to return only true values and added another method to get raw POL entries.

The changes are live on GitHub.

The Windows 10 global microphone off switch

Today I spent a good chunk of time hunting down an audio issue with Windows 10. A USB audio input device (the microphone embedded in a webcam in this case) was recognized by Windows and applications, but never seemed to actually provide any sound. No bar appeared to indicate the level in the sound settings when the microphone was spoken into. Strangely, Google Hangouts just re-muted audio input shortly after the unmute button was pressed. 

After a great deal of messing around, I eventually found a "privacy" switch that essentially completely disables microphone use by desktop applications: Settings → Privacy → Microphone → "allow desktop apps to access your microphone." You'll probably have to scroll down; the switch at the top only affects modern/UWP/Metro apps.

It would be really helpful if there was a clear indication that this setting is off in the places one might try to manage the microphone: the Sound settings page, or at least the troubleshooter.

JUnit initialization failures are reported like test methods

One of my Gradle plugins processes JUnit results XML files and needs to examine the method in the class file corresponding to each test case. Loading the class and searching for the method specified in the XML usually worked fine, but recently I found out about a crash when looking for the method. Apparently if JUnit 4 is unable to initialize a test class (e.g. the static initializer fails or tests are improperly declared), it still emits a results file with failure information recorded in a test case named classMethod or initializationError. Nothing else seems to distinguish them from real test methods, so my plugin tried and failed to find a method with that name. I adjusted the plugin to watch for those names specifically - providing a useful notice - and gracefully ignore other unknown methods.

When MATLAB says "too many output arguments" after newly declaring one

I recently adjusted a MATLAB function in an existing codebase to add a new optional output argument. I was confused to see that trying to invoke it with the new argument failed with "too many output arguments." (I did save the file! And tried restarting MATLAB.) The problem was that the function was also declared in a class file, which I hadn't modified. Updating that declaration to match the implementation file fixed the problem. I imagine this also applies to "too many input arguments."

FMod - Minor audio bug fixes

Tonight I got some Abiathar audio improvements packaged into a beta release for the person who requested them. Along the way I found a couple minor other bugs. Previously the music mapping patches would be generated if audio files were linked regardless of whether Song Mappings had been used, leading to a zero-length song list being patched. Audio export apparently (probably as of v2.11.2) wrote zero-byte IMF files for empty chunks in the IMF range. This is fixed now so that empty chunks are always properly skipped.

FMod - Automatic audio chunk boundary adjustment

Today I spent a little time continuing improvement of Abiathar's audio import process. It now checks whether the chunk ID regions used by each type of sound are compatible with the archive layout that the project is configured to use for export. If not, it offers to adjust the project configuration to be consistent with the added chunks. The adjustment code is fiddly since I'm trying to avoid shifting the "original" IMF start chunk when possible so that Music Mappings doesn't break unnecessarily.

FMod - Relaxing audio assumptions

FleexCore2's GalaxyAudio class used to require the layout of the audio archive to be provided in a chunk settings object. As of v2.11.2, only the ImfEnd field is used, since chunks will only be encoded or decoded when needed but the total number of chunks must be specified to save the archive. Abiathar still populated the chunk settings directly from its project's audio settings block, which assumes that an IMF region will exist and will be at the end of the audio archive. As part of the audio IO improvements started yesterday I relaxed some assumptions in this area. There might not be any IMF songs, so the maximum chunk ID given to FleexCore2 is now computed as the maximum of the last chunk ID of each of the three audio chunk ranges.

FMod - Starting audio IO improvements

A PCKF member is working on a mod with several extra sounds and often needs to adjust Abiathar's audio archive layout settings. This is somewhat difficult to do - it requires editing the project file manually. I don't have time for a major overhaul of Abiathar's audio support or for the addition of a fancy UI, but I have a couple ideas for useful improvements. Tonight I added a + directive to the ASNDS format that can be used in place of a chunk ID to automatically assign the next ID, which can remove the need to manually renumber everything after inserting a chunk. There's also now a BLANK directive to skip a chunk and advance to the next ID.

JGit's addCeilingDirectory adds an exclusive bound

A Gradle plugin I worked on uses the JGit library to check the state of the project's Git repository, if it's in one. A discrepancy in behavior between Java versions led me to investigate the FileRepositoryBuilder setup. addCeilingDirectory is supposed to set the outermost directory that will be searched by findGitDir, but due to the order in which findGitDir updates its "directory under examination" variable and checks whether the ceiling has been hit, the specified ceiling will actually not be checked. To work around this, I specified the parent directory of my intended ceiling.

Local Julia packages

I'm working on some Julia scripts and recently needed to write some utility functions usable from multiple files. I could be very careful with includes to make sure the file is included exactly once when needed, but that seemed rickety to me. I wanted it to be like a package I could confidently bring in with a using directive. Unfortunately this doesn't seem to be common in the Julia ecosystem and only really became possible (?) recently.

I had to make a subdirectory for the utility package containing a separate project with its own metadata. So the MyModule subdirectory had a Manifest.toml, Project.toml, and src subdirectory, containing MyModule.jl. The metadata files were created by the package manager, but I had to edit Project.toml to give the package a name and uuid. Then in the package environment for the root project, I ran develop MyModule to add the utility package as a development/local dependency.

Policy Plus - ADMX download suggestion

A Policy Plus user filed an issue today stating that it was not showing many policies. I suggested using Help | Acquire ADMX Files if they were on a Home edition and this brought in all the expected policies. That step is mentioned in the GitHub README, but not all users will see that. So I added a first-run check: if Policy Plus is running on a Home edition and there are several orphaned non-empty categories (which happens if referenced ADMX files are missing), the user will be asked whether they want to download ADMX files. Selecting Yes produces the standard Acquire ADMX Files dialog. This should improve the new-user experience.

FMod - v2.11.3

I received a report from a PCKF user that Abiathar was unable to open a certain mod's levels because one level was wider than 255 tiles. Indeed, Abiathar's level size restrictions structure uses one byte to store each dimension, so it couldn't handle larger levels. Such levels have weird behavior in-game past coordinate 255, but they can be played. 

So I increased the default limits to 65,535 (the maximum that can be stored in the standard level format). It wasn't as simple as just increasing some constants, though, because some parts need to pack two coordinates into one 16-bit number, mostly having to do with links. I added some checks to reject link destinations with coordinates larger than 255. Then I found that saving really huge levels crashed because their compressed plane data sizes couldn't be fit into two bytes. I introduced a "maximum area" field to the level size restrictions and added checks for this to the level properties dialog and row/column manipulators.

Since inability to open valid levels is a major problem, I released these changes (and the IMF-related ones waiting in beta) immediately as v2.11.3.

Finding out what a "Host Process for Windows Tasks" is doing

Today I noticed a "Host Process for Windows Tasks" process (taskhostw.exe) with heavy disk usage. That name isn't very specific, and indeed the program seems to be just a holder for various tasks' modules. To look at it more closely I fired up Process Explorer as administrator and opened the instance of taskhostw.exe with the most activity. I switched to the Threads tab and looked at the thread with CPU activity. Its Start Address was in SetupCleanupTask.dll, which is located in C:\Windows\System32\oobe.

That was at least more specific, but it didn't tell me exactly what that module does. I went back to the main Process Explorer screen and set View | Lower Pane View to Handles to see what the process had open. Selecting the process and looking through the lower pane, I found that it had some log files open in C:\Windows\Logs\SetupCleanupTask. Skimming those logs suggested that this module checks to see if it's been long enough (10 days) since the last Windows feature update was installed, then deletes the C:\Windows.old folder, reclaiming disk space.

Tiny Blogger glitch: extra space

Since Google revamped the Blogger management UI to be more modern and bubbly, there's been a tiny glitch with the post editor. When starting a new post, the editor box initially contains one space, so clicking in the box and just starting to type makes the first line start shifted to the right slightly. Deleting the initial space is easy, it's just a strange UI thing.

Selecting a Python virtualenv in Visual Studio Code

I'm using Visual Studio Code to do some Python development over SSH. The project is in a virtualenv, so many packages are not available system-wide, so for autocomplete to fully work VSCode needs to recognize the virtual environment. For some reason I thought this would be more complicated than it is: when selecting the Python interpreter, pick the python symlink inside the virtualenv's bin folder. I also enabled the "Activate Env In Current Terminal" so the shell in the Terminal tab would be immediately ready to go. VSCode doesn't consistently display the environment as a virtualenv in the status bar, at least over SSH, but autocomplete picked up the libraries.

When VMs won't keep an IPv4 address

Today while troubleshooting Internet slowness on my laptop I noticed that a virtual machine I had running lacked an IPv4 address. It had an IPv6 address and could connect to some sites including the host machine. After some stumbling around I found this Ask Ubuntu answer which mentioned clock skew. Sure enough, my modem/router's time was an hour off, possibly due to automatic Daylight Savings Time adjustment being disabled? I fixed that and made the VM reattempt DHCP configuration. It got an IP address, but with a very short lease. A little while later I found that its IP became the same as the host's, which was concerning. I only got it to hold onto a unique IP by adding a DHCP reservation in the router settings.

Measuring mouse sensitivity

While writing the Super User answer mentioned in yesterday's post, I needed to measure the ratio of physical mouse movement to pointer movement, making sure my 1:1 acceleration curve was actually equivalent to the desired unaccelerated setting. One of the posts I linked supplies a ZIP file which contains a MouseMovementRecorder ZIP file which contains a program that prints every mouse movement. I launched that program, made a very sudden horizontal or vertical mouse movement, and identified an output line with large distances. Dividing the large component of the "pointer movement" column by its counterpart in "mouse movement" produced the desired ratio, 2.5 by default on my system, though it probably varies by DPI. After setting an acceleration curve with a different slope, I measured again to make sure the ratio was scaled by the desired factor.

Setting an arbitrarily precise mouse sensitivity

The Mouse Properties dialog has a slider with discrete tick marks for setting the mouse sensitivity. The MouseSensitivity value in HKCU\Control Panel\Mouse has twice the resolution but is still an integer, with a range from 1 to 20. The mapping of that number to sensitivity multiplier is tabulated here.

It is possible to set an arbitrarily precise sensitivity scale by taking advantage of the SmoothMouseXCurve and SmoothMouseYCurve values, which are consulted when "enhance pointer precision" is on. These are explained in great depth in this article, but in summary they are a set of points that describe a piecewise linear function of physical mouse speed to pointer speed. If the curve is changed to a straight line, the fancy pointer-precision-enhancing effect is disabled: a mouse movement's distance alone, not speed, determines the pointer movement distance. Since the curve points are 16.16 fixed-point values, there are far more than 20 options for the slope. Starting from a curve equivalent to the default sixth tick (provided by this site)...

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Control Panel\Mouse]
"SmoothMouseXCurve"=hex:\
    00,00,00,00,00,00,00,00,\
    C0,CC,0C,00,00,00,00,00,\
    80,99,19,00,00,00,00,00,\
    40,66,26,00,00,00,00,00,\
    00,33,33,00,00,00,00,00
"SmoothMouseYCurve"=hex:\
    00,00,00,00,00,00,00,00,\
    00,00,38,00,00,00,00,00,\
    00,00,70,00,00,00,00,00,\
    00,00,A8,00,00,00,00,00,\
    00,00,E0,00,00,00,00,00

...all Y values can be multiplied or all X values can be divided by the sensitivity multiplier of choice. Only the first 4 bytes of each line matter, and they're in little-endian. A logoff/logon cycle may be necessary for the change to take effect.

Based on my Super User answer.

Commas have high precedence in PowerShell even when intended for array subscripting

I recently received a confusing error message from PowerShell regarding an expression like this:

$Some2DLongArray[$SomeIndex + 1, 0]

Method invocation failed for op_Addition because Object[] has no such method. Indeed, there is no operator that will add a single value to an array (from the left at least). But there shouldn't have been any object array involved here. Apparently PowerShell interprets the comma as delimiting elements of the array literal 1, 0 then tries to add that array to $SomeIndex. To get my expected interpretation of using $SomeIndex + 1 as the first dimension's index and 0 as the second's I had to enclose $SomeIndex + 1 in parentheses.

Windows feature updates may remove support for old file shares

I recently updated a Windows 10 machine with the newest feature update. Afterward it was unable to open a folder shared from a Windows 95 VM. Opening the share produced an error about SMB 1.0 no longer being supported. Fortunately it is possible to reenable support for the old protocol by installing an optional component. In the Windows Features dialog (run optionalfeatures), enabling "SMB 1.0/CIFS File Sharing Support" or specifically "SMB 1.0/CIFS Client" made the share usable again.

The Woocommerce gettext hook receives already-escaped HTML

Today I needed to change the "Billing & Shipping" label on a Woocommerce checkout page. Conveniently there is a gettext hook to which functions can be attached to postprocess the results of translation. This thread has an example, but just adjusting the target text did not work. It turns out that the first $translated_text parameter receives HTML-escaped text, so I actually needed to compare against "Billing & Shipping".

Creating tables with SQL in phpPgAdmin

I'm working on setting up a PostgreSQL database hosted on Namecheap's shared hosting, which seems to only allow access through the phpPgAdmin web app. I'm testing things in a local PostgreSQL instance with a different interface and copying the table definitions to the shared server as SQL "create" commands when ready. I first tried pasting it into the box on the "SQL" tab of the database object, but phpPgAdmin seems to wrap the command in a way that makes it invalid. However, there is a small "SQL" link in the far upper right of the page that produces a new window. Arbitrary commands can be successfully run from there.

Invoking an object as a function in Kotlin JS

Some JavaScript libraries provide function objects with additional members. Expressing these as Kotlin objects in external class declarations is somewhat difficult. They can be given an operator fun invoke and called like a function using method call syntax, but the compiler will produce a call to an invoke method that doesn't exist. That could be re-@JsNamed to call and appear to work in some cases, but it's not really the right thing to do since call has the special behavior of setting this.

The expected approach seems to be to drop to weak typing by reinterpreting the object as a function. The class can be given an inline extension operator invoke along these lines:

inline operator fun Express.invoke(): ExpressApp {

    return this.unsafeCast<() -> ExpressApp>()()

}

Alternatively the receiver can be treated as the dynamic type:

inline operator fun Express.invoke(): ExpressApp {

    return this.asDynamic()() as ExpressApp

}

PostgreSQL users may need to be added to databases manually on Namecheap

I recently set up a PostgreSQL database on Namecheap's shared hosting platform. The cPanel interface allowed me to create the database and a user for it, but even after adding the user to the database my app was denied access to the tables. It seems that cPanel didn't actually grant the new user any permissions on the database. To do that, I had to go into phpPgAdmin, go to the database, switch to the Privileges section, and use the Grant link to grant the user privileges.

Watch out for returns inside inline functions

Blocks passed to Kotlin inline functions can return from the function calling the inline one - this is one of the advantages of inline functions. However, the inline function should be very careful that it's safe to return successfully (not just exceptionally) after calling a block passed to it. Today I was stumped for a bit on a database transaction wrapper function that mysteriously failed to issue the "commit" command after running the block. The block passed to it contained a plain return statement, which exits the entire enclosing function, when I meant to return a value from the block. The immediate fix was to change the return to return@theInlineFunction, but to avoid the issue in the future I adjusted the inline function to issue the commit in a "finally" block after checking that the "catch" block was not triggered by a crash in the block.

Running Kotlin Node JS projects outside Gradle

It's pretty easy to set up a Kotlin JS project targeting Node JS using the Kotlin JS Gradle plugin, but it's less easy to figure out how to actually run it on a server. The build/js folder contains the compiled JavaScript code, but accompanied by a ton of unnecessary stuff like copies of artifacts that are available on NPM. The compiled version of just the project under development is in a subdirectory of build/js/packages named for the project. Though that folder contains a package.json file that npm install can use, there is no clear way to start the app. For it to work with environments like Passenger, I amended the build process to write an app.js file there containing only:

require('./kotlin/the-project.js'); 

Then only the directory containing that new app.js has to be deployed.

When Android Studio consistently says "default activity not found"

Today I worked on updating my Android app to the newest versions of various components. When I went to run the app, it failed with a "default activity not found" bubble. I tried File | Invalidate Caches / Restart and clearing the .idea folder - both good things to do when Android Studio gets stuck - but they did not change the error. After some stumbling around I happened to look at the Merged Manifest tab of the AndroidManifest.xml editor. The right pane showed an error about one of my dependencies having a minimum API level above that of my app. Switching to a slightly older version of that dependency fixed the problem.

Kotlin JS should usually catch Throwable

As I worked on starting a Node application in Kotlin JS, I found that one of my endpoints failed when tested by Invoke-WebRequest with an error about an incomplete response. I knew this indicated a crash in my endpoint handler, but it was not obvious how to get details on what went wrong. A try-catch block to handle Exceptions did not catch anything. Changing the catch block to handle all Throwables, however, caught the problem and was able to display it.

Converting between coroutines and promises in Kotlin JS

Most modern JavaScript libraries handle asynchronicity by passing around "promise" objects that can be chained together to make a workflow with several suspension points. These are perfectly usable in Kotlin JS, but it would be even nicer if Kotlin's excellent support and cleaner syntax for coroutines and suspending functions could be used.

In a coroutine, a promise can be awaited (i.e. its behavior called in a suspending manner) with its await() extension function, which suspends. A coroutine context can be turned into a promise object by putting the suspending code inside a GlobalScope.promise block (or that of any other CoroutineScope). Such promises can be used by plain JavaScript code.

"getsockname failed: Not a socket" on Windows

New Windows 10 versions come with a Windows port of OpenSSH, so ssh can be done from the Windows command line even outside Bash on Ubuntu on Windows. I tried to set up connection reuse (ControlMaster and ControlPath directives in the SSH configuration), but then found that all connections to my specifed host failed with "getsockname failed: Not a socket." If I understand correctly, OpenSSH is trying to use UNIX sockets to coordinate between ssh processes. Alas, it cannot figure out how to use sockets on Windows. As far as I can tell, SSH connection reuse does not currently work on Windows.

No clear way to store resources in Kotlin JS projects

I'm working on a Node JS application written in Kotlin with the new Kotlin JS plugin. I needed to include some static files (web page templates) with the application, but it is not at all clear how I should connect them to the project. Gradle created a resources folder, but items in it didn't seem to be copied to anywhere in the build output. It looks like the old Kotlin2Js plugin worked with Webpack for deployment (?), but I see no such configuration or build step in my project. So I rigged up a post-compile step to just copy all the resources to the same folder as the compiled JavaScript files:

tasks.getByName("compileKotlinJs") {
    val resFolder = project.file("src/main/resources")
    inputs.dir(resFolder)
    doLast("copy resources") {
        resFolder.copyRecursively(project.file("build/js/packages/${project.name}/kotlin"), true)
    }
}

Invoke-WebRequest can work when curl doesn't

I'm working on a Node JS app hosted in Namecheap's shared hosting. I rigged up a test POST endpoint to make sure I configured everything correctly then tried to access it with a cURL invocation on the command line. That failed with a 404 and an Express message stating that it couldn't GET a document that looks likely to be used for 403 Forbidden errors. (It took an unfortunate amount of time to realize that it was an Express error page - I first assumed requests weren't being routed to the Node JS app at all.) Bizarrely, the equivalent Invoke-WebRequest command in PowerShell worked. It looks like cURL's request offended a "security" feature provided by Namecheap. Specifically, it provided different headers:

POST /endpoint HTTP/1.1

Host: example.com

User-Agent: curl/7.55.1

Accept: */*

In contrast to PowerShell's request:

POST /endpoint HTTP/1.1

User-Agent: Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.19041.1

Content-Type: application/x-www-form-urlencoded

Host: example.com

Content-Length: 0

Connection: Keep-Alive

Get-Content string values have file information stapled to them

Today I wrote a quick PowerShell script that involved reading lines from a file with Get-Content and putting each into an object that was later serialized with ConvertTo-Json. To my surprise, instead of lines being serialized as standard JSON strings, they become JSON objects with a "value" property and several others describing the file the line came from. I checked the line variable's type with .GetType() and it was a standard string. Piping it to Get-Member, however, showed that it had several NoteProperty members. Note properties can be associated with arbitrary objects. They don't touch the actual .NET object, but PowerShell keeps track of them and ConvertTo-Json serializes them. It's strange (IMO) for Get-Content to staple this information to every line string since I'd expect transformation to a different serialization format to be a common task. I worked around it by calling .ToString() on the annotated string object to get an unannotated copy.

The kotlin.js plugin obsoletes kotlin2js

I tinkered with a new Kotlin for JavaScript project today and ran into some confusion. The documentation I found frequently mentioned features that didn't seem to be recognized by Gradle and third-party examples did a lot of things that I wouldn't have expected to be necessary. Older documentation mentioned applying a kotlin2js plugin, which apparently had a lot of features but is deprecated in favor of the new Kotlin JS plugin that is applied by default when creating a Kotlin/JS project in IntelliJ. Alternatively, it can be added as a plugin with kotlin("js"). 

Manual Node setup doesn't seem to be necessary. Dependencies from NPM can be added with implementation(npm(artifact, version)) in the standard dependencies block. I have not yet found a good way to package and deploy for NodeJS, though.

Theming the PCKF

This past week I spent some time customizing the theme of the Keen forum. A new theme was long overdue from the migration to phpBB 3. I started by just replacing a bunch of CSS colors to a green-and-yellow theme like the previous forum, which helped a lot by itself. Community members suggested other adjustments including a header background image from the game. That took some fiddling to get right but looks really good!

Policy Plus - Versioning with Git

For a long time, Policy Plus's Help | About dialog stated that the program had no version number because the project was in early development. The development is no longer so early, and I received a request to release stable binaries, so it seemed like time to add a version system. Since Policy Plus isn't at 1.0 yet, I'm not ready to start issuing normal version numbers. Instead, each Git commit can represent a version. I rigged up a batch file to write the current tag or commit into a file that gets compiled into the program and used to show the version. I then released the May2020 snapshot.

Markeen level project progress 5/30

Today I finished decorating the ninth level in my Markeen-inspired level pack. While playtesting again I noticed that it was not that hard on the "hard" difficulty setting, so I added a couple creatures. I then selected a machine-generated starting point for the tenth level. I'm starting to run out of good material from the original batch of 100 levels: there were none with the last of the four fuse machine colors, so I had to swap out a machine and give the surrounding area a makeover. I've started cleaning up the platform tiling. So far I haven't been able to come up with a route for the player to take through the level, though I do have a theme planned.

More machine-assisted level design progress

A couple days ago I had started the "fix the glaring tiling errors" phase for the ninth level of my Markeen-assisted level pack. Markeen had produced a neat centerpiece for the level, but I wasn't sure about the route I was going to have the player take through it. Today I punched a small number of holes in the generated structure to create a path for the player that makes me think of convection. The player goes down one side and cannot return up that way, so they must instead return via a door whose path they should clear earlier. I got on a roll and made the level playable, fixed the distribution of point items, and did the background. All that's left to do for this one is decorate.

FMod - Grid size inconsistency fix

While using Abiathar to make some of my own levels today, I noticed a small bug. I enabled the grid to see tile boundaries in the simultaneous tileset, but the grid was too fine: 16x16, even though I had zoomed the tileset to a tile size of 32x32. The grid size was appropriate for the level viewer, though. Apparently I missed a spot when giving the simultaneous tileset its own zoom level. This is fixed now.

Gradle scripts for publishing to Maven Central shouldn't sign specific files

About a year ago, I deployed a Gradle project to Maven Central, which was made difficult by the signing requirement. I found a third-party guide that provided a very long snippet of buildscript to explicitly sign the POM (project manifest XML file) and various other files. After updating the project to Gradle 6, that didn't work anymore; the files that got signed were not the ones actually uploaded, so the signatures were invalid. Gradle 6 also introduced a new file that wasn't getting signed properly. After some poking around I discovered that signing specific files is not necessary when using the maven-publish plugin. The signing plugin can sign an entire publication as long as its configuration is written after the part of the buildscript that declares the publication:

signing {

    sign publishing.publications.getByName("mavenJava")

}

That change also made my check for duplicate signed files unnecessary.

A full working version can be found on GitHub.

Machine-assisted level design progress

Intermittently over the last few years (!) I've been working on a Keen 5 level pack based on levels generated automatically by Markeen. I have a lot of time over the summer, so I have a chance to make some progress. Previously I had fixed all the major tiling errors in the eighth level and made it largely playable, but I wasn't happy with the flow. It's OK to have some optional areas, but it was very difficult to design a route for the player, given the constraints, that would visit even a third of the level. Looking at it fresh today I thought of a new route that I like a lot. It makes the level less sensical as a physical place, but I think a theme of "space station under construction" salvages it. The foreground is done and I just need to finish decorating.

Facebook's /me/feed endpoint can't create posts with just one image

I recently wrote a script to migrate posts from a Facebook download-your-information archive to a new Page, including all attached photos. Since many of the posts had multiple images, I had the script upload each as photo to the /me/photos endpoint unpublished, then attach all of them to one new post created with /me/feed. I then made another request to edit the post to backdate it. This worked perfectly for posts with multiple images, but for posts with just one image, the backdating request failed with "you cannot backdate unpublished posts." Apparently single-photo posts are a different kind of thing than posts with multiple photo attachments. For the former, I had to make a normal (published) /me/photos request and get the post_id from that for backdating.

Facebook's download-your-information JSON encodes non-ASCII as UTF-8

Today I wrote a script to reupload Facebook posts from the JSON files produced by their download-your-information tool. Getting the post text itself was straightforward except for non-ASCII characters. JSON escape sequences for control characters specify 16 bits, but Facebook's never used more than a byte. Apparently they store their strings as UTF-8 and encode each byte to JSON. The .NET Framework (used by my script) represents strings as UTF-16, so Facebook's text was mangled. To get the correct text, I had to take the string from the JSON, map each character to a byte, and decode the byte array as UTF-8 to produce a UTF-16 string in memory.

FMod - Song format flexibility

Today I rearchitected the GalaxyAudio class in FleexCore2 (which represents a Keen Galaxy audio archive) to lazily parse the different kinds of sounds only when requested. If a parsed sound is requested, the stream of bytes is parsed and the sound is stored in a cache. Other components can request raw chunks, which serialize the sounds back to bytes if they've been parsed or just return the original byte stream if not. Abiathar now avoids triggering the parsing of IMF songs unless needed to turn a headerless IMF into a chunk. This allows preserving audio entries of arbitrary format in IMF chunks.

Since FleexCore2 changed, the ImfPreview extension had to be updated as well. While I was working on that I also made ImfPreview use Abiathar's standard "one moment" dialog instead of its own copy.

FMod - Song name migration

An Abiathar user asked for the ability to import a different format of song into the audio archive. I think the best way to implement this would be to allow importing any file directly into the audio archive without having Abiathar try to parse it. Unfortunately FleexCore2 currently isn't designed to load files without parsing them fully. Abiathar also assumes that IMF songs in the audio archive can have a "tag block" after the song data and takes advantage of that area to store the user-provided song name. Not every format will have space for a tag, so today I changed the Music Mappings dialog to store the name in the ADEPS (Abiathar project) file instead. 

That would be a simple change if not for the backwards compatibility concern. Users may already have set song names that existing Abiathar versions then wrote into the audio archive. It would be bad to lose those when updating. I wrote some migration code to detect when the project's audio files haven't been saved since an old Abiathar version and, if so, load them using the old architecture just to copy the names from the tags to the project file. With this working, I can start on a new architecture to allow greater format flexbility.

AdviceAdapter's exit advice may trigger multiple times

I recently tried to use AdviceAdapter from ASM Commons to insert a bit of code that would run before a method exited. It was important that the new code run exactly once no matter what happened inside the original function. Unfortunately the adapter seems to call onMethodExit right before every return or throw instruction. Advising on returns is fine, not so much on throws because the exception could be caught inside the method. Checking for containment in a try block would not help without some serious rewriting because a called function could be capable of throwing multiple types of exceptions, only one of which might be caught by the function being instrumented. So in AdviceAdapter's defense, its job is essentially impossible. I ended up renaming the original method and generating a "bridge" method with a simple try-finally around the call to the original.

Java monitorenter instructions cannot be interrupted

A Java application I'm helping with sometimes needs to terminate untrusted threads that got out of hand. Our sandbox seemed to be working great until I tried running two tasks that tried to synchronize on the same object and spin indefinitely once the lock was acquired. Trying to terminate the locked-out thread failed until the thread that acquired the lock was stopped. Apparently the monitorexit JVM instruction used to enter a synchronized block cannot be interrupted by stopping the thread, much less with the normal interrupt mechanism. We were already doing some bytecode editing for our sandbox, so I extended that to rewrite uses of synchronization-related instructions or methods to call static sandbox functions that simulate the originals using ReentrantLock, locking functions of which can be halted by thread shutdown.

Infinite try-catch loops can crash the JVM

Previously I discovered that the Java compiler sometimes emits exception table entries set to "handle" an exception by jumping back to the start of the covered region. This led to deadlock if an exception was indeed thrown in that region, usually due to bytecode rewriting inserting instructions. It was manageable to fix during bytecode rewriting with ASM by dropping exception table entries (not calling super.visitTryCatchBlock) that had the same start and handler label. Unfortunately there are other circumstances, particularly with while loops inside synchronized blocks, in which the compiler emits an entry with the handler in the middle of the covered region. I saw this crash the JVM entirely, citing an "invalid class file" but providing no additional information with -Xverify:all. Based on the access-violation error code, it seemed to be a stack overflow in the JVM itself. Avoiding this problem is more difficult because ASM doesn't provide a way to know the order of labels until all the exception table entries have been visited. I had to make the ClassReader first accept a visitor that recorded the indexes of problematic try-catch blocks, then use that information during the real transformation that dropped the offending entries.

When the Android emulator says "not enough space to create userdata partition"

Today I helped a student who was trying to set up the Android emulator but received the message "emulator: ERROR: not enough space to create userdata partition." At first they suspected the specified virtual storage device was too small and tried increasing it, but that just made the "required" number in the error message bigger. This message is actually saying that there is not enough free space on the actual hard drive to reserve for the virtual storage device. Freeing up some disk space allowed creating the emulator.

Nested Git repositories can appear as phantom submodules

Tonight I helped a student who was trying to publish a project to GitHub with Android Studio but only got one subfolder in the resulting GitHub repository. git status showed that the published subfolder was a submodule with pending changes. Strangely, there was no .gitmodules file for submodule registration. There was, however, another .git directory inside that folder - it was actually a nested repository. To fix that accident, we:

1. Deleted the subfolder's .git directory
2. Removed the submodule from the outer repository's Git index with git rm --cached theFolder
3. Committed the change
4. Added the subfolder back, now as a plain directory, with git add theFolder
5. Committed the change

Exposing a Kotlin function only to Java code

I'm involved in a Kotlin project that's intended to be used from both Kotlin and Java code. Because Java doesn't have optional/default-value parameters, we use @JvmOverloads to generate overloads with subsequences of parameters, but we also end up writing a lot of overloads manually to remove optional parameters in the middle. These pollute Kotlin autocomplete lists, so I wanted to hide from from Kotlin code.

The @SinceKotlin annotation is meant to hide a member from sufficiently old versions of Kotlin. Specifying a bogus, extremely large version number hides the member from Kotlin in general. I defined a JAVA_ONLY string constant with a large version like 999.0, so the hiding annotation looks like @SinceKotlin(JAVA_ONLY), which is nice enough. This annotation doesn't mean anything to the Java compiler, so it's perfectly happy to use the function.

FMod - v2.11.2

Today I did some final testing on the new Abiathar version comprised of my long-languishing beta and my tweaks from the past week. I noticed a discrepancy between the default side tile properties' names in freshly created projects vs. older ones using generic tile property definitions. I fixed that and published the new version as v2.11.2 for the first update of 2020. Somehow it had been almost two years since v2.11.1!

FMod - Less VeriMaps advertisement

A long time ago, I introduced a levels signing feature to Abiathar, mostly for fun and to recognize people who contributed. However, it doesn't actually seem to have been used despite being almost always mentioned in the status bar. To avoid confusing new users, I've now changed the status bar text to not mention VeriMaps unless the levels are signed.

FMod - Unknown configuration section detection

For a long time, Abiathar has allowed extensions to save additional information on a project by adding another configuration section to the project file. While this has never been used to my knowledge (except by my as-yet-unreleased Abiathar Live Studio extension), I happened to be thinking about it recently. When Abiathar sees a configuration section, it asks all loaded extensions whether they recognize it. If an extension claims it, the configuration template provided by the extension is used to load the section.

If no extension claimed a section, it was silently ignored. This could be a problem if a project was opened without the extension used to create some important information, since the section would be lost upon saving. So I added a confirmation/warning when an unclaimed section is loaded.

FMod - More color customizability

I've had some Abiathar progress sitting in a beta release for a long time now. Since I've heard no reports of problems from that beta's user, I figured I might as well get a release out including those Tile Property Modifier updates and a few tiny tweaks.

A long time ago I added brightly colored bars to active planes' bays so that you can tell at a glance which ones are active. I tried to keep the three colors distinguishable from each other without looking garish and repelling new users. They ended up different shades of blue to match the overall Abiathar color scheme. But some users might want to sacrifice an attractive color scheme for more distinguishability, so the colors of those bars are now customizable on the Colors tab of the Settings. The bold blue color of the label "Active" can now also be changed, e.g. in case someone wants the very noticeable but eye-searing red of the Abiathar 1.x series.

When Java exceptions don't include a stack trace or message

I was helping someone investigate a crash in their Android Studio project's tests today. One exception totally lacked all context: it was just the exception class name, no message, no stack trace. It turns out Java has a performance feature that skips generating the stack trace in certain cases for common exceptions. The feature can be disabled by adding -XX:-OmitStackTraceInFastThrow to the JVM command-line arguments. That brought the stack trace back, which was very helpful for finding the bug.

JavaCompiler Diagnostic objects might not include a source

Today I was working with a project that uses the JavaCompiler API for in-memory compilation of Java sources. It accesses the compiler's Diagnostics to report compilation errors. It assumed that getSource would return a source file for each diagnostic, but that is not the case and it choked on a warning about a lack of annotation processor for a certain annotation. Initially I just wrote an inert annotation processor to squelch the warning, but then I went through the project that was the direct cause and fixed it to not assume a specific source for each diagnostic.

Ubuntu's default Node JS version may not be what you want

I was trying to get a Node JS application running inside Bash on Ubuntu on Windows. I successfully installed Node with the Apt package manager, but some library failed to install, citing a missing node program. Symbolic-linking that to nodejs kind of worked but produced errors about unknown (seemingly newer) syntax. After some fiddling around I learned of NVM. Installing that and the current LTS Node version was easy and made the program work as expected.

Check reCAPTCHA after updating to phpBB 3.3

I operate two phpBB forums that use reCAPTCHA during registration for spam prevention. I just updated them to phpBB 3.3, which switches from checkbox captcha to the newer(?) "invisible reCAPTCHA." The demo in the board settings, though, showed an error message about invisible reCAPTCHA not being enabled for the site key. That makes sense since I registered the sites with the v2 checkbox verification. To make the captcha work again, I had to reregister the sites with reCAPTCHA as the v2 invisible verification and get new keys.

Avoiding infinite throw-catch loops

Previously I encountered a problem when doing bytecode instrumentation of try-catch blocks. The Java compiler sometimes emits exception table entries that register short regions as their own exception handlers. When code that can cause exceptions is inserted into the middle of these regions, an infinite loop results.

To avoid this with the ASM bytecode library, I just added a check in visitTryCatchBlock to see if the handler label was the same as the start label. If so, the function returns without passing the event along, removing the exception table entry. This can't break anything because if the handler was ever used under normal circumstances, it would trigger a thread-termination-resistant infinite loop, which I don't observe.

Java exception tables can allow infinite loops

Today I worked with an application that does some Java bytecode instrumentation to intercept exception handling. The exception table of a method specifies which regions to detect exceptions in and where to jump if an exception occurs. In one class, an exception table entry's target ended up being the start of the covered region. If an exception occurred inside the region, an infinite loop would begin. That would even prevent the thread from being terminated because the ThreadDeath exception couldn't propagate out of the function.

Restarting the audio service can fix microphone problems

Today in the middle of a call my laptop decided to stop getting audio input from its integrated microphone. Running the audio troubleshooter on the microphone didn't do anything, nor did disabling and reenabling the device. It started working again after restarting the audio service (Audiosrv). That can be done with one command from an administrative PowerShell prompt:

Restart-Service Audiosrv

Peril of reentrant defineClass: ClassCircularityError

Today I tried to make a Java application faster by eagerly loading a lot of classes into a classloader at once when the first of a set was requested. This did not work well. When the JVM loads a class in response to a defineClass call, it makes sure that there are no cycles in the inheritance hierarchy. This check is done immediately, requesting all parent classes from the classloader. If those haven't been loaded yet, my classloader would be called again (with the first defineClass call still on the call stack) and try to eagerly load a bunch of additional stuff. Apparently the JVM doesn't expect unrelated classes to be defined in this fallout from a defineClass call: if the next class defined isn't the next one up the inheritance hierarchy, a ClassCircularityError will be thrown even in the absence of actual circularity.

Implications of the Kotlin coroutines dispatcher initializing only once

I'm working on a project that tries to run several tasks in sandboxes in a Java process. It would be ideal to support Kotlin coroutines inside the sandboxes, but the default dispatcher causes a problem. The first time a coroutine is scheduled on the shared thread pool, the pool threads are started inside the current thread group. If that's triggered by trusted code, the threads belong to a trusted group but can be borrowed by untrusted coroutines. If the creation is triggered by untrusted code, the pool threads are associated with that one job which can be torn down, ruining the pool for any other jobs.

Since the pool is a static member of some internal class, I ended up needing to reload the coroutines library into each untrusted classloader. That way, every copy of the class has its own static members that are initialized inside the sandbox and apply only to one job.

app-ads.txt for Google Play apps

I recently got an email from Google AdMob urging me to add an app-ads.txt file for one of my Google Play apps. This email directed me to a PDF of a technical specification which referred to another prior specification. While I could have spent the time digging into the details, it seemed strange to me that Google would say "here's the specs, go do the thing!" since they're usually quite developer-friendly. I looked through the specs a bit and tried researching what app-ads.txt is supposed to do and did not get far in the brief time I investigated.

But reading on in the email and poking around, I eventually discovered that understanding all the rules and goals of the file was not necessary. The AdMob console's "how to set up app-ads.txt" button mentioned by the email contains an AdMob-user-specific snippet that can be the entire content of the file. This was not clear to me because step 1 was to create the file "using the spec provided by the IAB Tech Lab."

So I specified an URL for my app in its Google Play listing, created app-ads.txt in its root as directed, and pasted the snippet into it. (I also went ahead and added a CONTACT line.) The next day, the AdMob console confirmed that the file was set up properly.

Gson can create Kotlin objects with surprisingly null parameters

One of my projects uses Gson to load a bit of state from a JSON file. The state is represented by a Kotlin data class with non-null properties. I expected that instances of this class would fail to initialize if Gson didn't find a JSON property for all the properties, in which case the program would fall back to my default. Instead it created an instance with that non-null-declared property set to null, which produced a confusing stack trace. Checking the property against null works, of course, but Kotlin produces a "senseless comparison" warning.

IntelliJ "apply patch" doesn't use Git

The class I'm on staff for historically distributed updates to its assignment as patches generated by git diff. Students would apply them to their projects with the VCS | Apply Patch feature of IntelliJ (Android Studio, in our case). This appeared to work pretty well, but eventually I noticed a case where not all changes from the patch were actually applied to the project. Apparently the "apply patch" feature is not powered by Git, even in Git diffs for Git projects. Using git apply from the Terminal tab works better.

Using Weka in Android

I tried to build an Android app involving the Weka machine learning library, but adding Weka to the project makes the build fail with a ton of duplicate class errors. For some reason, Weka seems to have two very similar dependencies that both get selected for the Android packaging process. One of them can be removed with a bit of extra Gradle configuration (Kotlin shown):

implementation("nz.ac.waikato.cms.weka:weka-stable:3.8.4") {
    exclude(module = "java-cup-runtime")
}

Android Gradle 3.5.3 doesn't clean the desugared local JARs directory

One of my Gradle plugins helps swap out a handful of JAR dependencies in Android projects. I noticed that after using it for a while on an Android Gradle 3.5.3 project, building an APK would eventually start failing with a "program type already present" error.

I determined that the problem occurred when two copies of the same JAR dependency ended up in the external_file_lib_dex_archives intermediates directory. Files there are named with a combination of a number (increasing within one run) and the original filename. I observed the same JAR with different numbers, presumably from different builds. That directory is populated by DexFileDependenciesTask, which from reading the code does not seem to ever clean its output. So if the order of JARs changes or if one is removed, extra JARs will remain and clog up the dexer.

I worked around this in my plugin by registering a doFirst action on all DexFileDependenciesTasks that deletes and recreates the task's output directory.

Directories may or may not have their own entries in ZIPs

Today I found that one of my Gradle plugins choked on a JAR file that should have been perfectly normal. My plugin tried to extract the JAR by creating a file on disk for each entry, which ended up including directories. That was surely a bug, but it was strange that only some JAR files triggered it. Specifically, JARs created by a Gradle Java project jar task caused the problem, but those created by an Android createFullJarVariant task did not.

Apparently it is optional for ZIP (JAR) "entries" to be created for directories, even if there are files in those subdirectories. I examined example JARs in PowerShell with the ZipArchive class, finding that only the crash-causing JAR had directories in its Entries collection.

Creating symbolic links on network shares requires the privilege on the remote host

Someone asked on Super User how to create symbolic links on network shares. Trying to create one with mklink, even after adjusting symlink evaluation settings with fsutil, failed with an access-denied error. This is because Windows requires a privilege to create a symbolic link, both on the computer issuing the command and the one hosting the network share. That is, the user one connects to the share host as must hold the "create symbolic links" privilege on that host. The privilege can be assigned in the Local Security Policy snap-in (secpol.msc).