The phpBB 2 to phpBB 3 converter messes up encodings

When I upgraded two forums to phpBB 3 from phpBB 2, some text apparently got messed up. The problem is classic; it took UTF-8 text, interpreted it as Latin-1, and then re-encoded it as UTF-8. This appears to be a problem with the converter - there are several threads on the phpBB forums about it, which I would link if their forums were currently available. I can correct that with a bit of SQL.

Unfortunately, the second forum converted got doubly butchered. I'm not sure exactly how, but it seems some sequences were unrecoverably damaged. That is, some are fine (well, at least recoverable by double-decoding), but some end with an invalid character. I'll need to manually search-and-replace the most common miscoded characters.

Policy Plus - Show the last sources

Since it's possible that people might not remember what policy sources they were editing last time, and given that Policy Plus now remembers the last policy source (which is almost always the convenient thing to do), it's important for people to be able to see what they're currently editing. Therefore, I added a bit of code to the startup routine that sets up the "Open Policy Resources" dialog to be in a state that would produce the last policy loaders.

When the phpBB 3 upgrade fails with 500 Internal Server Error

Today I migrated two phpBB 2 forums to phpBB 3. The first went perfectly smoothly with no unexpected messages at all. The second had a couple apparently non-fatal warnings, but then in the middle of the database upgrade process it just halted with an HTTP 500 error. I could get no useful information out of the log. After hours of fiddling and refreshing and Googling, I found that some posts in the old database were tripping up the parser.

To figure out exactly which ones were problematic, I went into the new database and got the maximum post ID out of the posts table. I then queried for the posts in the old database with an ID starting a little bit before that number. Sure enough, one a little after the last successfully migrated post was insanely long with a bunch of weird markup. After replacing its content with a little note about why I had to remove it, the updater continued successfully!

Students can get AutoDesk products for free

AutoDesk makes software for architects, engineers, manufacturers, and animators. For example, they're the company behind 3ds Max, the powerful 3D modeling and animation program. I had heard of their products, but until recently I didn't know I could actually get them.

If you're a student with a .edu address, they let you have quite a bit of their software for free for three years for educational purposes. Visit their education section for more information and to install the software.

Policy Plus - Remember the last settings

If you open an alternate ADMX source, it's likely that you want to continue using it in future launches of Policy Plus. Before, Policy Plus just opened the default PolicyDefinitions folder and the normal local GPOs at startup. Today I added a little configuration manager to read and write settings from the Registry. At startup, Policy Plus loads the last ADMX source and policy sources. The ADMX source setting is updated whenever a full new source is opened; the policy source settings are updated when the policies are saved.

Batch-like wildcards in PowerShell

Several commands and utilities used in batch scripting take filename filters that use * and ? as wildcards for multiple characters and exactly one character, respectively. In PowerShell, that kind of matching can be done with the -like operator, but it has a couple extra features (e.g. # for a numeric character) that make it not match batch behavior exactly. Therefore, to use it on arbitrary filters and get it to work like classic batch, you should first escape some characters in the filter:

$input -like ($filter.Replace('[', '[[]').Replace('#', '[#]'))

Based on my Super User answer.

AbiathRPC - Little bugs

While combing through AbiathRPC, I found a couple small bugs which are now fixed.

First, authentication wasn't actually being checked in the WCF method that updates a batch of tiles. Fixed by adding the appropriate login-checking call.

The Abiathar client with the AbiathRPC state wrapper would crash when updating tiles when not connected to a server. This was caused by the replacement state wrapper trying to submit updates from a list that didn't exist to a server that didn't exist. Fixed by checking that there's actually an RPC connection and stopping with the vanilla behavior if there's not.

(Also, merry Christmas from Abiathar - from now until sometime tomorrow night, the Abiathar splash screen is Christmas-themed.)

The surprising OWNER RIGHTS principal

Windows has a security principal called OWNER RIGHTS. Like several other all-caps principals, it doesn't apply to anyone in particular. It defines the access that the owner of the object is given even if not otherwise allowed. The surprising part is that the mere presence of an access control entry for this principal blows away the default rights granted to the owner (the ability to read and write the access control list). If you add an entry for this principal with no permissions, being the owner is no longer special other than in that the file size counts against the owner's disk quota.

What does the Distributed Link Tracking Client service do?

Windows is surprisingly good at keeping track of files that have shortcuts pointing to them. If you move a shortcut target to somewhere on the same volume, using the shortcut will find the target file by its NTFS object ID. You wouldn't even know anything had happened in the background.

If you move a file across volumes, though, it's object identifier changes. (Check this with the fsutil utility.) Yet somehow Windows still manages to find the file if it's pointed to by a shortcut! This is the work of the Distributed Link Tracking Client. You can test this by stopping the DLTC, moving a target across drives, and trying to use the shortcut - it'll be broken.

On a standalone machine, the DLTC keeps an eye on file movements to repair shortcuts. On a domain, it reports to the domain controller, which stores the movement information in the Active Directory to help all client computers find files on the network.

Based on my Super User answer.

Windows does not keep track of last-access time by default

One might wonder why the last-access time never seems to update when files are opened. After all, a file has to be accessed to open its contents.

The reason is that Windows, as of Vista, no longer updates the last-access time by default because that would involve a write every time a file is opened, even if it was just supposed to be a read. If you really want that updating back, you can run this command:

fsutil behavior set DisableLastAccess 0

Based on my Super User answer.

When does text direction get affected by a previous RTL character?

Suppose you have this text in a text field:

abc123'ߡ'

That triangle character is U+07E1 NKO LETTER MA. It's marked as a right-to-left character, since it comes from the N'Ko alphabet, which is written from right to left.

Now for some experiments. Try typing a number after the triangle but before the closing quote. It will appear to be inserted before the triangle! Hit Backspace, then move to the very end of the string. Type a normal dash. It's not teleported. That character's bidi class is "European separator", which is listed as weak. But now type a number after that, and the string gets shuffled into this:

abc123'ߡ'-4

The order of the last quote, hyphen, and number appears to be flipped. Interestingly, digits have the "European number" bidi class, which is also weak. So why did the number cause the flip?

Unicode RTL rendering rules mandate that number characters (that don't already have their direction forced upon them) take on the direction of the last character behind them that had a strong preference. Neither the quote nor the hyphen did, so the new number becomes RTL, and it takes the in-between characters along for the ride.

Based on my Super User answer.

Holding an executable inside a batch file

Suppose you want to include an EXE inside a batch file and have the batch script extract the program from itself and run it. Just pasting the EXE into the file will almost certainly butcher it, because programs are binary files and batch scripts are text files. Therefore, you'll need to encode the binary data.

Start with this batch file:

@echo off
powershell -command "[IO.File]::WriteAllBytes('extracted.exe', [Convert]::FromBase64String((gc '%0')[-1]))"
extracted
del extracted.exe
exit

REM Base64-encoded program will be inserted below

The extra blank line at the end is important.

Notice that the PowerShell script embedded in the second line takes a Base64 string from the end of the batch file. To put it there, use this PowerShell command:

[Convert]::ToBase64String([IO.File]::ReadAllBytes('C:\fullPathTo\file.exe')) | Out-File 'batchFile.bat' -Append -Encoding ASCII

Running the batch file will decode the executable, save it to disk, run it, and delete it once it finishes.

Based on my Super User answer.

AbiathRPC - Anonymous is not a person

Today I realized that the usefulness of the VeriMapsApproved setting (which allows any VeriMaps-authenticated user to connect) for the AbiathRPC server is undermined by the public availability of the Anonymous key. Server operators might enable that option think it will allow in known members of the community - since only I can issue new certificates - but it actually allows anyone who bothers to download the Anonymous key. Therefore, I made that account an exception to the approval granted by that server option.

If server operators want to allow the Anonymous certificate to be used for authentication, they can still add it as a normal account without a password.

AbiathRPC - Save levels locally

If you're working on a leveling project with some people, it's great to have a centralized copy of the levels (on the AbiathRPC server), but you also need to be able to get the newest version so you can test your changes in a game. Today, I added an option to the RPC menu that saves the current state of the levels to local files. Its implementation is very straightforward; the only special thing it does is check for the presence of a GAMEMAPS file in the current folder and fill the suggested file names with that file's extension.

AbiathRPC - Administration GUI

Yesterday I wrote the AbiathRPC server functions for user management. Today I put together the client UI that actually uses those.

In the above screenshot, Anonymous can only authenticate with VeriMaps; it has no password set. It is also a server administrator. The Test account can authenticate with a password, but it is not an admin.

The buttons are fairly intuitive: New Account adds a user, Set Password adds or sets a password for a selected user, Disable Password removes a user's password (allowing VeriMaps only), Promote/Demote toggles administrative privileges, and Delete Account removes the account from the server's configuration.

Changes to user configuration are saved to disk when the server shuts down.

AbiathRPC - Administration APIs

There should definitely be a way for AbiathRPC server operators to manage their servers without having to actually remote in to the host machine, stop the server program, fiddle a textual config file, and restart the program. I think it makes sense for some management tools to be included in the Abiathar extension itself. So today I added a few WCF endpoints on the server: to get the allowed users, to delete a user, to create a user, to reset a user's password, and to set a user's access level. Those are all implemented, but I haven't yet put together a GUI on the client to use them.

I also adjusted the part of the server that decides a newly connected user's authentication plan. Now, even on an open-access server, users with a definite password will actually be required to enter their password if they use their registered username. Users without accounts are still free to pick whatever name they want and use it with no special access or authentication.

You can play "Myst 3: Exile" with ResidualVM

It's very difficult to get Myst III: Exile running on modern Windows if you only use the original media. Though ScummVM can run the original Myst and its sequel Riven, it doesn't work on Myst 3. For that, you'll need its sister project, ResidualVM.

The folder structures you'll need to copy from installation media vary depending on which ResidualVM build you use. If you use the latest stable release, follow the instructions on the web site. If you use the newest development build, use the ones in the GitHub repository.

AbiathRPC - Clean up old WCF instances

I just noticed that the AbiathRPC server never properly disposed of communication objects for clients that had abruptly disconnected; they just hung around in the list of server object instances. Since they're in that list, the server always tried to notify them of new events. Today I added some code to the routine that runs every ten minutes to save the levels if necessary. It now goes through all the current connections and removes the ones that are no longer working.

I also found that non-authenticated clients received event notifications because they were in the list, so I made the "notify all the other clients" routine only send the data to authenticated users.

Volunteering at a rescheduled FTC meet

The last FTC robotics meet in the area was cut short due to difficulties with the electronics in the beacons. Its second slate of matches was hastily rescheduled to today. For an unexpected meet, it was arranged very nicely, with a practice field and a competition field that worked well. Though I was assigned to do field inspections again, I could not arrive in time for those because I had to take some exams. I did manage to handle field reset for all but two of the matches. There was only one beacon-related problem, and the referees had a spare one on hand with which to swap it out.

Testing whether a given user is an administrator with PowerShell

Given a user object $u, you can use this PowerShell line to check whether that user is an administrator, i.e. a member of the Administrators group:

(Get-LocalGroupMember 'Administrators' | ? {$_.SID -eq $u.SID}).Count -ne 0

User objects come from the Get-LocalUser or Get-LocalGroupMember cmdlets. The former lets you get a user by name, which is more useful here.

Based on my Super User answer.

How Windows Firewall knows whether it has prompted the user to allow a program

When a program tries to start listening in a manner that doesn't match an existing firewall rule, Windows Firewall pops up a dialog asking whether to allow the program through the firewall. No matter whether you choose "Cancel" or the choice that lets it through, a new rule is created, to block or allow traffic from that EXE, respectively. The interesting part is that rules created in this way are marked specially in the Registry. All the rules are kept here:

HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules

Each program-related prompt creates two rules, one for TCP and one for UDP. The Registry value names start with TCP Query User and UDP Query User, respectively. To bring the prompt back, remove those rules either by deleting them in the Windows Firewall MMC snap-in, or by deleting the corresponding Registry values then restarting the Windows Firewall service.

Based on my Super User answer.

AbiathRPC - Password hashing

AbiathRPC server operators can allow users to authenticate by password or VeriMaps certificate. The VeriMaps certificates themselves are never seen by the server, but servers did store the plaintext passwords. Today I made the password authentication system using hashing, so people's passwords aren't on disk. This has the added advantage of stopping bizarre characters in new passwords from breaking the configuration file.

Server operators can still easily give people initial passwords; there's a PasswordIsHashed field that determines whether the user record's password field should be considered a hash instead of a plaintext. A plaintext will be automatically hashed at startup.

AbiathRPC - Reconnection crash fixed

Yesterday I mentioned an AbiathRPC bug that caused the opening of a remote level set to fail after closing one that was opened successfully. I found that the crash occurred when setting up the watchdog render planes; an original GalaxyLevel was incorrectly cast to a TaggedGalaxyLevel in the method that looks up a level ID for a given level wrapper.

The problem was that the bookkeeping on level substitutions wasn't nullified at disconnection. That made the ID lookup function think the connection was complete when it's wasn't. The old level objects were also kept around, which is less than ideal. I added a single line to the connection-closing method to fix the problem.

AbiathRPC - Change password

Today I added a way for AbiathRPC client users to change their passwords on the server. There is a new "RPC" menu that is only visible when connected to an AbiathRPC server; currently, its only item is Change Password. The resulting form has the user enter the current and desired new password. The server verifies the existing password and updates the user record with the new password. The password changes along with other configuration changes are committed to disk when the server exits.

I noticed that the AbiathRPC client crashes when reopening a remote level set after having closed one, so something isn't quite right with the disconnection routine.

Reading Word document properties in PowerShell

Word documents include some extra properties like "total time spent editing." These properties aren't visible in Explorer because they're just data in the file. You could use COM to interop with Word and read them, but it's not super hard to grab the data directly from the file.

Modern Word documents are just ZIP files, mostly containing XML files, both of which PowerShell can handle. This little script outputs the total editing time of a given Word document:

$zip = [System.IO.Compression.ZipFile]::Open($filename, 'Read')
$propsentry = $zip.GetEntry('docProps/app.xml')
If ($propsentry -ne $null) {
    $stream = $propsentry.Open()
    $reader = New-Object System.IO.StreamReader $stream
    $content = $reader.ReadToEnd()
    $xmldoc = [xml]$content
    $xmldoc.Properties.TotalTime
}
$zip.Dispose()

The time is measured in minutes.

Based on my Super User answer.

Volunteering at another FTC meet

This Saturday, I volunteered at another qualifying FIRST Tech Challenge robotics meet. Like before, I was a Field Inspector, but I also got to do Field Reset this time.

I started out with field inspections. There were 11 teams, most of which got through without any issues. Since this wasn't a league championship, outdated software was acceptable, so I just warned teams that needed to update that it wouldn't fly at the big meet. I was apparently the only Field Inspector, so when multiple teams came over at once, the Control System Adviser helped inspect some. There were a couple instances of robots not driving due to hardware profile mismatches, but those got sorted out and they were all ready by the time matches started.

Field Reset is much easier with this year's game than last year's. Here, I just had to move some large exercise balls back into the middle and put the small wiffle balls back on the sides. (As opposed to last year, where I had to pick up dozens of cubes and wiffle balls and then hurl them onto the field at the start of the next match.)

Unfortunately, the electronic beacons started failing after being buffeted by the robots throughout the matches. We tried to simulate them with volunteers holding alliance flags to indicate beacon color, but that was always a judgment call as to whether the robot actually hit the button. After the first of two sets of 17 matches, the teams' coaches convened and decided to reschedule the second set. That was for the best, since we were already more than an hour behind from the beacon trouble. Hopefully a venue and date can be found for this meet's thrilling conclusion.

Using Chrome's --disable-extensions-except

As of version 55, Chrome supports the --disable-extensions-except flag, which as you might expect, disables all extensions except the one specified. The trick is that it has to be specified as a path to a folder containing an unpacked extension. For an extension you're building, that's easy, but for ones you got from the store, you have to hunt down the folder where they're kept. Chrome unpacks installed extensions here:

%LOCALAPPDATA%\Google\Chrome\User Data\Default\Extensions

Inside are a bunch of folders each named with an extension's ID. Inside those are folders for versions of the extensions, and those are the folders that have manifest.json. Pass that folder to Chrome like so:

chrome --disable-extensions-except="C:\path\to\versionfolder"

Based on my Super User answer.

AbiathRPC - Make sure the connection gets closed

When a remote level set is closed, it's important that AbiathRPC clean up and realize that it's no longer supposed to be talking to that server. I already had an event handler on the Close menu item, but a level set can also be closed in the process of a new one opening, which bypasses that event handler. Fortunately, the level label is updated during the close process, so I added a check for the current level set being gone to the existing handler. If the main level set is missing, it initiates AbiathRPC disconnection, which cleans everything up.

Version number strings aren't automatically sorted as versions should be

Let's say you have some strings describing version numbers, like 1.1, 1.2, all the way up to 1.11. If you try to do a newer-version check using only string comparison, it will appear that 1.11 is older than 1.2. Most sorts sort those texts that way because "1" indeed sorts before "2", so it stops checking. (It doesn't think about numeric value at all.)

Instead, you should do some number comparisons. For example, using the SMBIOSBIOSVersion WMI property (a string) will be more difficult than using the SMBIOSMajorVersion and SMBIOSMinorVersion properties, which are integers.

Based on my Super User answer.

Unbanned from Dropbox public linking

A while back, my Dropbox account's public links stopped working and stayed that way for a good long while. Eventually, I got in touch with someone via e-mail and explained the situation as I had done in the original ticket. The ban was indeed automatic; as I surmised, one of the executables I was hosting (unlzexe.exe) was considered malware by their system. Again, it's not malicious, but it does modify other EXEs, so it could be used for bad purposes. Abiathar uses it to decompress the Keen games' executables.

On September 6, Dropbox manually lifted the block. I am no longer hosting unlzexe.exe there, but Dropbox has continued to work well for my other resources.