Windows 10 Developer Preview Upcoming

Microsoft has apparently scratched off the name "Windows 9" from their marketing materials and is instead calling the next version of the Windows operating system "Windows 10." I was under the impression that the Developer Preview would be released today, but it seems like that will actually happen tomorrow; it was just the developer conference event that happened today. When the release actually happens, you'll be able to download it at the Windows Preview site.

I have purposefully avoided demonstrations of the OS because I want to do a "first impressions" video on it. Allegedly, it removes the Charms bar and the Start screen, instead bringing back the Start menu, now metro-ized. Also in the feature list was the ability to create virtual desktops (kind of like that Windows XP PowerToy, I guess), which is pretty awesome.

Until the release, you can read the Windows blog article about it. Look forward to my video on Windows 10!

AdminPrivative: Design Specification for Partially Trusted Software Installation Delegation on a Windows Domain

In some Windows enterprise environments, it may be useful to grant certain people the ability to help users install stuff on their computers (probably laptops) via over-the-shoulder elevation without making them full domain admins. I outlined a strategy for doing something like this yourself, but now I have a pretty interesting idea that I might start developing.

Let's suppose you have issued laptops to a bunch of people on the assumption that each one is owned by a specific user and that the domain account for its owner will essentially always be the logged-on user. Suppose also that your users run as non-administrators and aren't allowed to do administrative things to their laptops, because it's usually a good idea to set things up that way. But your users occasionally do need to install programs, and you don't have enough time to OTS-elevate them, so you appoint a few people to do that. Since these people are supposed to just be administrators for the purpose of installing software onto these machines, they shouldn't be allowed to the domain or anything scary like that.

One possible option is to use Group Policy (or some system preparation script at set-up time) to set that group of installation helpers to be local admin on each machine. This has the major disadvantage of, well, letting them be local admin on every machine. Especially in a domain where remote server administration services are operating, having local admin is a very powerful thing. A group of people who you may or may not trust has full access to everybody else's stuff.

I thought of something better.

Forget about giving the install helpers' domain accounts any privileges. In this system which I call AdminPrivative (a pun on "alpha privative" and also a fairly accurate description of what it does), local temporary accounts are created whenever a normal user requests a software installation. The activities of those accounts are monitored, analyzed, logged, and reported to the real domain administrator's archives. A sketch of the design specification follows.

An AdminPrivative system consists of the following:

• Windows domain infrastructure
• A single authentication server/service, running as domain admin
• Local service on each computer eligible for AdminPrivative elevation

The AdminPrivative central server is a program running as a full domain admin. It should be under the control of the real domain administrator, not the install helpers. (All the install helpers have no special privileges at the domain level.) Its job is to listen for network requests (maybe serve a web form to gather the data) from the install helpers and authenticate them and get the control request. It then uses RPC (probably through PsExec or Sc.exe) to spawn the AdminPrivative service on the target computer.

The connection to the controller could be made on the target computer, but that would require entering credentials in a possibly compromised workstation. This system is designed in such a way that authentication to the central server should be done on a computer under the control of an install helper.

The local service prompts the machine owner to allow or deny the account creation. If the owner OK's it, the service prompts the owner for their domain password, checks it with Directory Services Account Management, and creates a local admin account. This account is added to the "Deny access to this computer from the network" list (using NTRights.exe) to prevent remote tampering. If the owner denies it (i.e. it was unsolicited), the local service tells the controller, which screams loudly into the e-mail box of the domain admin.

At this point, the local service tells the server that the account creation was successful, hands over the password to the install helper, makes a note of the time, and starts using WMI to monitor process creation. The server sends the temporary account's password to the install helper, who walks over to the laptop to perform an OTS elevation. The installer runs to completion, and after a few minutes, the local service destroys the temporary account and any processes laying around that it owns. It sends the log of processes created using that admin account to the controller, which generates a report of the entire transaction and sends it to the domain admin. If any issues are encountered (e.g. the local service stops responding), the controller complains to the domain admin.

The purpose of this entire system is not to try to stop the install helpers from doing bad things to the computers after they are granted control by the owner, but to log everything so that the domain admin can know what happened and who did what when.

Developing this would be a complex undertaking. I'm sure it will be an adventure!

How to Stop Windows Group Policy Refreshing

Now that you are able to override the domain Group Policy locally, you might have found that it occasionally gets reset, especially after a reboot. Windows can apparently detect when the policy cache is out of sync with the domain and will attempt to resync it, destroying local customizations.

Fortunately, this can be disabled. We can do so by revoking the system's privilege to modify those registry keys. Fire up RegEdit as admin and go to:

????\Software\Microsoft\Windows\CurrentVersion\Policies

The "????" should be HKEY_LOCAL_MACHINE if the annoying Group Policy entries are Computer Configuration or HKEY_CURRENT_USER if the offending policies are in User Configuration.

Right-click the key in the left pane and choose Permissions. Notice that SYSTEM is allowed Full Control. Uncheck that box in the bottom pane, but make sure SYSTEM has Read access. If you deny read access to the SYSTEM (or remove its DACE), then it won't be able to even check whether you can do things and stuff will blow up.

Then, open up the Owner tab under Advanced and set the owner to the local Administrators group. This ensures that [1] any Group Policy-deployed script can't undo our changes and [2] you'll be able to more easily edit the settings if you choose to change the permissions again.

OK out of all the dialogs and close RegEdit. Congratulations, you are now invincible to Group Policy.

Why You Can't Launch Task Manager on the Secure Desktop

Why can't you launch Task Manager when on the Windows secure desktop (logon/lock screen)?

Imagine if that were possible. Since there is usually no user logged on when you're at the logon screen, any programs running are running as NT AUTHORITY\SYSTEM, the full-power local system account which is more powerful than any administrator. If a process spawns another process, the new process inherits the owner and elevatedness of the old. Basically, you've given full control of the computer to whoever walked up to the lock screen because Task Manager has that "New Task" menu option.

Why not restrict the process-creating abilities of Task Manager when it's on the logon screen? Because it would still have the authority to kill any task, and that's kind of the entire point of having Task Manager. (Similar reasoning can be applied to the proposition of having it run as NT AUTHORITY\ANONYMOUS).

Most importantly, if you're on the logon screen, you haven't logged in to run any programs. If you're a computer administrator, you can log in even when other users have locked the machine and create a super awesome elevated Task Manager from there.

Location of the OEM Windows 7 Secure Desktop Background

If you have a computer (usually a laptop) from a particularly flamboyant manufacturer, there very well may be an OEM background for the secure desktop (the logon and lock screen). If you want to change it, you'll need to poke around in the Windows system directories.

The location of the secure desktop background is:

C:\Windows\System32\oobe\info\backgroundDefault.jpg

You can drop in any JPG that is less than 256KB for backgroundDefault.jpg and it will be used. If you don't want to bother with that and you just want the normal Windows background, go to this registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\LogonUI\Background

Change the OEMBackground value to 0 and Windows will supply its own.

Introducing ActiveNav

Suppose you manage a domain (or want to poke around on one) and you want to browse the Active Directory. This is easy to do on a server or on a computer that has Remote Server Administration Tools installed, but impossible if you don't. RSAT also requires local admin to install, so that makes it a little bit more of a challenge to get running.

I have something more convenient: ActiveNav. ActiveNav is a console application for reading and navigating around Active Directory. It operates at the same level as the low-level ADSI Editor, but does some polishing to remove all the gross LDAP prefixes and stuff. It's designed to be like DOS, with LDAP servers as drives and entries as directories/files.

Another aspect of ActiveNav's design is to be scriptable and automatable. I already have some batch file-ish processing, but no branching or logic instructions. Also in development is an extension API for .NET programs to control the application. All the actual AD connection and browsing business is done and extensively tested. I want to do a little more scalability testing and development of the automation side before I release anything to the public, though.

Some things it can definitely do:

• Connect to an AD-LDAP server with current or specified credentials
• List the sub-entries of the current entry, with optional masking
• Navigate inside OUs or container CN entries with relative or absolute paths
• Display all properties or a specific property of the selected entry
• Interpret the userAccountControl and sAMAccountType properties
• Display group memberships and quickly navigate to members
• Extract real data from byte array and COM object date properties
• Find entries in the tree based on any property with various comparisons
• Sort entries based on integer, string, date, or COM object date properties

Here are some things I have found it useful for:

• Check whether a user has been locked out
• Notify users after logon as to when their password was last misentered
• Find users whose passwords never expire
• Locate the OU where a user resides

Further developments and eventually a download coming soon!

Robotics - Nothing is Happening

It's been at least a week since anything happened at robotics that was worth reporting. Progress is slow this year thanks to the particularly difficult engineering challenge produced by the game mechanic of having to get balls from the floor into more-than-30cm-high tubes.

I think some people are messing around with the design and producing prototypes, but I haven't seen anything that requires motors to be running. We have done an excellent job of disassembling last year's robots for parts - the components are all over both rooms' floors.

Meetings for NHS have been taking a lot of my time recently, so I will be able to do even less at robotics for a week or so. Hopefully, when I return in full force, there will actually be something for me to program.

Trying Windows Server 2012 R2

Since I'll eventually be setting up a Windows domain for the office, I figured I should probably learn how to set up and operate Windows Server. So, I downloaded a trial copy of Windows Server 2012 R2 and prepared to dual-boot it on an extra machine that is rarely used.

It actually took me a while to find the place to download the trial because the most commonly linked-to one doesn't actually work. No matter where I say to place the downloaded files, the weird downloader plug-in thing throws an error and says the download is done. (The one I linked above does it better. I chose the ISO version.) Unfortunately, that link also downloads a whole bunch of weird stuff before the actual ISO. I think those other programs are installers for advanced server management stuff; whatever they are, they seem to want to be run on the server OS. Once the download finished, I extracted the ISO onto a new partition with WinRAR and began the install.

Installing the server OS was a breeze. I especially like that, unlike the normal modern Windows OS, it actually tells you what it's doing instead of something cute like "Getting things ready". When it comes up, it launches Server Manager, a really awesome dashboard for handling server operations. I changed the computer name in Step 1, ignored everything else (it seemed to not have any data, probably because I hadn't set up anything), and used Step 2 to add some roles, specifically Active Directory Domain Services.

I was moderately confused when it asked me to create a forest, but it seems that a forest is just a collection of domains; the DNS-style name you enter here will also be used for the domain. It just has to look like a real fully-qualified domain name; you don't have to hook it up to a real web site. (Later, I found an article explaining forests and domains.) The NetBIOS domain name, which is the one you type before the backslash in CredUI to specify your logon domain, is different.

At the end of that wizard, I had to promote the computer to domain controller, which also went smoothly. I did some messing around in Group Policy and Active Directory Users and Computers, creating a VM on my other computer to test out this domain stuff. So far, I'm fairly comfortable with Windows domains - the set-up was successful.

Possibly-required FTC/FCC disclaimer: I'm not affiliated with Microsoft, nor did they pay me. They probably don't even know about me. I'm just a happy customer!

Abiathar Confidential: The Testing Cycle

If you've seen my video tutorials series on Abiathar, you've noticed that I almost always run into problems with the program while recording it. That's because I typically use halfway in-development versions while recording, which is good because it helps me find bugs but somewhat embarrassing because, well, people see crazy bugs.

Full released versions of Abiathar get tested a lot more. I have a battery of tests that are run against it before deploying a new version:

• All the operations in the first video
• Opening dependency files from v1.4
• Creating new dependency files
• From defaults
• From modified files
• With the strange options in the NPW
• Running the resulting files in-game

I'm particularly fond of the test that involves Keen Dreams maps with tileinfo dumped from memory and embedded into the MAPHEAD on saving. (It exercises the most code paths.) Additionally, I send beta versions to testers, who use it like normal people do. They usually find any problems quickly.

So, between my IO-focused tests and the beta testers' UI-focused tests, we get it covered pretty well.

The For-If Anti-Pattern

This isn't anything new in terms of coding advice (you can find stuff on the Internet about it everywhere), but I just caught myself using the for-if anti-pattern. The term anti-pattern refers to a coding style (a pattern) that is suboptimal or flawed in some way. For-if is an inefficiency that fails to leverage the given lookup or access mechanism and instead enumerates every single entry in a collection, checks to see whether it has the right ID, and then does something with it. In obvious terms:

For n = 0 to 300
 If n = CInt(userChoice) Then
  Console.WriteLine(messageTable(userChoice))
 End If
Next

It's obvious here; the above code could be:

Console.WriteLine(messageTable(userChoice))

And the message table is accessed directly. I noticed that I had been doing this with the Properties collection of DirectoryEntry:

For Each p as PropertyValueCollection In GetCurDir.Properties
 If p.PropertyName = Arguments(1) Then
  val = p(0)
 End If
Next

It hit me that I could have simply written:

val = GetCurDir.Properties(Arguments(1)).Value(0)

Moral of the story: check to see whether your collection supports direct accessing. You should definitely not be looping through a Dictionary to find a pair with a certain key and extract the value from the KeyValuePair - get the value directly by passing the key to its default property!

How to Actually Get the Active Directory Date Information out of __ComObject in .NET

If you're familiar with the .NET LDAP library (System.DirectoryServices), you may have noticed that some entries in the Properties collection for a given DirectoryEntry are instances of System.__ComObject. This is really inconvenient because there's no way to tell what type of thing such an object really is. Even when you know what it should be, it's kind of a pain to extract the data.

For instance, the pwdLastSet and lastLogon properties of user objects show up as __ComObject despite being date-time things. Specifically, they (and most AD objects that you expect to be date-times) are IADsLargeInteger. To convert them into normal DateTime objects, you need a reference to a COM interop DLL in your project, specifically the "Active DS Type Library".

Let's say you have a __ComObject that is supposed to be a date-time, I'll call it p. Use the following code to get it as a nice .NET DateTime:

                Dim li As ActiveDs.IADsLargeInteger = p.Value
                Return DateTime.FromFileTimeUtc((CLng(li.HighPart) << 32) + li.LowPart)

Robotics - Disassemble Everything

While the brilliant engineering people thought up designs in the other room, I continued typing away on my laptop for various projects. They did actually come back to the main team room, so I got to hear and see some of what's going on.

Last year's robots are being disassembled, and quite thoroughly. There is now a huge spread of small parts and screws all over the floor in the main room. We should probably figure out some sort of inventory or sorting system soon, but currently we're more interested in getting all the pieces apart so they can be put back together in new ways.

The engineers have started making prototypes. This is made slightly difficult by our tools being not very organized, but I did manage to find the Allen wrench used for normal Tetrix parts. (It's 7/64", for the record.)

While they did that, I refreshed the firmware on the NXT to clean all of last year's test programs off of it. I also created a flash drive with the Samantha configuration settings, but I have not yet found the power adapter for it.

Zhongkey: An On-Screen Chinese PinYin Keyboard

As you might know, I'm taking an online Mandarin Chinese course, so I frequently have to type PinYin (Latin letters with various accent marks on top, plus the U with diaeresis). The on-screen keyboard provided by the web app has some issues. Specifically, it always places the inserted character at the end of the text field plus a line break (?!), it requires me to use the mouse to focus the text field again after inserting a special character, it can't be moved outside the web app workspace, and it's missing some of the more unusual combinations (e.g. capitals with accents or U with diaeresis and accent). After a while, I got tired of it.

So I made a better one.

Written quickly in .NET, Zhongkey is a simple PinYin keyboard that also has keys for Spanish (I had extra space in the key grid). It makes sure focus stays on your text field instead of taking it when you click a key in the grid. It also always puts the character where your insertion point was.

Download Zhongkey

Press the Capitals button to change the characters to, well, the capitalized version. (It becomes Lowercase as well.) If you want to remove its stay-on-topness without closing it, press Demote. (That becomes Promote, which restores its topmostness.) The Focus button just makes it steal focus so you can drag it around on old systems. (If you turn down the Windows graphics settings, dragging non-focused windows looks weird.) Obviously, press the lettered buttons to insert that character.

Zhongkey requires .NET 4.5 and is a portable (no install necessary, no elevation necessary) app.

Enjoy!

Whence the Elevated Run Dialog

You might have noticed that, occasionally, the Run dialog will display a little Windows Security shield and message indicating that anything you run will be considered to already have passed through UAC.

Raymond Chen blogged about this quite a while ago, but I think it's a fascinating little tidbit in the huge complex machine that is Windows; it deserves a tiny bit more recognition. (And, hey, more trivia for us geeks!)

Basically, that line appears if the Explorer instance that controls the taskbar and desktop is considered elevated. I'm guessing it will also appear all the time if you disable the UAC boundary, which is a very bad idea from a security standpoint.

How does Explorer get elevated? The only non-suspicious way this could happen is that Explorer died (bad shell extension, probably), you launched Task Manager, elevated it (with the "Show all processes" button on Windows 7 or by checking "Create this task with administrative privileges" on Windows 8), and recreated Explorer.

Since Explorer is responsible for the Run dialog, any processes created with it will inherit its elevatedness. (By the way, it is very difficult for me to resist the urge to print out a picture of the Windows Security shield and post it on every elevator I see.) If you don't want the new processes to be elevated, kill the elevated Explorer with Task Manager and make sure not to press any UI elements with the Windows Security shield.

How to Securely Install Stuff for Your Domain Users

Suppose you own a domain and have issued laptops to your users, who are not local administrators because that's your policy. Some users want to have more stuff on their computer, but they can't install it because they're not administrators. So, they ask you to pass the UAC dialog for them so they can install it. Being a helpful domain admin, you check the installer program, enter your domain credentials, let them have their software.

This is a bad idea. Besides the possibility that the user might be waiting for you to authenticate on his machine so he can steal your access token, you're almost certainly leaving DCC hashes behind. Remember, physical access is complete access.

So, how can you install software on client computers without compromising your domain account? (Besides not giving computers to dishonest people. Legitimate users might have been hacked already, possibly infected with a backdoor or other badware to steal your credentials.) I will demonstrate using the principle of least access to produce a local admin account on that computer.

You'll need the Sysinternals Suite, specifically PsExec, but it's all really cool stuff. Use your domain admin account to run this command on any other computer (should be all on one line, replace the placeholder credentials with your real ones):

psexec \\client-computer -u DOMAIN\admin -p Se(ure-P4$$w0rd net user Administrator adminpass /ACTIVE:YES

This uses your domain account to authenticate to the client computer (using Kerberos, so the resulting hash and session are useless) and spawns a temporary service that activates the normal Administrator account with the password set to "adminpass" or whatever you want.

If the psexec program fails with an error that mentions the ADMIN$ share being disabled (it should be enabled on a domain), your user has already become local admin and is doing something shady.

Now, go over to your user's computer and enter the local credentials in the UAC box. Once you're done, use the following command to disable that account:

psexec \\client-computer -u DOMAIN\admin -p Se(ure-P4$$w0rd net user Administrator adminpass /ACTIVE:NO

Using this method, you avoid spawning any processes with your access token on a possibly-compromised machine; the service processes from PsExec are created as SYSTEM.

If your domain has Remote Administration running on all the clients (it should), you can always use MMC to check whether your users are doing anything suspicious.

Quick Note on Ethics

Hey there everyone,

You have probably seen my recent series that gives the details on gaining access to things on Windows domains that most users probably can't access. There are a lot of dangerous procedures written on the internet, dangerous either because the person writing them doesn't know what he's doing or because they are too vague and leave a lot of room to mess something up majorly. So, I have included the "hack" tag to attract search results away from such procedures.

Those articles are not intended to help you do illegal or unethical things. I'd like them to be used only for testing (e.g. you're a sysadmin and you want to see if your setup is vulnerable) or in situations where you should legitimately have access (e.g. you forgot your admin password). I obviously can't stop you from doing bad things with what I write, but I write it because other people will and they might not do a good job. If you want to run my procedures against a production environment, [1] please don't and [2] if you're really going to, check with the admin first.

If you're learning about security, I write for you! Try my instructions out in a test or virtual environment to see what you can get. Audit your systems to make sure they're secured well! But please, don't do anything to get yourself into trouble.

And if you do bad things and get in trouble, it's not my fault. Even if you followed my instructions to the letter, I am absolutely not responsible for anything you do as a result of reading my blog. As they say in the alcohol commercials, enjoy responsibly.

Sincerely,
Fleex

Impersonating Domain Accounts with Incognito (and avoiding 0xC0000142)

This series is getting more and more sketchy. Please don't do anything illegal or unethical with things you learn from me. I am, of course, not responsible if you mess something up or get in trouble as a result of following my instructions.

So, now that you have local admin access and can check out the Active Directory (and override Group Policy if you want to), you might have set your sights on becoming a full-power domain administrator.

To accomplish this, we will use a tool called Incognito to seize control of an in-process access token belonging to a domain admin (or anybody else). Download Incognito v2 from the bottom of the official page. Your antivirus will probably try to stop you from extracting the ZIP; disable it temporarily and then add the resulting directory to its exception list. I recommend placing this directory on your PATH environment variable because we'll be jumping all around the drive while spawning new CMD windows.

Using Incognito is simple: you type incognito list_tokens -u to see all the available tokens, then incognito execute followed by the account you want to take and then a program to run (I recommend cmd).

This works great for seizing access tokens of local users, but since Windows XP it doesn't work for domain accounts. Specifically, Incognito will claim to succeed, but CSRSS will throw 0xC0000142 (init error). This is because only NetworkService can duplicate domain tokens, and NetworkService is itself protected similarly. (Note: not technically correct. This is true enough, and the real explanation is lengthy and unnecessary.)

To get around 0xC0000142, you must be able to act as part of the operating system. Fortunately, since you're local admin, you can do that kind of thing. Run secpol.msc and find "Act as part of the operating system" in User Rights Assignment under Local Policies. Add yourself to this list.

Now, you're ready to take all the access tokens. For a token to be in the system, the target account must be signed in or have used UAC to launch an application that is running. (Convince a domain admin to install something for you?) Let's say the target account is called NAIVE\unsuspecting. I would run incognito execute NAIVE\unsuspecting cmd. This produces a command prompt as the user. There, you can reset the password of a domain admin (preferably one that isn't used anymore) with something like net user oldadmin somepassword /domain. Alternatively, launch MMC, connect to the domain controller, and add yourself (or a less obviously named new account) to the Administrators group.

And there you have it. Very easy to do; the hard part is getting a domain admin token on the system.

Robotics - Shirt Design

I didn't actually get in on a lot of what happened in today's robotics meeting because I was messing around with WiFi settings and the Robot-C IDE.

Nevertheless, I did see some people working on the team T-shirt and design. I think the last thing they were talking about was a long sleeve shirt with black sleeves and a blue(?) main area. They were looking for some non-cheesy pictures of apples to use (because our team name is "Apple Pi"), but did not appear to be having much success.

There was some talk of adjusting the teams after one member of my team quit robotics (scheduling issue if I remember correctly), but so far nothing has come of it.

Robotics - Document Sharing

A lot more people attended today's robotics meeting than yesterday, so we managed to start getting everybody into the team collaborative spirit. The coach appointed me the document manager for my team, so I threw together some OneDrive shares for us and him to handle all those documents. I generated two documents for it today, each of which has a (short) story.

The first task - now that we have almost everyone together - was to assign roles. Of course, I'm the programmer. We put most of the new people on the engineering notebook and a returning non-engineer (but very capable people-person) as the marketing manager. The usual crew got on engineering and building teams. I also produced a small table of specific role descriptions, combined that with the people-role table, and shared the document.

The designers went to another room to start brainstorming, so I was left with the PR and notebook people. I noticed last year's WiFi router lying around, so I pulled out an Ethernet cable and a router power supply and began configuring it. We haven't purchased a license for the new version of Robot-C, so I didn't have a chance to flash the Samantha module. I did, however, verify that the router works. I also changed the password and updated the SSID, which had been left generic the previous year. The password was written in a text document shared in our team's OneDrive.

Robotics - Team Allocation

Today was the first official meeting of Rivermont Collegiate's two robotics teams. This also included the unveiling of the team allocations, which allegedly required a conference of four teachers deciding (based on skills and personality types?) how to best equalize the teams and promote communication in and between them. Apparently, whoever made the teams decided to not actually execute the plan of having a "super-team" of skilled veterans and a "newbie team" for the new people to learn at their own pace without bothering the experienced people too much.

We also started taking inventory. We have quite a few tools, motors, and especially LED lights (Why? We had extra money in the budget, I guess), but an alarming shortage of pristine building materials. The field this year is fairly complicated, so we will also have to devote quite a bit of time to building that. I imagine I will be on that task force because my part - programming - doesn't come in until the robot is built; the designers can design while the others prepare the field. I will need to set up the Samantha and Wi-Fi situation, though. Oh, and apparently our coach wants us to use Microsoft OneDrive to manage documents and schedules. I imagine I'll be in charge of that.

Overall, I am pleased with my team. It's small and contains a lot of people I know (including two other very skilled veterans), so we should do well.

Robotics - The Season Starts

This post isn't about a meeting - we haven't had one yet - but about the game, which was announced yesterday. As always, the official publication consists of a video and a super-huge legal-ish list of rules, neither of which is my cup of tea. So, I wrote a textual summary with optional pictures (Ctrl+click to view in new tab)!

• Same basic stuff: two drivers, one coach, robot fits in 18x18x18" at start, 2 teams in alliance, 2 alliances in a match
• This is a picture of the field, shamelessly ripped from the video
• Speaking of the video, here's a direct link or a link that starts after all the standard gracious professionalism and important values stuff
• In the center of the field is an elevated rectangular-prismal bucket of balls of 2 sizes (mostly small), here's a sense of scale
• Each robot starts with one small and one big ball on it
• The center bucket and holding assembly is set to one of three random rotations after all robots are set, here's a better view of that assembly
• Infrared beacons are mounted on the sides of that assembly, one on each
• It also has tubes mounted on the sides above the infrared beacon - mouth at the very top
• Each team gets three assemblies consisting of a tube on a rolling base: 30, 60, and 90 centimeters tall, here's what those look like
• Each alliance has one elevated corner with a ramp down and a delineated "parking area" on the floor; one robot starts on top of the ramp and the other in the parking area
• The goal is to get the balls into the rolling tube assemblies and then push those assemblies into the parking area
• During autonomous, points can be earned by driving down the ramp, knocking down the kickstands (balls everywhere!), getting those pre-loaded balls into the tube assemblies, getting those into the tubes mounted on the sides of the main bucket assembly, and pushing the rolling assemblies into the parking zone
• During manual control, points are earned for each centimeter in the container that the balls reach (multiplied by 1, 2, and 3 respectively for the varying heights of the tubes); the high mounted center tubes are off-limits
• During the endgame, points are earned by pushing the tube assemblies into the parking area or up onto the ramp
• Also during the endgame, balls may again be placed into the center-mounted high tubes for a minimum total of 90 points if at least one ball is in the tube
• Robots may not control over 5 balls at any time, interfere with opposing tube assemblies, or enter opposing parking area or ramp during the endgame
• Sponsors!

For details on the standard rules and tournament information, see Game Manual Part 1. For details on rules and specifications for Cascade Effect, see Part 2.

This looks to be a good game!

Abiathar Confidential: Why Closed Source

I've been away from electronic communications this week. I may or may not be back at the time this article is published. Enjoy this last automatically-published issue of Abiathar Confidential!

It is fairly obvious that Abiathar is a closed-source program. I have made known, and I will say again, that I intend to keep it that way. You might think that I'm being self-centered and control-freakish in doing this, but I intend to convince you here that being closed-source is a good thing. Also, I guarantee that Abiathar will always be free.

So, The Omegamatic ships with source code. TOM also has more concurrent versions than I can count on both my hands. This makes it intensely difficult to determine if a given distribution has a certain feature (e.g. KeenGraph graphics), a certain bug (e.g. crashing on undo), or whether it's adapted for some special purpose (e.g. NetKeen). There is no one version that does everything. I would like there to be only one version of Abiathar that can do everything - and so far this has been accomplished quite handily. Unofficial versions that branch off the main line would be a support and compatibility nightmare.

People don't need the Abiathar source code to add things to it. The entire point of the extension API is to save everybody time and effort by having a centralized point for addition of new features. Instead of recompiling and distributing a separate executable, simply write an extension. Unless you whack internal structures, extensions will be supported across all versions of Abiathar. The internals might change, but the extension interface is constant.

If you want to create a different type of tool (i.e. not a level editor or Abiathar extension) that uses the same formats, then there are two existing resources for you that do not involve the source code. First is FleexCore2/FMod; using this is a guaranteed way to be compatible with Abiathar and the real game. If you want to make a tool that does something with the Abiathar-specific formats, you have the format specifications. Some of those appear in the help file; I will provide the others upon request. Though Abiathar will never be open-source, it has always been and will always be open-standard and open-spec.

Of course, since Abiathar is written in .NET, it is extremely easy to decompile. If you really want to know how a feature works, go ahead! Even if I wanted to, I couldn't stop you. All I ask is that you don't use it to create derivative works; save you and your users lots of time by taking advantage of the supported ways to extend Abiathar.

Abiathar Confidential: The Only Abandoned Feature

I'm away from all electronic communications this week; enjoy automatically-posted issues of Abiathar Confidential!

From that first day when I sketched out what-was-then-called-Fudge's menu hierarchy to the release of Abiathar v1.2, there was a strange and little-known feature called Variable Width, found under the Tileset menu.

Variable Width was something I added because it seemed like an interesting idea at the time. I mean, you're usually going to have a boat load of empty space to the right when you're in tile palette mode. So, to get the most tiles accessible on the screen as possible, Variable Width ignores the convention of having 18 tiles on each row and instead checks the window width and does some division to see how many it can cram in.

This was a bad idea for two reasons. First, it was extremely difficult to code because it requires UI state to get back into the tool logic. The feature was added when there were no tools; picking tiles and placing tiles were hard-coded into the mouse-clicked procedure. When all that stuff was offloaded into extensible tools, I had to pass Tileset objects around that kept track of the actual width so the tiles would actually be where they appeared to be. It also caused a lot of headaches in updating tile palette view plane state: when the window was resized in tile palette mode, the render planes had to be notified. (It was also a huge pain to get Variable Width cooperating with Assimilate Infoplane.)

Secondly, Variable Width messed up pretty much every palette copy operation you might want to do. Most modders draw related tiles together so they can be copied as a block to be easily pasted into a level. Since Variable Width changes the width of the tileset, it jumbles up all the tiles. A tile that was directly below something related might now be on the same row. So, the feature never got any serious use.

Variable Width was made an "advanced option" before the v1.0 release, requiring a config option to be changed before the menu option was visible. This was done to help clear the menu item clutter, but it also made sure that even fewer people even saw Variable Width. I didn't really want to keep supporting something people didn't use, so the feature was officially cut in v1.2. (A little bit of code remains around for it, but I'm fairly certain v2.0's changes are not compatible at all with it.)

And there we have it. Variable Width is, to the best of my knowledge, the only Abiathar feature to have existed and then gotten cut.

Abiathar Confidential: Sneaky Ways to Get Around Fast

I'm away from all electronic communications this week; enjoy automatically-posted issues of Abiathar Confidential!

Over the time I developed Abiathar, I received boat loads of requests for shortcuts and navigation options, and added quite a few myself that most people probably don't know about. In this post, I'll make known all the shortcuts and navigation options that I remember - but it still might not be comprehensive!

In the beginning, Fleex compiled FMod.dll there was only one way to move around inside a level: press the arrow keys to move the viewport by one tile in the appropriate direction. Moving fast was done by holding down an arrow key. To go to a different level, you had to pull down the Level menu and pick from the list.

Once the basics like the selected tile bay were functional (this was when Abiathar was called Fudge), I added a feature that remains widely unknown. If you have a level entrance sprite in the infoplane bay, you can middle-click its icon to go to that level. This was adapted to Galaxy levels once support for them was added.

Still back in Fudge times, I added some modifier checks to the key-down handler to help you fly across the level with even more alacrity. If you hold down Shift while pressing an arrow key, it moves two extra tiles in addition to the base of one. If you hold down Control, it multiplies the value that considers whether Shift is down by five. Executive summary:

• No modifiers: 1 tile
• Shift down: 3 tiles
• Control down: 5 tiles
• Both down: 15 tiles

If you used Control and Shift, you could easily fly extremely far away from the level (bounds on scrolling weren't added for a long time), so the Backspace key moved you back to the upper-left of the level.

Remaining in pre-release times, but now called Abiathar (I think), support for the mouse scroll wheel was added. If you roll the wheel toward you, the viewport moves down; if you roll it away, the viewport moves up. If you hold down Control, the scroll wheel changes your horizontal position: roll it towards you to go right, away to go left. Each roll of a normal mouse goes two tiles.

Sometime after release (I think?), I added more ways to switch levels. The Omegamatic used PageUp and PageDown to switch levels, and this is entirely reasonable. In it went.

Around v1.2, people asked for a way to lock the scrolling so you can't fly away into no-level-land. This was actually very difficult to implement, but it got in, and remains under the config option FreeScroll (default: True). There are also some very complicated rules concerning the behavior of the mouse wheel at bounded level edges. I think the general rule is to start moving along the next wall when out of scrolling space.

To make arrow key modifiers even more complicated, there are two more configuration options as of v2.0 that control how far each unit of movement will actually make you go. ArrowScrollMul is a factor applied after keyboard modifiers to make each key press go even further. (ArrowScrollMul is an Integer, so if you set it to a less-than-one decimal number, it is entirely likely that it will be rounded to zero and make it impossible for you to move.) If you set it to 2, held down Control and Shift, and hit just one arrow key, you would fly 30 tiles over! 

The other option, MouseScrollDetScale, is pretty strange. It could be said to control how many mouse wheel detents are necessary to scroll one tile. The mouse I tested on, for example, sends 120 detents for each notch of the wheel. Divided by the default value of 60, this scrolls 2 tiles for each notch rolled. (It's non-linear!)

In the upcoming v2.2, there is a feature added that allows you to drag with the middle mouse button to pan the level. This makes it the first way to scroll in a non-perfectly-straight method.

So there we have it. This is all the ways of moving around in the level and switching levels that I can remember. There might be more, and there are always sneaky ways of performing edge cases efficiently in all areas of the tool.

Abiathar Confidential: A History of Backward Compatibility

I'm away from all electronic communications this week; enjoy automatically-published issues of Abiathar Confidential!

There have been a lot of Abiathar versions (over 20 if you count betas given to testers), and I always do my best to ensure you never have to run through the Dependency Collector or New Project Wizard again. Here, I provide a history of what compatibility measures are taken to make sure new versions of Abiathar can understand files produced by old versions.

The pre-release versions of Abiathar, back when it was still called Fudge, used a very simple format for the *.fudge (now *.adeps) files. It was basically:

• The profile name
• Whether it used default graphics
• Names of resources
• Local config (XML)

(To be honest, I don't remember when the XML blob got shoved onto the end of the file. It appears in v1.0 dependency objects, though.) One thing you will notice is that there is no marker for the version that saved the file. There is, as of v1.0, a version line at the beginning; that wasn't always there. In fact, it was added only a few days before the release; the testers never had a version that included it. So, when I realized that versioning is important, I had to implement backward compatibility right off the bat. Even v1.0 checks to see if there is a VERSION directive at the top and, if not, assumes the version is VersionId.PreRelease.

v1.1 added the ability to edit tileinfo in addition to displaying it, so the tileinfo line had to be extended. The new format was a metamap option character, a space, and the tileinfo file name if necessary. The metamap option character was used to get a set of policies for the handling of tileinfo e.g. whether to allow modifying it or whether it should be loaded from and saved to the MAPHEAD. When loading *.adeps files from v1.0, v1.1 gave it a default MOC of "R", a separate file that cannot be edited.

v1.2 was a fairly boring update if you're only looking for new features. It fixed a lot of bugs and changed the defaults for some settings in persist.aconf. These new defaults were not forced upon anybody and I would have no way to do so because Abiathar didn't save versioning information in the config files at this point. To get the new defaults, people had to delete persist.aconf to make Abiathar regenerate it.

v1.3 added the Level Inspector and Resource Accountant. Both of these required a huge amount of configuration and would have been totally crippled without updating persist.aconf. I had still not added versioning to config files, so the auto-update script for v1.3 just deleted persist.aconf and that was that.

v1.4 was, despite not having a lot of shocking new features, a pivotal update for Abiathar. The entire profile system was done away with, instead being replaced with a decentralized system of templates. Additionally, the XML structures used everywhere were thrown out and in came FrofConf. With this level of change going on, I gave up on backward compatibility. After v1.4, no version of Abiathar can read anything from a pre-1.4 version. Additionally, profiles.aconf was split into editor.aconf (for GUI settings) and defaults.aconf (for very complicated template stuff).

In v2.0, lots and lots of new features were added, but I took advantage of some amazing FrofConf features to ensure backward compatibility with v1.4. Here, every piece of information saved by Abiathar has a version tag. Whenever it's loaded, it calls AdaptVersion to process all the legacy entries and convert them to new structures. With load-only entries, I can read legacy information but not have to worry about it outside AdaptVersion. This is used on HuffCompressedMaps, for example, to set v2.0's MapType to "Carmack" or "Huffman".

After that, as I'm adding lots of new structures, the default value is always Nothing. All the code checks for null and will produce a "blahblahblah is not configured" dialog box. The upcoming v2.2 will check to see if AudioFormat is Nothing before trying to show the user audio-related stuff.

How do new features get enabled, then? AdaptVersion creates an in-memory default configuration file and copies important sections into the real one. In v2.1, Bio Menace templates were injected into the template list using AdaptVersion. (The Dreams template was also rebranded "Keen Dreams".) I imagine I'll use something like that in v2.2 to copy in the audio chunk layout.

So here we are. It is my goal to make sure you can always use old dependency files in new Abiathar versions to continue working seamlessly while also integrating new features for innovative modders.

Abiathar Confidential: The Strange and Unusual GalaxyGraphicsChunkSettings

I'm away from all electronic communications this week. Enjoy automatically posted issues of Abiathar Confidential!

This isn't about Abiathar itself, but FleexCore2/FMod, which is an important part of Abiathar.

If you poke around in FMod.dll, specifically the GalaxyGraphicsChunkSettings class or the GalaxyGraphics class that makes heavy use of it, you will find that the structure is a giant mess of strangely-named fields and flags, controlling a huge mess of control flow in GalaxyGraphics.FromFiles that has a cyclomatic complexity of almost 100.

GalaxyGraphicsChunkSettings is a class that determines the chunk layout of the EGA files. You must pass one to GalaxyGraphics.FromFiles so it can know how to interpret the byte streams it's getting. You could make one yourself, or use GetDefaultChunkSettings. Abiathar makes its own, for reasons that will be explained shortly.

Originally, GalaxyGraphicsChunkSettings was a simple and clean bunch of fields. It told FromFiles where to start interpreting things as a certain type of resource and where to stop. (If you set the table chunks to occur after the chunks containing the resources they define, FromFiles will die horribly because it assumes that tables always come first, which they do in all iD Software games.) For example, it had a SpriteTable entry to give the ID of the chunk containing the sprite table, a MaskTileStart entry to give the ID of the first masked 16x16 tile, a matching MaskTileEnd entry to give the ID of the last masked 16x16 tile, and of course a whole bunch of others.

But what are these strange entries with underscores before them? The most used one is _HeaderEntryLength: it tells FromFiles (and ToFiles actually, making it the only reason GalaxyGraphics saves the GalaxyGraphicsChunkSettings you give it) how many bytes should be used for each entry in the EGA header. This and all the underscored options were added while I was genericizing GalaxyGraphics to support Dreams as well. Why are they named with ugly underscores before them? Because I wanted them to be obviously not a chunk setting, and for a long time there was just _HeaderEntryLength.

Enter Abiathar version whatever-it-was-as-of-February. I had just shoved a whole bunch of EGA resources into the executable to use as default graphics. I noticed that the entire EGA file was loaded, decompressed, and rearranged, but only the tiles and masked tiles were used by Abiathar. To save load time, I added _LevelTilesOnly and _TilesOnly. (Why both? It seemed like it might be useful to somebody.) That saved the bit-twiddling to convert the chunks into the various resource types, but decompressing, especially in the default resources, still took quite a while. So, _DecompStart and _DecompEnd were added to ensure that only used chunks would be decompressed when loading the EGAGRAPH resource. You don't actually have to set these; FromFiles will assume you want to decompress all the things if _DecompEnd is 0.

In fact, the _LevelTilesOnly and _TilesOnly options are kind of redundant now because FromFiles has lots of null checks everywhere to make sure it doesn't load non-decompressed chunks as resources. You could, say, set only the range belonging to sprites to be decompressed - only the Sprites list will be populated.

Fortunately, the audio version of this structure (GalaxyAudioChunkSettings, which actually hasn't been published yet because it's part of the unreleased Abiathar v2.2) is a lot less of a mess; it contains no table chunks or _underscored options. Additionally, GalaxyAudio.FromFiles delegates the loading and saving of audio chunks to their classes instead of having a 500-line loader of everything. There are no fancy exclusion options for GalaxyAudio because it takes barely any time at all to do its work.