FMod - Yay, Galaxy Levels!

List of compression types which I never want to hear about again: {LZW, Carmack}

It is really awful when documentation is inaccurate, inconsistent, and confusing. I did finally manage to navigate and experiment enough with the gamemaps format to make it work! To start out, I made things easier for myself in the future: I changed the compression/decompression algorithms to take in a reference to an output stream instead of creating a new stream and returning that.

Something I missed when reading the documentation was that the address mentioned by a Cramack far pointer needs to be multiplied by two because it's in words. Two of the three documents I read made no mention of that and the one that did, did so very unobviously. With that done, I was able to load and resave gamemaps and maphead successfully!

Next up is Galaxy graphics, which includes Huffman compression. I have absolutely no idea how that works, but I have set up some nice interfaces and classes for the strange chunk handling.

FMod - Galaxy Levels are Complicated

This FMod development series is monopolizing my blog. It will probably take a break when I resume classes in a week. Until then, old file formats engage!

Despite my tendency to busy myself with many projects at once, within each of those I focus on one task. Continuing from yesterday, I worked on the Galaxy level class. After a more careful read of the existing documentation, I discovered that my Carmack decompressor was incorrect. After changing it to use the right order of bytes, it expanded the chunks much more sensibly. However, there was another problem: some extra bytes are present between the address referenced in the level header and the apparent start of actual data. I determined the offset to be 8, but soon ran into the problem of decompression flags pointing to negative addresses. After digging through ever more obscure documentation, I discovered that four of my eight offset bytes had a purpose: decompressed length. The other four were actually part of the important data; I had dismissed them because they didn't represent the first tile - they were an RLEW flag. After adjusting my reader and writer accordingly, it worked a lot better. I still have one problem left to work out: it starts behaving unpredictably after encountering a far reference.

FMod - Galaxy Levels

With basically all the Vorticons resources done, I figured it was time to move on to Galaxy modding. As I started in Vorticons with levels, I figured it would be good to do that with Galaxy as well. The game maps file is just a single file containing multiple levels, so I created two classes: one for the collection and one for a single level. With a new format came (of course) another compression method, this one called Carmack. It's very easy to decompress, but considerably difficult to compress. Fortunately, unlike LZW, it doesn't have to be compressed to maximum efficiency.

The decompressor appears to work well and my temporary compressor just escapes the flags instead of actually compressing. It can load all the level information from maphead and the individual headers in gamemaps, but I'm still uncertain as to whether it is reading level data correctly. Due to an issue with my handling of unsigned integers, resaving the maps collection does not work at all.

FMod - No More LZW

If I ever see the letters "L", "Z", and "W" together in that order ever again in my life, I think I will scream.

Including my research today, I have discovered three incorrect implementation of the LZW compression algorithm. Surprisingly, I got correct results from, of all places, Wikipedia. My problem was that I was always using the newly generated code instead of emitting the code for the longest string already in the dictionary. With that done, Keen 1 accepted my compressed version with one problem: sprite colors planes were obviously offset. It turns out that Keen 2, against which I had been testing uncompressed IO, never had small-sprite aberrations. My handling of those was incorrect. After correcting that, everything worked great in Keen 1 - but trying to compress the graphics of Keen 2 resulted in something incredibly messed up. Apparently, the "decompressing graphics" message shown by Keen 2 never actually decompressed the graphics; they removed that functionality for the later games. I'll just have to not compress those games' graphics.

LZW: done. Next up: Carmack.

FMod - EGA Recompilation

The EGA format is surprisingly very simple. It doesn't require crazy bit-nonaligned IO, just shuffling some bits into different groups. That only has to be done at the end, so before then, I can just worry about putting the right color bytes down. Just as I expected, the header is written last, compiled from a list of entries generated while copying image data. It worked almost perfectly the first time. I had assumed, under the guidance of an online file format, that I could leave the smooth movement sprite duplicates to Keen. That was not the case. Actually duplicating the entry fixed the clipping problem.

However, when I tested the VortGraphics ToFiles method with compression, it failed in the most annoying way possible: it only partially succeeded. The tiles are only partially corrupted; I can still see the outline of the items and some of the color planes are correct. However, without looking closely, everything looks like garbage. I rewrote the core of the compression algorithm according to a different online guide and it still fails similarly. Just when I thought I was done with LZW...

FMod - The Vorticons EGA Reader

It's very difficult to follow file format specifications when they're wrong. This problem occurs semi-frequently when using community-produced documentation for old games like Commander Keen. Therefore, creating the EGA graphics exporter of FMod was challenging. First, the least significant bit in the header's compression flags does not tell whether that header is compressed; it refers to the latch file - the header is never compressed because that would be impossible.

Also, the tileset is not one big bitmap. It is actually about six hundred small 16x16 pixel bitmaps (32 rows each). If you try to read it as a large bitmap, you will spray rows nonsensically throughout the tile array.

Fortunately, those errors were the extent of the confusion. It was actually pretty easy to read the separate color planes into a more easily accessible array and then use that to put together the rows. The extractor first reads the header to get all the information about each entry and then puts each into the VortGraphics instance's appropriate list.

For my test program, I just had it use my ToFile method of the bitmap class on each of the graphics entries. I was shocked when it worked almost perfectly the first time. (The tiles were corrupted as I mentioned above, but that was an easy fix.) Tomorrow, recompilation of the three EGA files!

FMod - LZW Done

I have definitely had enough LZW compression for a while. Today, I modified the decompression algorithm to use byte arrays instead of strings and bytes instead of booleans (booleans always take four bytes). That was easy, but there was still one major thing left to do: compression.

Since I am now very familiar with the LZW way, compression was not as difficult. It did, however, require consulting a few more sources for examples in other languages. It didn't help that the site I referenced for decompression had an incorrect algorithm for compression. I did not heed the Internet's suggestions to use a real dictionary or any kind of hashing, instead using my trusty list of byte array. It required some more super inline lambdas to make accessing it work, but it's pretty fast.

Bit-level IO is done in the opposite order as decompression; it reads the input by bytes and keeps an output list of bits, which is shoved into bytes when the output is finalized.

Check it out - Keen LZW compressor in VB.NET

FMod - LZW Decompression

It took me two days total, but I finally got an LZW decompression algorithm for the Keen EGA format working in Visual Basic. It uses all manner of crazy stuff, from bit lists to multiline lambdas. It could definitely use some cleaning up and refactoring; during my work I switched from byte arrays to strings.

It successfully decompressed Keen 1's "egalatch" and "egasprit" files, which the game accepted after I changed the "egahead" compression flag manually. Tomorrow, I will do my refactoring on the decompressor and implement compression, which hopefully will be easier.

View the algorithm on PasteBin

FMod - LZW Rage

After opening a gazillion tabs about file formats in my browser, I got to work this morning on the EGA graphics and LZW compression format. The first order of business was to get a quick mock-up of the header ready in the FromFiles method of the new VortGraphics class. With that ready, I put on my pay attention face and added a DecompressLZW method to my compression library.

It took me an hour just to get any idea of what LZW is even supposed to do. I found three different algorithms, two of which were psuedocode and one of which was in C. (I did later find a really messy un-object-oriented one in QuickBasic.) At first I was going to use an jagged array of bytes to manage the dictionary, but realized that would become terrible with the extended-bit entries and changed it to a dictionary of unsigned integer and byte array.

Since bit operations are truly awful in high-level languages, I created a list of booleans to store the each bit. Later, thinking that was a waste of memory, I created a bit getter lambda (which are the super inline functions I wanted) to find a single bit in the data stream. That failed due to weird side effects of the overloaded And operator. So, I created an inline iterator function to figure out the bits as I needed them. That failed because some buffering thing done automatically caused it to read more bytes and get out of sync. At the end of my fiddling with the data handling, it was back to the boolean bit list.

I ran into all kinds of problems with dictionary keys. First, it was crashing with KeyNotFoundException immediately after preemptively adding the value for that key. Then, after I did some value remembering stuff, it crashed with NullReferenceException in the outputter section. Tweaking the data reading loop to accommodate that just resulted in early exit.

The core algorithm has been rewritten at least four times. I have been working with Visual Studio, XVI32, and Keen on this the entire day. It is still not working.

FMod - Bitmap

I was right. FMod is going to usurp all my free time for the next few months. Sigh... let's dev!

Since it will probably become important at some point, I wrote a class to encapsulate a 16-color (4-bit) bitmap that doesn't really conform to any standard but Keen's. That was exceedingly painful because [1] two pixels are packed into each byte, [2] each row has to be a multiple of four bytes - not pixels - long, which wouldn't be so bad if it weren't for the fact that [3] the entire thing is actually stored upside down - but still left to right. The first problem I encountered was that the bottom image was interspersed with black horizontal lines and made the entire image. It didn't take terribly long to figure out that I had made an error in calculating the start of a new line of pixels. However, the next problem was much more painful: assigned pixels were being shifted to the left one. It took literally two hours of messing with ten lines of loop code to make any headway on it. The shift was fixed by adding calls to Math.Ceiling in various places, but then I was stuck with a black vertical line on the left. Fortunately, a little more shuffling of "+1", "-1", "*2", and "/2" fixed it.

With the bitmap class working well, I ripped the compression algorithms out of the level IO code and into their own reusable module. That module will later contain implementations of LZW and Carmack compression.

FMod - Just a Side Project

Time to start another large project! Now I have four ongoing: HigherPower (suspended until Forge updates), Logofrag (never made it to the add frag screen), NetBLAM (unsure where to go now), and this.

What's this? It's FMod: a total mod creator for Commander Keen. My vision is for this to be able to do all operations necessary to creating a Keen mod - tiles, levels, sprites, texts, behaviors, and other miscellaneous stuff. I really want this to just be a side project that I work on when I genuinely don't have anything else to do. Unfortunately, it will probably usurp all my free time like my other projects initially did.

Anyway, I started by creating a class for Vorticons-engine levels. Since it was essentially ripped from FleexCore, it wasn't that hard to write the original code. The trouble came when I tried to remove extraneous functionality and use streams instead of VB6-style files for the IO. I deleted a bunch of unnecessary code, cleaning it up quite a bit. In the process, I moved all the number-shifting functions (e.g. converting an unsigned 16-bit integer to a byte array for writing) to their own module, accessible to any class.

When I tested the new class, it worked - kind of. When it resaved the level, everything was shifted two tiles to the right. I had no idea what was going on, but continued adjustments to various numbers and addresses created a different problem. Tiles weren't shifted, but sprites were going completely crazy, having a million junk sprites covering the top half of the level, shifting the real ones down. It took a hex editor and Mindbelt to figure it out, but I finally noticed that I had forgotten to multiply the area by two in calculating the plane size. With that fixed, the test succeeded. It's very important to remember variable sizes.

Here we go!

Robotics - Smoking Pointers

Though there was a robotics meeting this Wednesday, I didn't log it here because I had more interesting thoughts to put here instead.

Since it was the first day of finals, there were not very many people there. In fact, it was just me, the supervisor, and a builder. The builder wasn't doing much of anything when I arrived; there really was not much to be done without the input of the lead builder. He left, leaving just me and the supervisor. As I am not a builder, I couldn't do anything at all (besides adjust a little bit of wiring). I did some poking around in the intrinsics definitions and learned all manner of interesting things about how RobotC does stuff. I finished coding my attempt at circumventing the field control system, which I began testing at the prompting of the mentor. It used jumps to hardcoded pointers in addition to all manner of other terrible stuff, so I really didn't expect it to work. However, it definitely did. Ending the program in the debugger failed to stop the motors, resulting in some smoke.

After those tests, an engineer from John Deere arrived to help us. We did some fiddling with the hanger bar, but discovered its motors could not maintain the position and trying to do so resulted in bowing of the channels. Since my attempts at end-of-game circumvention were uncontrollable, they were abandoned and we still have a lot of work left to do.

Super Inline Functions?

I did some more thinking about what would be awesome to have in .NET and remembered something about my new implementation of SupaChat Server. For a lot of the commands, it tries to find a group object for a group ID specified somewhere in the arguments array. Therefore, I have to have code like this in those case blocks:

Dim TargetGroup As Group = Groups.Find(Function(g) g.ID = _
 Args(1))
If TargetGroup Is Nothing Then
 Sender.SendData("*G* Group doesn't exist!")
 Return
End If
If Not Sender.RepUser.InGroup(TargetGroup) Then
 Sender.SendData("*G* You're not there!")
 Return
End If

In addition to being simplified by the breaker subs I talked about last time, this is kind of a pain and is very repetitive. I would like to always define variables for things like the target group, target user, and sending rank and have them only set by one line - a "set-up" call if you will. However, that would require a new kind of inline sub. It's hard to explain, but I was thinking of something like this:

Breakable Sub HandleCommand(Command As String, Args() As String)
 Dim TargetGroup As Group
 ' other variables
 Inline Breaker(1) Sub GetGroup(ByInline TargetGroup As Group, _
  ArgPos As Integer, NeedsPresent As Boolean)
  TargetGroup = Groups.Find(Function(g) g.ID = Args(ArgPos))
  ' etc from code in last section
 End Sub
 Select Case UCase(Command)
  Case "LEAVEGROUP"
   GetGroup(1, True)
   ' other instructions
  ' other commands
 End Select
 ' other stuff
End Sub

That "inline" sub inside the command handler sub is only visible and runnable in the context of that one sub. Variables passed ByInline pass a double-strong pointer, so setting where it points inside the inline sub would change what it points to back in the root sub. Inline subs must appear after all local variables they use. When invoking inline subs, ByInline parameters cannot be specified; they are taken from the current accessible variable set.

This would be an amazing feature. Get on it, Microsoft!

Force Multiple Returns - "Breakers"

I would really, really like a way in .NET to unwind the stack twice in a return-like statement. This would be exceedingly useful in request handling methods that check some conditions and exit the big handler if the check fails. There is currently no way to do this without inserting "if check is false then exit" everywhere. It's even more of a pain in ASP .NET applications that require the request to be aborted if the page will not be fully served.

Sub Page_Load() Handles Me.Load
 If Not IsUserAdmin(Session("UserID")) Then
  Response.Redirect("PermissionDenied.html")
  Exit Sub
 End If
 ' actual page loading stuff
End Sub

Here, it would be excellent if the check function could cancel the Page_Load routine. (It might be necessary to pass the Response variable to an auth function.) I was thinking something like this:

Breaker(1) Sub EnsureUserAdmin(ID As Integer, _
 Handler As HttpResponse)
 If Not IsUserAdmin(ID) Then
  Handler.Redirect("PermissionDenied.html")
  Break 1
 End If
End Sub

Breakable Sub Page_Load() Handles Me.Load
 EnsureUserAdmin(Session("UserID"), Response)
 ' administrative things
End Sub

The parameter to the Breaker keyword specifies how many frames up the stack the method can break. The Breakable keyword flags the method as being able to use breaker methods.

I realize it's kind of redundant for void returners as they could be changed into Booleans, but for functions that can actually have a useful output, it would be a real pain to use a composite/nullable structure. Please, Microsoft?

Watch out for Shadows

While messing around with types for a personal project, I ran into some unusual problems involving the Shadows operator. For background, let me explain that I had two classes, one of which extended the other, each of which containing an identically named class that would hold additional data for it. I wanted one class to override the other in the derived class, thereby swapping out the data type of that extra storage field. Since classes can't be declared Overloads, I settled for Shadows, which seems to have the same effect. I applied it to both the class and the field.

But it doesn't! Members declared Shadows will only be called if the invocation is on an object that is strictly known to be of the derived type! I was actually declaring two fields with an identical name but different data types. When I used the polymorphic approach and simply read from the field, I was only accessing the field declared by the superclass, never the subclass.

Shadows is not the same as Overloads. Make sure you know you're creating two different members when using it. In my case, the issue was solved by changing the field to a property and giving the conflicting members (both class and field) different names. Then, I could just have the subclass report its own field of its own data type.

Assembler Simulator

I was reading about old assembly code today and I thought "I would really like to see how drastic these crazy speed tweaks are!" I don't have access to an assembler or a machine slow enough to let me notice, so I decided it would be a good idea for someone to build a program that simulates running assembler code on various old CPUs. It would be fine if it was in some high-level language; I just want to be able to control the clock speed and watch memory locations. There should be readouts for the registers and a display corresponding to a graphics memory location. Of course, all of this would be simulated - but it would make perusing old documentation so much more interesting. I just might make this.

Crushers - Seedy

I was in the mood to play a little Crushers today. Shortly after playing a few games, I was in the mood to make some additions and changes to it. First, I fixed a bug that allowed the player to suffocate in thin yet open areas. After I saw some amazing events unfold in-game, I decided that I wanted a way to seed the RNG with a value that would cause it to repeat all the block-related events. It took me quite a while to figure out that, if I wanted block falls to be repeatable, player actions could not be sources of randomness. Once I got the idea, it was trivial to remove randomness from the oxygen particles and add a seed dialog to the main menu. Now, players can replay awesome situations!

Robotics - A Little Trim

We were again very understaffed at the robotics team meeting today. There was, fortunately, one more person - a builder - present. I was alone for the first few minutes and finished up our wiring changes. The rest of the time was used to switch out the gears - getting us under the height limit - and to cut off the very ends of the flag raiser which were hitting the hanger bar. I also adjusted the programming in an attempt to keep motors running after the end of the tele-op period.

Robotics - Wires Everywhere

As there was only one other person at today's robotics meeting, not a whole lot was done. Since neither of us is actually on the build team, pretty much nothing got done. We did, however, figure out how to use wire crimpers to extend some tightly-strung cables, routing them along the metal beams. We also discovered a problem that caused the flag turner to yank the cables out of the hanger bar, which was also fixed by making the wires longer and getting them out of the way. Hopefully, more people will be at tomorrow's meeting so we can actually attach stuff.

Robotics - Redesign Again

Over the last few weeks, we have yet again revamped our robot's layout. Since we decided not to use the conveyor belt as a block-moving mechanism, it has been coerced into a tank-like movement system. The center of the robot has been cleared out and replaced with a swingable bar controlled by two motors, which we use to hang! The flag raiser, due to length constraints, has been flipped around and now points forward. It's somewhat difficult to set it up in such a way that it doesn't interfere with the hanger bar or our many cables. Surprisingly, pretty much everything worked on our first run of the new system; we just need to reaffix the crate and find a permanent place for the battery.

The CLI/CIL/CLS/CTS Specification

Most .NET programmers aren't aware that it's compatible with other languages that we don't usually consider part of that family, like Mono. This is because its written assemblies and execution environment are standard - set by Microsoft and quite a few other companies. The spec provides details on the Common Language Specification, the Common Intermediary Language, the Common Type System, in addition to a lot of helpful tips for programmers. The thing, at its heart, is comprised of a list of rules for compilers and translators that, if followed, guarantee security and compatibility. The ECMA also packages an XML file than contains information on all necessary base classes.

ECMA page
Direct PDF link
Mirrored BCL XML

RSS MP3 Downloader

A lot of RSS feeds always contain an attached MP3 file, like podcasts. It's very difficult, if not impossible, to download them all at once without manually initiated every download. I don't know of any browser that is good at handling RSS that it can do so for multiple different feeds.

I encountered this issue while trying to fetch all podcasts from Reasons to Believe and therefore invented RTBDownload. It's a tiny little application that takes a folder path and an RSS XML file, parsing it for titles and file links. It then downloads all the MP3 files and places them in the folder you specify.

RTBDownload for Windows 7 and above

When it launches, navigate to the folder you want to put them all in, paste the URL of the XML file (with Edit -> Paste), and press Enter. It will inform you when it begins downloading each file. It is possible to run multiple instances of it downloading different feeds.

F#

I saw the name of F# in my Visual Studio add-ons dialog today and it made me remember how interesting that language is. I can't write anything it it without extensively using reference material, but it appears very powerful and very well integrated with the CLI. A while ago, I thought it was Microsoft's answer to R, but that does not appear to be the case. It is more like a .NET-compliant version of Haskell with more object-oriented features.

Some of its most interesting abilities are its handlings of the functional pattern. Types are handled in a way less painful than in VB and even more flexibly than in C#. However, the nature of variables and assignments to them seems a little bit strange, at least from my VB perspective.

Who should use F#? I'm not sure, to be honest. It does look like a good language to know should you ever stumble upon an amazing possible use of it.

Do Not Use - Pixela Camera Utilities

Canon "FS" video cameras ship with a whole ton of software installation discs. The cameras are great, the software not so much. This is similar to the junk you get when purchasing a new computer from most big-name companies: not-so-great video editors, default music files, and generally useless stuff. It's true that one of those utilities is "necessary" to transfer the video files from the camera, but there's another, probably cheaper, way to do it. Instead of buying all those discs (I'm fairly certain it's actually optional), just get a USB-adapted SD card reader. Pop the card out of the camera, connect it through the reader, and copy the video files out with Explorer. Much cheaper, less junkware.

.NET - Writing Your Own .NET Compiler

Reflection in .NET is exceedingly easy. It's also incredibly versatile; in fact, it's possible to create new classes, methods, and other fancy stuff with the *Builder classes. Add those to an Assembly and you've got yourself an entire library in memory. There's one method that makes it all even more amazing: the Save method on AssemblyBuilder. It will take all the dynamically generated code and dump it into a legit .NET library on your hard drive. I think you can even make DLLs by specifying the appropriate options in AppDomain.DefineDynamicAssembly.

So, if you're good at using ILGenerator, you can take in a text file, do whatever parsing and understanding you need, create the appropriate classes and method bodies on the fly, and create a new CLI-compliant executable. This is, essentially, a compiler. Learn more by reading the documentation on the types in System.Reflection!

Do Not Use - Print Shop Pro

Broderbund Print Shop Pro is a pretty nice application. It contains a huge amount of templates for printing on and designing for special papers types in addition to having a massive library of clip art-like stuff. However, it really messes with your system when you actually go to print. The times I have attempted to use it, my printer spooler went insane. I don't remember what I had to do to fix it, but it was awful. There's got to be another good way to lay out labels and special documents - PSP just breaks stuff.