Gash - More CGA Insight

Yesterday while using Gash (well, FMod's NextGenGraphics class) to inspect the CGAGRAPH from the Keen 5 CGA edition, I discovered a little more information about the difference between the CGA and EGA editions.

First and most obviously, there are two fewer chunks in CGAHEAD than EGAHEAD. A quick inspection of the chunk lengths revealed that the two 32KB B800 texts were right next to each other rather than spaced by the two Terminator-style texts. That makes sense, considering that smooth scrolling is impossible in CGA and therefore the CGA editions do not display the scrolling Terminator-style intro, so they have no need for the texts that represent them.

I suppose there were no placeholder chunks because the missing ones are very close to the end of the array; shifting the ones that follow down two isn't hard to work around. (It's just the universe-exploding screen and the five demos at the end.) Compare that with the missing font in CGA, which is replaced with a small unused picture so that the entire array isn't shifted.

Gash - Everything Serialized

Today I finished the ToFiles method of NextGenGraphics, essentially completing the back-end of Gash. All I had to do today was add code to serialize texts, miscellaneous already-binary data, tiles, and small tiles. The trickiest part was the small tiles because of the way they're all stored in one chunk. The routine for saving them looks something like:

• Are there any small tiles cached? (This will be true if any have been requested or set.) If so,
• Get the decompressed version of the small tiles' chunk.
• Is it null? (DecompressChunk returns Nothing if that chunk is not present.) If so,
• Create a new one, with appropriate length, and use it instead.
• Figure out how many bytes an individual small tile takes.
• For every cached small tile,
• Serialize the small tile like any other image.
• Figure out (from the length of small tiles) where it should go in the chunk.
• Jam it into the main small tiles chunk, overwriting whatever was there.
• Assign the chunk into the appropriate index of the raw chunks list.

Tomorrow I will test the new additions, then I'll be ready to start working on Gash proper.

Gash - Does No Harm

I continued investigating NextGenGraphics's reimport corruption problem demonstrated yesterday. The first thing I noticed was that I had completely forgotten to check if the chunk had a constant decompressed size; ToFiles always wrote the UInteger representing the decompressed length. That was certainly the cause of the image distortion, but attempting to launch Keen with EGA resources from the fixed version resulted in an MM_GetPtr: Out of memory! error immediately after the "loading" B800 screen. I was fairly confused about that too, especially because Abiathar could use those EGA resources just fine - there were no extra lines or other distortions.

A little looking around in the FromFiles method revealed a precarious situation: the MemoryStream stored in RawCompChunks included the 4-byte decompressed length. It wasn't the source of any corruption because the stream's position was moved to the start of the real data, but I'm sure the arrangement would have resulted in baffling bugs down the road.

It was after fixing the FromFiles chunkifier that I realized there must be a problem with the chunks representing the Terminator-style texts because those were the resources about to be called when the game crash occurred. Interrogation of RawCompChunks showed that NextGenGraphics was not changing the decompressed size of any chunks, but one of the two Terminator text chunks was immensely larger than the other. Even more suspiciously, the entry for the smaller chunk was claiming to be the same size as a masked tile.

The problem was that I had configured chunk settings wrong in all my tests. I must have miscopied MaskTileEnd as I was transcribing the settings from the old GalaxyGraphics format; the mistake was replicated into all my testing methods. Setting the correct end of masked tiles (which have a fixed decompressed length and therefore don't get the heading UInteger) fixed the crash. Everything else appeared to be working correctly.

Tomorrow, I'll see if the image serialization code I wrote yesterday actually works.

Gash - Writable

Today I filled in the ToFiles method and the setters on all the NextGenGraphics properties. There is code to serialize all the graphics into their binary representations, but right now I'm trying to get it to read in some EGA resources and then spit them back out without modifying them (essentially decompressing the EGAGRAPH). That's not quite working correctly; sprites, pictures, and fonts make it through safely, but tiles, other images, and miscellaneous resources (e.g. the Terminator-style text) get mangled to varying degrees.

All tiles have a 2px-high line through them and the scoreboard is messed up, but sprites are fine.

I am fairly baffled as to how anything is getting mangled; no actual parsing or image processing is done - it's all using the deferred processing system to avoid doing unnecessary work. Further investigation will be done tomorrow.

Gash - Deferred Processing

I didn't actually get anything done on Gash today, but I'll take this opportunity to write down my plans for the implementation of its deferred processing (also known as laziness) because it's getting a bit hard to handle when the plan is only in my head.

When the xGA resources are opened with FromFiles, the main file is carved up into chunks which are stored in the RawCompChunks list because they haven't been decompressed yet.

When the chunk data is requested via a call to DecompressChunk, a check is made for the presence of the decompressed version of that chunk. If it is present, that same MemoryStream is returned. If not, the version in RawCompChunks is decompressed, and the resulting MemoryStream is added to RawChunks. The extra compressed version in RawCompChunks is removed.

When an image resource is requested, a check is made for the presence of the fully parsed version of it in the appropriate field. If it's there, it is returned. If not, the decompressed chunk is acquired and then parsed; the result is added to the appropriate cache field and returned.

When a new image object is assigned into a NextGenGraphics property, it is added to the appropriate cache field.

When it's time to save the xGA resources with ToFiles, the equality of the decompression dictionary and standard recompression (null-effect) dictionary is checked. If they're the same, entries remaining in RawCompChunks can be directly copied into the output. If not, they must be decompressed. Every image object in the caches is then serialized into the appropriate binary form and placed into RawChunks. Every entry of RawChunks is then copied to the output.

Gash - Rebinarizing

I finally started writing the part of FleexCore2's NextGenGraphics that actually recompiles the xGA resources. I wrote the methods that transform a LowColorBitmap into a byte array of colors-with-mask (the reverse of GetLowColorBitmap) and that transform that byte array into the actual binary data found in the image chunks (the reverse of GetPixelData). I have not yet tested either of those methods, because that would require having a functional ToFiles implementation.

I did start on the recompilation of the three xGA files; it is made a bit tricky by the ability to cache cache chunk data and reuse the raw chunks when the resources they represent weren't modified. When I next work on this I'll finish the xGAHEAD/xGAGRAPH writer to make sure the chunks are laid out correctly, then image serialization can be worked out.

Gash - Font Success

Today I finished up the export part of the NextGenGraphics class that will form the core of Gash. This required parsing the font chunks, which fortunately are documented on ModdingWiki. The EGA version of the exporter was very easy; the format in that article is correct. (I did run into a surprise with VB.NET while coding that: non-explicitly-initialized variables inside a loop retain their values across iterations.)

The CGA version was a little trickier. CGA font chunks apparently use what that article calls the version-1 format of fonts, so there are two bits for every pixel. That is, the font chunks are really just a bunch of standard CGA pictures jammed together, whereas EGA fonts are a single plane of EGA data.

I also wrote simple exporters for text chunks (becomes a String) and miscellaneous data chunks (becomes a Byte array). Next, I'll see about importing.

Gash - CGA Success

I did lots of work on Gash today, meeting with much success. I finally got the GetPixelData and GetLowColorBitmap functions working with CGA graphics. The trick was that CGA places the full mask of the image before the image itself, with 2 instances of the mask bit per pixel. I had also completely forgotten to add unmasked support in GetLowColorBitmap, so that also caused issues when loading background tiles.

NextGenGraphics can now extract sprites, tiles (background and foreground), pictures (masked and unmasked), and the 8x8 tiles. The small 8x8 tiles were a challenge because all of them are stored in one chunk. I wasn't sure how to get the individual tiles out because there's no information on that in the ModdingWiki and the GalaxyGraphics code is super horrible. It turns out that the small tiles are just 8x8 pictures in the usual format all pushed up next to each other. That is, extracting a single one doesn't require any shuffling of chunk-wide planes, just grabbing a section of the chunk and treating it as a picture.

Another interesting fact is that the CGA version of Keen 4 has one fewer font and one more picture to fill that chunk. The extra picture seems to be a placeholder; it's just a pink square with a blue edge on the top and left. (Probably to make sure all the other chunk offsets don't get messed up from the font removal.)

All the extraction routines work for EGA graphics also. No bit twiddling is actually done until the image is called for, so the initial load is very quick.

All that's left to do in the core is extract fonts and miscellaneous resources, then figure out a way to put back modified graphics. I'm still looking into VGA support; it seems pretty tricky thanks to an extra (non-VGAGRAPH) file that holds the sprites.

Gash - Sprite Extraction

Today I wrote two important routines for the NextGenGraphics class in FleexCore: one to transform raw decompressed chunk data into a 2D array of pixels representing color, and another to transform that array into an instance of LowColorBitmap.

The EGA (planes shuffled) version of these functions is working perfectly. After writing the getter for the Sprite property on NextGenGraphics and throwing together a quick test program, I am able to dump out all the sprites of Keen 4 into a collection of transparent PNGs:

The CGA and VGA (normal bit arrangement) version of these functions is not working so well. Even when I finally figured out exactly where the offset table (header) and Huffman dictionary is in the CGA version of Keen 4, I'm just getting garbled versions of the sprites:

It looks like the mask is actually after all the color data, not interleaved with it like I assumed. There's also some sort of horizontal duplication and overall squishing going on, but I'm not sure what's causing that.

Gash - Masking Understood

A while ago, I had a conversation on IRC with a programmer in the Keen community who has some experience working with the CGAGRAPH format used in the CGA distros of some iD games, which I would like to support in Gash.

While starting out on the rewrite of FMod/FleexCore2's GalaxyGraphics class, I quickly ran into a problem: non-EGA formats use a sane human's intuitive expectation of bit layout for each pixel. Rather than the EGA business of having all the red bits, then all the green bits, etc., the others have all the bits necessary to represent one pixel right next to each other. (Makes sense.) The problem is that such a scheme seems to require a non-factor-of-two number of bits per pixel in masked graphics, e.g. 3 for CGA (color, intensity, mask). That would be a problem because handling constantly-shifting non-byte aligned bit sequences is a huge pain and, as far as I know, not done in the actual game.

This community member and I, after some poking around, discovered that non-EGA layouts have a fairly obvious but also clever fix for the byte alignment problem. Rather than CGA doing this:

cimcimcimcimcimcimcim...

(Color, intensity, mask; respectively. The underline shows byte boundaries.) It does this:

cimmcimmcimmcimmcimmcimm

The mask bit is duplicated, rounding out the pixel representation to 4bpp. VGA does something similar, guzzling 16 bits per pixel! (Eight for the actual color, and eight instances of the mask bit.) Also, I learned that VGA games tend to keep only the UI/menu bitmaps in VGAGRAPH; the game pictures and sprites are in another file.

Armed with this knowledge, and lots of summer free time, I'll be able to create the most versatile xGAGRAPH manager the Keen community has seen.

Gash - Plane Problems

I started implementing the function for NextGenGraphics that converts a bunch of raw data into pixel values. This of course has to take into account the bits per pixel of the graphics adapter and whether it shuffles the planes around like EGA. I thought it would be easy until I remembered the mask plane. If included in CGA, there would be three bits per pixel, which would be very messy if it was arranged linearly. Sadly, there is no format documentation on masked CGA data or CGAGRAPH, so I just have to figure out where CGA keeps the mask information before I can move on.

Gash - Understand Sprite Shifts

The ModdingWiki's format documentation on xGAGRAPH claims that there are 8 shorts in each sprite table entry. However, I did some reading of the ModKeen code and determined that there are 9. The extra one is the number of shifts the game will create for that sprite, used in smooth motion. The old GalaxyGraphics implementation knew about the extra short, but it simply called it "flags" and didn't know what to do. NextGenGraphics labels it appropriately and provides an enum of the three valid choices: 1, 2, 4.

Gash - Better Bitmaps

I wrote the part of the new NextGenGraphics constructor that carves the xGAGRAPH file apart into chunks. Since decompression doesn't happen immediately, the raw compressed chunks have to be labeled with their size, taking into account the fact that some special chunks have their decompressed size hard-coded (rather than it being the first four bytes of the raw chunk).

I also realized that my existing SixteenColorBitmap class, had I tried to use it here, would have been both poorly named and really difficult to deal with. It doesn't support masking (you need MaskedSixteenColorBitmap for that) and doesn't have a very convenient way of dealing with palettes. So, I wrote a LowColorBitmap class that reduces the craziness (using the .NET Color struct instead of 4-tuples) and has built-in support for masking.

Gash - Delay Decompress

One of the most serious issues with the existing FleexCore2 implementation of Galaxy-style graphics was that it always decompressed every chunk as soon as the files were opened. All that bit-twiddling in such a high-level language takes a lot of time, so I had to do the whole workaround with the underscore-prefixed options on GalaxyGraphicsChunkSettings to make Abiathar's EGA loading not take a million years.

To remedy this and include the special CGA/VGA features necessary for Gash, I started an entirely different FleexCore2 class called NextGenGraphics. It uses the .NET Lazy(Of T) class to delay the construction of the graphics objects. Now, chunks will not be decompressed unless the resource they represent is called for. I have not yet started the actual graphics loading for this class, but it should be pretty easy once I figure out how to genericize the code that already exists in GalaxyGraphics.

Gash: The Superior xGAGRAPH Editor

There have been murmurings in the Keen community of bugs, limitations, and general shortcomings of KeenGraph. KeenGraph is the more modern replacement for the original ModKeen, and it is a very good tool. Sadly, it is written in QuickBASIC and is therefore difficult to maintain and extend. The main issue I find is that there is no way to give it custom chunk offsets - every time somebody mods another game, the program must be updated or forked to add configuration for that game's EGA arrangement. KeenGraph also does not support CGA or VGA graphics. (The main graphics file can be called EGAGRAPH, CGAGRAPH, or VGAGRAPH depending on the adapter supported.)

So, I plan on soon starting a replacement called Gash - xGAGRAPH Shell. I will address the following issues with KeenGraph:

• Language. Gash will be written in .NET to take advantage of existing FleexCore2 (FMod) functionality. This will greatly help debuggability and extensibility.
• Output format. KeenGraph only exports BMP images, which do not support transparency. Masking is done with a designated "transparent" color. Translucent colors are produced using a strange color-shifting system that I haven't been able to use successfully. Gash will support the export and import of PNG images, interpreting a transparent pixel to be an invisible black pixel (transparent) and a translucent pixel of any color to be that color unmasked.
• Tolerance. I personally enjoy the Paint.NET editor very much, but it butchers the 16-color bitmaps into something... not 16-colored. This causes corruption when the images are imported by ModKeen or KeenGraph. Gash will use the .NET image loading routines to accept any bit depth. If slightly wrong colors are used (I'm still not sure whether EGA white is supposed to be $FFFFFF or $FCFCFC), it will look through the palette for the closest match, use it if it is found, and warn if it's really far off.
• xGA format. KeenGraph only deals with graphic-planar EGA, as far as I can tell. Gash will support CGA and VGA as well, and row- and byte-planar EGA if any Keen-ish games actually use it.
• Configurability. KeenGraph does not support the use of custom chunk offsets. This is a very simple thing to do with FleexCore2, so Gash will support it.

Work on this may start tomorrow. I hope to have this done will all these features by Keen Day 2015.