After a long winter break (and a long series about FMod that everyone is probably tired of now), the robotics teams got back together to finish it up before the competition in a month.
Despite my use of the word "finish", nothing was finished or really done at all today. The main builders were away at basketball, so we really couldn't do anything serious. I did manage to move the infrared sensor over to the correct side, which took a surprisingly long time. (It's amazing what things in the way will do to ease of access.) A whole lot of people left early, leaving me the only person on my team left doing anything. I used that time to rewrite the autonomous code to reflect our new design, which needed to be updated since November. It now takes advantage of the daisy-chained controllers, saving three NXT sensor ports. It works pretty well, but I still need to get the servo angle and drive times right.
Wednesday, January 8, 2014
Tuesday, January 7, 2014
FMod - One Last Hurrah for Huffman
With full Keen Dreams support in the graphics part of FMod, the last (important) thing left to do about that strange episode was to edit its levels. Since Carmack compression didn't exist at the time of its release, they used Huffman on the map chunks instead. They also applied Huffman to the level headers, giving the compressed length of the main header in the super header.
Since the uncompressed structure is identical, I copy/pasted a lot of code from the GalaxyLevels class into the new DreamsLevels. Since the only existing thing using Huffman was GalaxyGraphics, I centralized some methods relating to IO of the dictionaries. People haven't done a lot in the way of modding Dreams, so I had to manually rip the map superheader out of the executable for testing. I am very confident as to the correctness of the place I took it from; there's no other instance of ABCDh followed by a relatively sparse region. However, I am considerably less certain that I took the maps' Huffman dictionary from the right place; I can't get the decompressed data to make sense no matter what I do to it. I'm almost 100% sure that my decompression and dictionary reader algorithms are right and that it's just a question of using the right dictionary.
Since the uncompressed structure is identical, I copy/pasted a lot of code from the GalaxyLevels class into the new DreamsLevels. Since the only existing thing using Huffman was GalaxyGraphics, I centralized some methods relating to IO of the dictionaries. People haven't done a lot in the way of modding Dreams, so I had to manually rip the map superheader out of the executable for testing. I am very confident as to the correctness of the place I took it from; there's no other instance of ABCDh followed by a relatively sparse region. However, I am considerably less certain that I took the maps' Huffman dictionary from the right place; I can't get the decompressed data to make sense no matter what I do to it. I'm almost 100% sure that my decompression and dictionary reader algorithms are right and that it's just a question of using the right dictionary.
Monday, January 6, 2014
FMod - I Have a Dream
Huffman decompression is pretty easy, especially now that I know about that weird bit order reversal that goes on with the dictionary! Compression was a little harder because I need to create a dictionary, but I had it default to the trivial dictionary (no compression) unless a better one is specified. There was also the issue of writing down that dictionary to EGADICT, which I did in a way a little messier than the original. It's OK; it's the same size and is equally fast, just looks strange to the eye.
While trying to add new GalaxyGraphicsChunkSettings instances, I discovered that some episodes have texts in a different order. That immediately presented an issue: it always assumed the order used by Keen 4. So, I changed those variables into a list of strings. Shortly, I discovered that Keen 6 lacks some resources that both 4 and 5 have. So, I went back through and inserted checks to see if various fields of the chunk settings object were zero before trying to read the resources.
Though I didn't get a chance to test reading episodes 5 and 6, I went on to Keen Dreams, that strange in-between episode that is full of exceptions and special cases. It took an hour just to get the EGAHEAD reader generalized to its four-byte arrays instead of the newer three bytes. I also discovered a problem with the hardcoded chunk sizes of the small tiles; Keen Dreams has fewer of them. That was solved by adding new fields for that to the settings instance. I'm still having trouble with getting decompressed sizes for KD, though; they always return zero.
While trying to add new GalaxyGraphicsChunkSettings instances, I discovered that some episodes have texts in a different order. That immediately presented an issue: it always assumed the order used by Keen 4. So, I changed those variables into a list of strings. Shortly, I discovered that Keen 6 lacks some resources that both 4 and 5 have. So, I went back through and inserted checks to see if various fields of the chunk settings object were zero before trying to read the resources.
Though I didn't get a chance to test reading episodes 5 and 6, I went on to Keen Dreams, that strange in-between episode that is full of exceptions and special cases. It took an hour just to get the EGAHEAD reader generalized to its four-byte arrays instead of the newer three bytes. I also discovered a problem with the hardcoded chunk sizes of the small tiles; Keen Dreams has fewer of them. That was solved by adding new fields for that to the settings instance. I'm still having trouble with getting decompressed sizes for KD, though; they always return zero.
Sunday, January 5, 2014
FMod - Denounment
Even though I didn't write very much new code at all today, I did make a lot of progress on FMod! First, I adjusted my bit twiddling in the EGAHEAD writer, repairing the overflow issue and allowing Keen to read my graphics. When I did run Keen with those graphics, I found that everything was blue; only the third and fourth planes had been used.
It took quite a while of scratching my head, stepping through the code, and running tests to determine that I was running a for loop backwards. I was using my "check if value minus two to the n is not less than zero" trick to split the bits, but I was running it with n starting at zero and getting bigger, which would obviously put the bit in the wrong plane. Making it a "step -1" fixed it, and it only took me a little while to find the places where I forgot to reverse the loop bound constants, which resulted in all-black tiles.
Then, I checked the ModdingWiki and found that my question about Huffman tree reading had been answered. Keen apparently likes to reverse the bit order of the character data in the EGADICT, which was the cause of the problem. I will be able to fully reintegrate Huffman compression tomorrow.
The last little issue I had was with the large scrolling texts at Keen startup. In my reader, I had used the wrong variable inside a loop, causing the loop to have no effect. With that corrected, all that remains is to stop Keen from wigging out at the edge of those texts.
It took quite a while of scratching my head, stepping through the code, and running tests to determine that I was running a for loop backwards. I was using my "check if value minus two to the n is not less than zero" trick to split the bits, but I was running it with n starting at zero and getting bigger, which would obviously put the bit in the wrong plane. Making it a "step -1" fixed it, and it only took me a little while to find the places where I forgot to reverse the loop bound constants, which resulted in all-black tiles.
Then, I checked the ModdingWiki and found that my question about Huffman tree reading had been answered. Keen apparently likes to reverse the bit order of the character data in the EGADICT, which was the cause of the problem. I will be able to fully reintegrate Huffman compression tomorrow.
The last little issue I had was with the large scrolling texts at Keen startup. In my reader, I had used the wrong variable inside a loop, causing the loop to have no effect. With that corrected, all that remains is to stop Keen from wigging out at the edge of those texts.
Saturday, January 4, 2014
FMod - Put It Back
I learned a little bit more about Huffman compression and the dictionaries involved there today. I determined that my way of reading the dictionary is flawed and results in the strange mangled bit-backwards values I encountered yesterday. I found a lower-level guide to reading Huffman dictionaries, which I will mess with tomorrow. Unfortunately, I might not get to use my nice generic recursive binary tree class.
Continuing to ignore the massive issue of putting stuff in an actually compressed format, I decided to keep going and write the Galaxy graphics writer. Given the documents I acquired yesterday and the reader code I wrote based on them, getting the first draft down was easy. Of course, it didn't work the first time. I had mis-renamed one of the mask plane variables after adapting it from the unmasked version. There was also an issue with my byte math when trying to write the locations of chunks to the EGAHEAD, but that was also easily resolved. Keen still can't read an EGAGRAPH written by FMod, but XVI32 says I'm close - there's something wrong with the length of something starting at the fourth or fifth chunk.
Continuing to ignore the massive issue of putting stuff in an actually compressed format, I decided to keep going and write the Galaxy graphics writer. Given the documents I acquired yesterday and the reader code I wrote based on them, getting the first draft down was easy. Of course, it didn't work the first time. I had mis-renamed one of the mask plane variables after adapting it from the unmasked version. There was also an issue with my byte math when trying to write the locations of chunks to the EGAHEAD, but that was also easily resolved. Keen still can't read an EGAGRAPH written by FMod, but XVI32 says I'm close - there's something wrong with the length of something starting at the fourth or fifth chunk.
Friday, January 3, 2014
FMod - Exporting all the Galaxy Graphics
Until I figure out what's going on with the Huffman dictionary, I have commented out the decompressing part of loading the Keen Galaxy graphics. It appears that EGAGRAPH is already fully decompressed from being run through KeenGraph.
Today, I wrote a little bit more code to finish extracting the other resources found in EGAGRAPH. After a little bit of testing, I determined that my temporary decompression fixed worked with the trivial Huffman dictionary. I also discovered some issues in the masked 8x8 tile reader, which were fixed by using the correct plane array. I also inverted the exported font colors to be more like KeenGraph. After polishing everything up, I wrote some test code to dump some of the resources to my test directory and it worked! Font characters, tiles, sprites and other graphics sprayed everywhere!
In the two days I have left before school resumes, I want to have a way to get all this stuff back together in the EGAGRAPH and get actual Huffman stuff working.
Today, I wrote a little bit more code to finish extracting the other resources found in EGAGRAPH. After a little bit of testing, I determined that my temporary decompression fixed worked with the trivial Huffman dictionary. I also discovered some issues in the masked 8x8 tile reader, which were fixed by using the correct plane array. I also inverted the exported font colors to be more like KeenGraph. After polishing everything up, I wrote some test code to dump some of the resources to my test directory and it worked! Font characters, tiles, sprites and other graphics sprayed everywhere!
In the two days I have left before school resumes, I want to have a way to get all this stuff back together in the EGAGRAPH and get actual Huffman stuff working.
Thursday, January 2, 2014
FMod - Galaxy Graphics
Today and yesterday, I did some fiddling with Galaxy graphics for FMod.
At first, I continued my idea of having a complicated chunk management system with different manager types for the various chunk types and interrupt-like interface hooks. Managing requests and dependencies (for the table chunks) became a terrible mess very quickly, so I scrapped that idea and moved onto a simpler allocation table. Since I would have to have different manager sets for different episodes anyway, I created a class that stores the starting and ending chunk ID of each type of chunk and a generator that makes one of those for the requested episode.
As a precursor to actually reading things, I wrote the Huffman decompressor. It was fairly simply based on the example code on the ModdingWiki. With a quick BinaryTreeNode<T> class, it was loading dictionaries and decompressing non-byte-aligned data shortly.
With that done, and file format documentation at the ready, I began writing the readers for each range. It was surprisingly easy; the hardest part was the bit gymnastics to get the EGA plane data in the right order. The code could probably be cleaned up by using lambdas instead of copy/pasting similar code, but that can always be done later. Once I had pretty much everything in place, I actually ran a test. It failed miserably on essentially the first thing: the fonts. The height was something crazy like 64 and the offsets were way past the end of the chunk. Similar crazy number errors appeared in the tables, which only quietly bungled the data.
Apparently, there are still some things about Huffman compression that I still don't understand. I'm using Levellass's "trivial Huffman dictionary", which is said to work on Keengraph-exported graphics, which is what I'm trying to use. Somehow, the Huffman effect didn't get applied to some of the later bytes. FMod doesn't know which to reverse, though Keen uses them perfectly fine.
At first, I continued my idea of having a complicated chunk management system with different manager types for the various chunk types and interrupt-like interface hooks. Managing requests and dependencies (for the table chunks) became a terrible mess very quickly, so I scrapped that idea and moved onto a simpler allocation table. Since I would have to have different manager sets for different episodes anyway, I created a class that stores the starting and ending chunk ID of each type of chunk and a generator that makes one of those for the requested episode.
As a precursor to actually reading things, I wrote the Huffman decompressor. It was fairly simply based on the example code on the ModdingWiki. With a quick BinaryTreeNode<T> class, it was loading dictionaries and decompressing non-byte-aligned data shortly.
With that done, and file format documentation at the ready, I began writing the readers for each range. It was surprisingly easy; the hardest part was the bit gymnastics to get the EGA plane data in the right order. The code could probably be cleaned up by using lambdas instead of copy/pasting similar code, but that can always be done later. Once I had pretty much everything in place, I actually ran a test. It failed miserably on essentially the first thing: the fonts. The height was something crazy like 64 and the offsets were way past the end of the chunk. Similar crazy number errors appeared in the tables, which only quietly bungled the data.
Apparently, there are still some things about Huffman compression that I still don't understand. I'm using Levellass's "trivial Huffman dictionary", which is said to work on Keengraph-exported graphics, which is what I'm trying to use. Somehow, the Huffman effect didn't get applied to some of the later bytes. FMod doesn't know which to reverse, though Keen uses them perfectly fine.
Subscribe to:
Posts (Atom)