Called it Matchbox since I figured that's what you use to start fires: this is what you use to start modding a game in which you make fires. Might change the name later on though, we'll see.
Anyway, this is basically a GUI for Daxar's tools. Nothing to get excited about (yet), but it'll let you decompress and such without having to hassle with command line or dragging.
I haven't written a ReadMe yet but here's the lowdown: as soon as you open it for the first time, it'll let you choose your custom LI executable. Then you can click "Make Little Inferno Moddable" to strip the .exe. This is basically how I think I'll have it once the modding scene is ready, so I decided to implement this "setup" bit already.
Everything else is in the Tools menu. You can decompress a .pak, compress a .pak, and (if you've selected your custom LI) merge a .pak, which merges it into the .paks of your selected LI. For the moment you can only do one at a time, even though modManage.exe supports multiple.
Just so you know, error checking and such is minimal and there are probably a lot of loopholes right now that can seriously confuse you if you don't know what your doing. For example if you try to compress a .pak without having a filelist.txt, it won't give an error or anything. I'll be adding lots of checks in the future though.
I'd be glad to hear any feedback or suggestions or anything. Obviously once a mod format is decided I'll make it much more user-friendly and have stuff like a mod list (similar to GooTool).
@puggsoy: Excellent! I'll give that a shot later. Got some company coming over now. As for the .filelist.txt, my own readme states this is a temporary measure; I may change it in the future... Any suggestions for an alternative method, or do you think it's good?
@Allan: Fantastic, thanks! And Merry Christmas to you guys as well!
So I got that parsed out to XML, looks like http://pastebin.com/j38ruUVG so far, which I think is a fairly decent XML format. Any comments on that?
In any case, it's cool to see stuff coming together. I don't suppose there's any sort of string-based names for the item IDs or sound IDs? Maybe we can make our own list if not...
Cheerio,
Daxar
EDIT: Just checked out Matchbox, puggsoy, and it looks like a great start. For now, of course, there's not much you CAN do, but it looks great so far. Would it be possible to make some sort of progress bar when decompressing that gives an estimate of the time left, and says the current time elapsed, or something like that? I can format output from liDecompress or such to make that easier, if need be. I'm not sure how Air works for that sort of thing; might be too difficult, I dunno.
Only a couple bugs I found:
1. It's possible to decompress more than one .pak file at a time, since the "decompressing" window doesn't block the main window, which runs more than one copy of liDecompress at a time. Because of the way liDecompress writes stuff to the temp/ folder before decompressing, this crashes horribly. Partly my fault, I know, but if you could make the decompress window block the main window, it'd be a good tempfix for now.
2. You can strip the exe multiple times. I know you probably know this already, and I made strip.exe check and see if the exe has been stripped already before it does anything, so it's not an issue, but worth pointing out.
Overall, though, very nice. Good show ol' chum. Let me know if you want to do something, and the way my tools are set up is blocking you somehow, because I can (hopefully) easily change my tools' behaviors to accommodate whatever you need.
i think the sound ids should match the resource ids of sounds in the pak file for sound that have only 1 take. the sound ids for sounds with multiple takes are the hash of the file path of a take with the numeric part removed so the sound id for 'data\SFX\cheer.01.flac' and 'data\SFX\cheer.02.flac' is the hash of 'data\SFX\cheer.flac' (which happens to be what the file name for that sound would be if it only had 1 take.) i'll have to post the hashing process i do to generate the ids from strings at some point but if you want to reverse the hash of a sound id with multiple takes (just use the res id map for single take sounds) you could look up the sound id in the sound manifest, look up the resource id of its first take, look up the id string for the first take resource, then strip the take number out of the path and that will be the correct sound id string.
item ids are the hash of the item id strings which are just the name of the item's directory in the tree (so 'AlarmClock', 'BombshellBetty', 'BFReplacementPills', etc.) there's no direct way to look up the strings for those but you can tell which id goes with which string by looking at the path for the item's animation resource in the item manifest. every item has an animation that is in its own directory so the format of the animation resource id string will always be data\items\ITEMID\ITEMID.anim.xml.
1. This actually bugged me a lot. I can't find any built-in property for this, the closest I got to was keeping it on top (which I've done here). I'll try and see if I can find anything, but if not I'll either 1) code it myself or 2) disable all buttons and menus while it's working.
2. Yeah, I noticed that last night but couldn't be stuffed to do anything about it yet. I'll make it check the .exe's size, if it's less than the normal size it won't be strippable.
I was also thinking of a progress bar, but since that'll probably take a bit of work I decided to just have a "please wait" dialogue for the moment.
By the way, just so you know: the only ways I can know what your programs are doing are through output and error streams (stdout and stderr). I can also give input through the input stream (like giving "y" when confirming that I want to strip). So what I could do is analyse the output your program gives when it says it's decompressing file number X out of Y, but that would just say the amount done, and I couldn't give a proper estimation of time left (although I could say time elapsed). If it's possible for your program to give a time estimation then I could do it, but if that's not possible or too much work then that's no problem, I think just saying how many out of the total files are done. I could probably say the name of each file too.
In any case I'll try and make a progress bar window. I haven't made one before so it'll be interesting.
If I need you to change anything else with your programs I'll say, but for the time being they're pretty much perfect.
Also helps out sndmanifest.dat.xml as well; still working on figuring out a way for all this to work properly if you add new resources... Right now I think my code would crash terribly...
@puggsoy:
2. You could also check strip.exe's output; I think it spits out an error message if the executable has been stripped.
As for the progress bar, what I was thinking was some way to estimate the amount of time left from the elapsed time and the percentage of items decompressed. So, for example, if the program has been running for 5 seconds and has decompressed 256 of 3796 resources, the time left would be 5.0/(256/3796) = X/(3796/3796) -> X = ~74.14 seconds. Just a quick calculation; it'll likely be wrong, but all progress bars are always wrong for time estimation, so it's nothing new. If you need me to format the output of my program so it's easier for you to parse, let me know.
A "time remaining" bar is better suited for a GUI than a commandline tool, which is why I didn't do it. I think there is a way to reformat text already drawn to cout-- maybe I'll look into that...
Seems to me that Adobe AIR is well-suited to being a GUI frontend for commandline tools, then. Cool!
Well, if I were to check strip.exe's error stream to see if the .exe is strippable, I'd have to run it every time I wanted to check (i.e. whenever you change the chosen location), and if it were strippable then it would already have started to process.
Although, I will arrange an error message just in case it is run on an unstrippable .exe, so thanks for telling me about it.
I'll be able to do the estimation myself then. As you said programs are rarely accurate, I mean even Windows can mess it up terribly so it's really a minor issue.
As for parsing output, what you've already got is really enough. I can just detect the first and second numbers on each line, which are how far we are out of how much (which I can then use for calculations and whatever). Actually, I only need to get the second one the first time since that's always the same. And if I want I can also get the currently extracting file name as the end of the line after the first colon and space.
So yeah, no need to modify it. The new executable's output is actually less usable, since it appears to spit massive chunks instead of one line for each decompression. Obviously to display progress I need to know the progress more regularly, so yeah, the current output method is pretty much perfect for now. If I have any problems I'll say but for now I think it's all right.
Also yeah, AIR is pretty good for this. It's not really what it's made for (it's actually just Flash in desktop form), but it can do it well and I enjoy doing it so that's cool.
Oh, hmm, might be because I wasn't flushing cout...
Try this one: EDIT: See below
Thing is, I'm tired of it spitting out so much on the command line, so I'm trying to write over stuff already written. I hoping it's possible to do it in such a way that AIR applications can still read it, but if not I can always supply you with a slightly different version each release. I'm hoping this one will work, though.
Only drawback is there's a limited amount of space to write stuff on before it wraps to the next line, and then my writing-over code completely fails, so I can't write filenames. I reckon it's personal preference, but if you like it how it is better I can leave it.
EDIT: Scratch that, why not both? https://dl.dropbox.com/u/31816885/liDecompress-progress.exe
Run with the commandline argument "--overwrite-progress" to have the previous output overwrite the old progress, or without that commandline switch to spit out the regular output with full filename and all. Either one should (hopefully) work fine with your code (just with slightly different parsing), but the way I've been doing it until now will write the filenames, too, which is more helpful anyway, just clutters up the commandline a lot. But in your case, you're not showing the terminal window anyway, so it should be fine.
tl;dr Ignore me. It works like normal.
Also note this will fail to decompress a file with a hyphen in the beginning of the filename. But if anyone ever puts hyphens in the beginning of filenames, they had it coming.
Aha, right, I see now. Previously I hadn't actually run it from the actual command line, I just wrote a batch file and made it write the output to a text file. I ran it from the actual console this time though, and I can see what you mean: it overwrites the previous progress as it goes. Obviously writing it in a text file (or tracing it, which is what I'm currently doing in Matchbox) doesn't reflect this, but it is helpful when using the console.
Well, the new .exe works with both methods: i.e. the old way works, but if I add the overwrite argument I also get each line at a time. I'll use the old one though since, as you said, I'm not showing all the output so clutter isn't a problem. Plus I also have the filenames, which may be helpful somehow.
As for hyphens: you're right, I mean who does that?
EDIT: By the way, just figured out how to keep windows on top active (i.e. not allowing you to access the main window while a dialogue is up). I still had to do it manually (no built-in feature), but it was much simpler than I thought, and works perfectly.
EDIT2: OK, so I've managed to parse the compression and decompression output and make a loading window with a bar and everything. There's no time estimation yet but I'll get to that.
However, there's no progress output when merging, nor when stripping the .exe. (Can't recall what happens when simply pulling the .paks though). So yeah, do you want to add progress for those, or should it just say "please wait"?
Excellent! Glad that's working for you. Kudos and all that.
I haven't looked at a progress output or anything like that for merging yet. I was thinking today about how I want to overhaul the program anyway, and I'll probably add that sort of support in soon. Maybe along with some optimizations to speed the program up. I'd like to make it into a more full-fledged mod-managing program, with some sort of way to add and remove mods from a queue type of thing, like how GooTool manages it. Dunno yet. But yeah, I'll be working on it.
As for stripping, the problem there is that it's really just two huge file operations. First, I load the "Little Inferno.exe" resources into memory, which is a Windows API call that I can't split up into good progress-bar-sized chunks. Then, I save it all to disk (again a single Win32 function call). This second step is likely what takes all the time in the program (though the first may take a fair amount of time as well- I should probably test this). I could theoretically split the file-writing up into chunks, and spit out some kind of output at intervals, though I'm not sure how much that would gain. Do you think the time delay is enough of an issue? I can do that if need be.
EDIT: Just changed the program to spit out some debug output; seems that indeed the LoadLibrary() call to load the resources from the executable and the WriteFile() to save resource.pak to disk are the two major time-consumers for this program. The first is taking about 2.3 seconds, and can't be split into chunks for me to spit progress data out about. The second is taking about 2.6 seconds, and can. Writing embed.pak and frontend.pak to disk is taking somewhere between 1-10 milliseconds apiece, so they're not really any concern. If you want me to split the saving-resource.pak part of the program into progress-sized chunks, I can. Your call.
Oh looks nice! Suggestion: Make the GUI version localizable, that will help much!
Oh it seems I forgot to tell you what those Chinese means. They are just the common program-crashed window in Windows 7. It is meaningless. No useful information provided.
@Mygod: When the GUI's mostly done I'll look into adding other languages. The only language I know other than English is Dutch though, and Google Translate will mess up grammar, so I'll probably need peoples' help with that (you can do Chinese if you want). But yeah for now I'll just try and get the main stuff done.
@Daxar: About the mod installing: I was planning to have a queue system similar to GooTool, which my program could probably manage. It'd basically take note of all mods that are in the queue, and every time you save it copies the original paks, then installs the mods in order of the queue (which you can customize of course). Just like GooTool, actually.
Unless you have a more efficient way of course, by all means do that. Just so you know you won't have to make your program keep track of a queue, if you don't want to.
The ouput-spitting version of the program right now isn't usable, but thanks anyway. If it's not too much trouble, giving some progress in the second step would be nice. The time it takes isn't really huge (much faster than decompressing resource.pak, for sure), but a progress bar might still be of some use. At the very least it'll let people know that something's happening (rather than a static "please wait").
If you do that, should I try estimating time on it as well?
@Mygod: Hmm, I can tell from your screenshot that you're running it in a folder with non-ASCII characters. I'll mess around and see if I can replicate the bug. Could be an error in my code. Strange that it would crash midway through, though. Anyhoo, thanks!
@puggsoy: Ok, if you want to make the queue in your GUI, that works too. The only potential issue is the possibility of wasting a lot of time if you call modManage multiple times with one .pak file each time, rather than once with all of them. So don't do that. Should be a fairly simple two-step process: Copy the original .pak files over, call modManage.exe [pakfile1]...[pakfilen]. The modManage program already merges the .pak files in order, so that should work fine. Sorry if you know all this already; I'm just kinda thinking out loud. Should work great, then.
Another thing, in case you didn't think of it: GooTool saves all the .goomod files to a central location when they're installed, in case the original files are deleted. It's probably a good idea to do that here, too.
And I'll get cracking on strip.exe progress. I think you don't need a time estimate on that one; a progress bar alone should be fine.
picking up with more item info - each item in the manifest has a binDataOffsetBytes field that is an offset in bytes into the itemsBinDataBytes section of the mainfest file where the rest of the data for the item is. once you find that data for the item, it's kind of like finding a whole new resource embedded inside the manifest resource (that's because each item was a separate resource right up until the last minute when i combined them all to help loading performance.) so, the binary data for each item starts with a header just like a normal resource would:
struct
{
BinHdrPtr skels;
BinHdrPtr joints;
BinHdrPtr bones;
BinHdrPtr boneShapes;
BinHdrPtr boneParts;
BinHdrPtr bonePartTreeVals;
BinHdrPtr rgnCells;
BinHdrPtr stringTableBytes;
BinHdrPtr burnGridUsedDataBytes;
u32 itemId;
i32 itemIdStrId;
u32 animResId;
vec2 iconAnimBoundsMin;
vec2 iconAnimBoundsMax;
BinLocStrKey name;
i32 costCoins;
i32 costStamps;
i32 valueCoins;
i32 valueStamps;
BinLocStrKey desc;
i32 unlisted;
i32 popsCoins;
i32 moneyItem;
f32 animThreshold;
f32 motorThreshold;
i32 absPosition;
f32 scaleVariance;
i32 quantity;
i32 shipTimeSec;
u32 initialBurnExportId;
i32 initialBurnPerGroup;
i32 mouseGrabbable;
f32 illuminate;
i32 enableHFlip;
i32 floorWallShadow;
i32 splitJumpLastFrame;
f32 purchaseCooldown;
i32 allowDirectionalLight;
u32 mouseGrabSoundResId;
i32 instantEffects;
i32 freezeOnCollide;
i32 enableFreezePostAnim;
u32 uniqueIgniteSoundResId;
i32 collideItems;
i32 collideEnvironment;
i32 orbitalGravity;
i32 allowExplodeStreaks;
i32 burnSlowsAnim;
i32 plagueOnCollide;
u32 spawnLimitBurnExportId;
u32 instAshSoundResId;
i32 canGetPlague;
i32 instAshDoesSplat;
f32 modXAmpMin;
f32 modXAmpMax;
f32 modXFreqMin;
f32 modXFreqMax;
f32 modXPhaseMin;
f32 modXPhaseMax;
f32 modXSpeedMin;
f32 modXSpeedMax;
f32 modYAmpMin;
f32 modYAmpMax;
f32 modYFreqMin;
f32 modYFreqMax;
f32 modYPhaseMin;
f32 modYPhaseMax;
f32 modYSpeedMin;
f32 modYSpeedMax;
};
it's going to take me a while to document all of the fields here so i'll do an overview of some stuff for now. the skeleton/joint/bone sections are related to the different ragdolls that you can get for each item. most items only have 1 but some have slight variations which would show up in the data as different skeletons, and the joints and bones define how to build the physical bodies in box2d for each skeleton. itemIdStrId is a key for looking up the items id as a string in the local string table for the item. iconAnimBounds controls how the item's catalog animation gets framed in the little window in the catalog when you view that item. (a vec2 is just a pair of 32bit floats) name and desc are the text you see in the catalog for the item (a BinLocStrKey is a u32 and an i32, the i32 being the key for the string table) cost and value stuff is related to how much things cost in the catalog and how much you get for them when you burn them. unlisted was just used for development and doesn't play any role in the shipping game. the "threshold" ones refer to things happening at different times in the burn cycle of the item. -1.0 is before you burn the item, 0.0 is when it catches on fire and larger values like 1.0 or 2.0 are after the item has been burning for a while. shipTimeSec controls how long the item takes to ship in case you want to mod everything down to 1 second : ) the mod X and Y stuff at the bottom is a rarely used bunch of data that causes the engine to give the item some non-physical movement (used only for things like bubbles and butterflies that fly up and away in a sin wave looking pattern.)
there's a bunch of stuff here (i haven't even gotten into the specifics of each section of data besides just the header) and i'm out of time again since i'm still on vacation for the holidays. i'll follow up again later with more info.
A lot of these seem to be pretty self-explanatory values, such as costCoins, quantity, etc. But then again, there's a fair amount of terse-looking stuff too. But rather than stressing over every detail for what every field means, how about just documenting the file format stuff, so we can get all the values out and play with them? The stuff we don't know we can probably figure out by tweaking values and seeing the effect on the game. I don't want you to feel stressed with having to explain in detail what every field of every structure does. If we have questions about anything in particular, we'll ask. :)
Thanks, Allan! And enjoy your vacation. Tell your family we say hi. :)
Yeah when I move it into an English-only directory, the problem fixed! Thanks!
By the way about the itemmanifest.dat.xml, I think we'd better use attributes instead of elements for itemId, itemIdStrId, animResId, costCoins, etc.
Yeah, I was thinking about attributes, but decided on separate elements for readability. I'll toy with it, though. I'll probably split the sndmanifest.dat.xml up a bit for readability... We'll see.
Glad to hear an ASCII-only directory works. My code should theoretically work with UTF-8 fine, but I imagine I messed something up someplace. I'll have to look into it.
I personally think this is a bit less readable, but I'm open to suggestions/questions/comments/whatever. I don't think there's a way with tinyxml2 to add whitespace to make that any more readable; that'd probably have to be done by hand.
It IS a lot of data, but I think I can make reasonable defaults for some of the values, and cut down on the amount of XML that way.
I also tried to replicate the non-ASCII-directory bug, and I think because I'm using an English version of Windows, it's not feeding the proper UTF-8 string to the program, which is quite annoying. I don't think I can test this bug because of this issue, so for the time being just make sure your .pak files are in ASCII-only folders if you're clicking-and-dragging pakfiles into the executable. :/
EDIT: Totally forgot about seeing if I could use Puggsoy's program to test this...
EDIT2: Nope, that one seems to work fine. But then again, it can only decompress one .pak file at a time, and you only had trouble with multiple .pak files...
I'll allow you to select multiple files for decompressing and merging in the next release. Hopefully that should be tomorrow.
By the way, I was checking out compressing and found a couple of bugs. First of all, if it tries to compress a file that doesn't exist (i.e. you write a nonexistent file in the filelist.txt), liCompress crashes. Nothing special, just "liDecompress has stopped working".
Secondly, there are issues when one of the files you want to compress into the .pak has a space in the filename. It appears to compress OK, but when I decompress it again the file with the space doesn't appear, and the filelist.txt generated has a blank line instead of the file path. Obviously it's generally better not to have a space at all, but some people might do it so I thought to check it.
@momo & movildima: Awesome! I'll most likely have the languages in XML files, which you'll have to edit by hand. Is that OK?
@puggsoy: re/ the nonexistent file: Yes, I didn't begin to bother testing that. The whole .filelist.txt thing is still just a placeholder for now until I can think of something better. I haven't yet. I especially haven't done anything to finalize the format with error-checking and stuff. In any case, even if it didn't crash, it'd stop later, since it checks the number of items in the .filelist.txt and compares this against how many files it was able to compress, and halts if the number was different. I'm still trying to figure out the best way to do all this, and I'll address this once I've decided. But thanks for finding that; I'll be sure to keep it in mind.
re/ the filename space error: This actually has absolutely nothing to do with spaces in filenames and everything to do with a file not being in the original residmap.dat. Note this line in my README.md (line 84, where I'm talking about sound resources):
Quote:
You can only modify existing resources; you cannot add new ones, as there is currently no way to get a resource ID from a filename. If you attempt to add a new .ogg file as a take somewhere, you'll end up with an undefined resource ID, which will likely cause a crash.
(come to think of it, I probably should have made this a lot clearer in the readme; put it in big bold letters someplace. I'll do that on next release)
So, if you're working on a GUI so people can select files to compress into .pak files, please stop now. It won't work at all. Right now we don't know how resource ID's are hashed from filenames, and until we do, there's no good way to get an ID from a filename if we don't have a filename-to-ID mapping already (i.e. unless it's in the original residmap.dat). For my code, the default ID generated for an unrecognized filename is undefined, and the default filename for an undefined ID is a blank string, hence the empty line in the .filelist.txt. Once we get the hashing algorithm, I'll make this all work, but for now it completely messes everything up. Long story short, If a file isn't in the original .pak files, it can't be compressed. Well, it can, but it ends up with a meaningless resource ID and can't be decompressed. For now. I probably should have added a test for that, too, but hey, alpha-stage modding tools and all that.
@all: I decided to find out if some of the item data (from the structure Allan just mentioned) could have default values, and so I made some debug code that spit out all the values for all the items. Seems to me I can simplify this XML down quite a lot in order to conserve space (and sanity, and all that kind of good stuff).
Here's my output: http://pastebin.com/Hwe2Bmtd
As you can see, a lot of the items have the same values for a lot of these. "costStamps," for example, is 0 for every item, and can be optimized out completely. Not everything can have a default value, of course, but I'm going to be going through this and figuring out what values I can make optional in the XML. I'll be sure to document it all, of course.
I think the game quotes everything with the ID so the hashing method is not important at all. You can create one yourself or let user enter IDs themselves.
You can try to extend the filelist.txt and make it hold another field called ID. (use : to separate maybe)
While running strip.exe or pullpakfiles.exe, generate a default filelist.txt. While running liDecompress.exe make it read the filelist.txt so that it knows what ID should be used.
Here are my thoughts, but I need a confirmation. What do you think, @allan?
While it is indeed possible for users to create an ID by hand, the chances can become quite high that IDs overlap, and the consequences for something like that could potentially be disastrous. This is why I'm holding out for getting the actual filename hashing algorithm, so we don't end up with a lot of random IDs that are potential conflicts floating around.
mygod is correct that ids could be anything in theory as long as there are no collisions and daxar is also correct that you probably want to derive them from file paths to make those collisions unlikely. here is the hash that we use (which itself is by no means a robust hash but surprisingly we never encountered any problems with collisions using it)
@momo & movildima: Heh, well, actually I'm gonna hold translation off until it's almost complete. That is, at least until we have a finalised mod format, and my program has a mod queue and all that. Sorry to get your hopes up, mostly I just wanted to check if XML was good with you guys for when the time comes (then I can sort of plan ahead). Good to know XML works though, so thanks.
@Daxar: No, I wasn't implementing any real interface for choosing files to compress yet. I was just running some error tests for compression (for example I made an alert dialogue pop up if my program can't find a corresponding filelist.txt). I'll definitely wait until you're done with compression, maybe even until we get a finalised mod format, before I allow people to choose files to compress. But for the time being the user just specifies a .pak, like your program.
I wouldn't like to drag all exe files to my game directory because that will mess up the directory. So I created a bat file like this. I think this will be a good idea. : P @echo off
title liTools Command Line
path %PATH%;"{liTools Path}";"{liTools Path}\util"
cmd
@Allan: Sweet, thanks! I'll get cracking on that, then.
@Mygod: Good idea for the batch file to add the tools to your PATH. I hope that works out good for you!
As for the images, after poking around some and fooling with GIMP a lot, I think I've figured it out. It's looking to me like Little Inferno is expecting any pixel that has an alpha = 0 to have RGB = 0,0,0 as well. I set GIMP to only modify the alpha channel of your image, and filled in the transparent areas without modifying the RGB values, and sure enough, here's what I got: http://i48.tinypic.com/p526w.png
I double-checked by compressing your image then decompressing it, but the same image gets spit back out, so I don't think it's my code. In any case, this is kind of a strange limitation imo. I've toyed around with low-alpha images, too, and it seems to behave in a strange manner with those, as well. For example, the "glassbulb" item has a low-alpha brown image that seems to display absolutely fine. However, if I try to make one of the chimneys in the main screen low-alpha, it treats it as if the alpha stays 255. Very strange to say the least.
You can just install this over the old version and it'll update it, no need to uninstall.
Here's what's changed:
-You can now decompress multiple .paks at once.
-You can now merge multiple .paks at once.
-You can now pull .paks from an .exe without stripping it.
-Added a bunch of checks and alerts (for example it won't let you try and strip an already-stripped exe). There's probably still a couple of things that I haven't checked for but I've done the most obvious stuff.
-Added a dialogue with a loading bar to show the progress of decompressing and compressing. For now it just says how many files out of the total it's done, but I'll make an algorithm for time estimation in the next release.
That's about it. I also fixed a small problem I had with the buttons being scaled down, but it's not a huge change and you probably won't notice it.
Note that unfortunately, when you select multiple files they're forced to be fed to the program in alphabetical order, even if you select them in a different order. This isn't really a problem when decompressing .paks, but it's not ideal when merging. Obviously once I get the whole queue system sorted you can drag them around and organise them by priority, but for now if you want to install them in a non-alphabetical order you should do them one-by-one.
By the way Daxar, I noticed that modManage doesn't give any usable progress output. Are you planning to add it or isn't it possible? No hurry of course (figuring out those IDs is probably more important), but just thought to ask.
@puggsoy: I'm planning on refactoring the modManage program sometime anyway, so for now there's no progress output. I'll put that on my TODO list though.
EDIT: Just testing out the new version of Matchbox. It works beautifully and looks brilliant. Great job!
@Mygod: Yep, that's what I did for now. Though it doesn't work for low-alpha portions of an image. It may have to do with normal images, and images without normal images don't work for low-alpha... I dunno. It's strange, to say the least.
@all: The new XML format with default values looks like this: http://pastebin.com/iH5qxjPT which looks fine with XML attributes rather than elements, so I stand corrected, Mygod.
Unless there are any objections, I'll stick with this format for the itemmanifest (until more is explained, of course. Maybe I'll end up splitting stuff into separate XML files like how it all was originally; I'll see). I parsed through all the values in itemmanifest.dat for all the fields of all the structures, and if any value for a field was used more than 50% of the time, I made that the default, which seems to work out great. The only field that was borderline was enableHFlip, which had the value 1 a total of 98 times out of 192 total objects. The rest of them were in the 180's or so.
Paste of the default values I settled on: http://pastebin.com/BSY4PpMy
Onward ho to adding support for new resources.
image data for the game is stored with premultiplied alpha. if you want to get back to the approximate values that would go into a PNG, you need to divide the color channels by the alpha channel when you extract the images from the game and multiply when you put them back in.
Ah, that would do it. Thanks, Allan! Works like a charm now.
EDIT: Ok, wow. I had no idea that this would make such a tremendous difference. I changed my modManage program to pull all the data from the .pak files and store it in RAM instead of in files in temp/ and tempmod/ before merging and recompressing. It's running somewhere between 7x and 20x faster compared to the original version, which is amazing. Time to refactor liDecompress and liCompress, I think.
@puggsoy: modManage is taking around 0.2 seconds now for 4 mods that each change one file in each original .pak file, so I might hold on a progress bar. XD
EDIT2: Done optimizing liDecompress to alleviate the need for a temp/ subfolder. I think it's somewhere around 40-60 seconds faster now. Good call, Mygod.
Wow. The effect is better than I expected. Nice job.
xar wrote:
pull all the data from the .pak files and store it in RAM
Do you mean your program store all the data in RAM? That will consume large amounts of memory. When it runs on a computer with low memory, it will produce large amounts of hard errors which will decrease the speed a lot. You can try to read one file a time (requires thread synchronization), decode it and write it to the disk. If not, that's great.
Anyway, I'm looking forward to your new version. : D
(Mygod and I chatted on IRC, and I answered his questions there)
Still gotta track down a new strange bug, but I think I'll just optimize liCompress in the same way I did liDecompress and maybe change the itemmanifest.xml format a bit before releasing. I'm going for more changes per release and less-often releases, hence the delay. Of course, you guys can always compile from source or come on IRC (irc.esper.net, channel #world-of-goo) and nag me if you want bleeding-edge builds. ; )
EDIT: Version 0.3.5 changelog:
* vdata/itemmanifest.dat is partially parsed to vdata/itemmanifest.dat.xml
* itemmanifest.dat.xml is NOT parsed back; it's incomplete
* Optimized liCompress and liDecompress to use RAM instead of hard disk, which means they should run ~40 seconds faster for resource.pak, just using more RAM in the process
* Commandline arguments for liDecompress and liCompress; see readme for details
* Optimized modManage program to merge mods within seconds
* residmap.dat now parsed to/from XML
* Images now properly converted to/from premultiplied alpha
* Initial Unicode support, which is still a bit buggy, but should help if you don't have an English localization of Windows
* Various bugfixes
EDIT2: Fixed a major bug that would botch decompression of resource.pak and updated the links above accordingly. As a bonus, you can now specify compression level for liCompress via the commandline switch --level=n, where n can be 0 (no compression), 1 (fast compression) through 9 (full compression, slow but smaller filesize). All files are now compressed by default, and if the resulting compressed file is larger than the original, it is discarded and the uncompressed file is used. After running a couple tests, at level=9 my compressed resource.pak is about 2.1 MB smaller than the original, which takes an extra 30 seconds or so. At default compression level, it's still about 1 MB smaller. It's fun to toy around with, at least.
Looks like now I'll get cracking on adding new resources into pakfiles; I was thinking of a small residmap.dat-type file in each .pak mod so that the files can still be decompressed easily.
By the way, @allan, if you have time, I would like to know the format of following files as well because I would like to make a non official translation for Little Inferno. (because kyle said the game cannot currently support Chinese and these will also help @Daxar's project sometime) : P
TwCen.font.xml
dialogmanifest.dat
letterdb.dat
@Daxar: Compression levels look interesting. Can the game still use the .paks if they're compressed than any level other than the default? Because if so I might add an option for it in Matchbox.
@puggsoy: Just tested it and it works great. I think most compression schemes work this way; they don't care about compression level when decompressing, since it's just an optimized compression that takes longer.
@Haapavuo: Hi there, and thanks for the offer! If you have any ideas for what should be done, feel free to fork my git repo at https://github.com/meh2481/liTools and go to town with hacking on it. For the most part, I'm just basically madly coding tons of stuff whenever Allan describes something new, and I seem to be keeping up for the most part. Of course, once school starts back up again in a couple of weeks, all bets are off. You could also work with Mygod's codebase; it's a little bit behind, but may be organized better than mine. : )
@Haapavuo: Awesome! The more languages the better.
@Daxar: That's cool, just wondering. In that case I'll make a little dialogue to choose compression level (which will probably cause me to also create some sort of menu (maybe "Preferences" or something) to let you choose if it pops up every time you compress).
I am now doing some experiments with itemmanifest.dat file. I found out how item prices, names and descriptions are stored. Seems very interesting... I will update this post soon with a screenshot!
@Daxar: Thanks for the link! I'll look into that!
@puggsoy: Could you release the future versions of Matchbox rather zipped than with an installer? I'm pretty sure most people will like it that way. Thanks!
EDIT: OK! I've now struggled with the bits for a while and here are some of the results:
Legend:
RED: Item price in catalog
BLUE: Item cash drop when burn
MAGENTA: Item delivery time
GREEN: Item name
PINK: Item description
BROWN: Crash when edited (item ID maybe)?
BLACK: Didn't change anything noticeable.
GRAY: Interesting...
Item names (and all texts) are always cut with 2 zeroes (0x00) so it should be pretty easy to write a parser for these dat files (or at least this one).
I'm still going to do some more research and updating the second image...
Unfortunately it actually compiles into an installer, apparently it's essential. Personally I'd also prefer to release it as an archive, at least these beta versions.
I'll snoop around and see if it's possible to put it in a .zip, but if not it'll need to stay like this. I don't find it too bad, it makes installing new versions easy and I'd probably put it in an installer once it's complete anyway (since it's more user-friendly than .zips, and user-friendliness is the whole point of the GUI).
@Haapavuo: While I appreciate the effort, and you have found some cool stuff there, check the posts above. Allan is already describing the itemmanifest.dat format in great detail, and my liDecompress program already spits out a lot of that data in XML form. While finding new stuff is always cool, and reverse-engineering via a hex editor is good, perhaps you should do so on a file we don't know the format to already. ; )
@puggsoy: Interesting that it only spits out installers, but imo it's not that big of a deal. What I'd personally gripe about is that you have it hosted on some ad-ridden site like Mediafire. Have you looked into a hosting service you can direct-link to, like Dropbox?
Also seconded for having an installer rather than a .zip; makes it more easy-to-use and professional in my opinion. Have you looked into an icon for it yet? One of our awesome artists (or even me) could make you one.
@Haapavuo
Hello and welcome to this site! : D
If you want, here is my full source: https://dl.dropbox.com/u/94637484/LIRMFullSource.7z
The packer module is still buggy. Audio encoder (converted from @allan's C++ source by hand) often throws an AccessViolationException. Though my program will ignore the error and retry immediately, the output file will be probably messed up. (if luckily no exceptions are thrown, the program will works fine) You can decrease the core number to 1 to decrease the probability of the exception. If you wouldn't like to edit these ugly codes, I hope it will be of some value one day.
P.S. These code wasn't changed since 12/22/2012 so it doesn't support some new features that is included in liTools. I spent a whole day to try to solve that bug but failed. And after that I decided to use Daxar's liTools.
@Daxar: Sorry about that: with AdBlock it's always hard to remember which sites have ads and which don't. The thing is that someone recommended Mediafire a while back, and since I didn't see any problem with it I just chose it. Back then I didn't host much (just zips for video game sound effects), but now I'm doing much more and I can see how it'd be a problem.
I'll check out Dropbox, since it can do direct links that's probably best. I'll have to take the time to move all my other things over as well though (since I don't want my files spread over separate hosts).
I'm pretty rubbish at icons (or computer art in general: hoping to remedy that eventually), so I don't think I'll make my own. It's not so high on my priority list yet so I haven't really looked around or asked anyone, but if you or anybody here is willing, feel free to whip something up. Something like a Little Inferno-themed matchbox (heh) would be cool.
EDIT: Relinked my previous posts, they're now direct links from Dropbox.
Oh crap, how did I miss all that... I used the previous version of liTools which didn't have the itemsmanifest.dat.xml part. Got to try it out now! You've done more than I thought!
@Mygod: Thanks for your source. I'll look into that!
EDIT: Ahaa, I see. It does yet extract only some of the data, not yet the item names and descriptions etc. I'll see if I can crack some other files we don't know yet.
Catalog
Combos
Forum
Get the Game
News
OK, version 0.1 of the GUI is done. Sorry this is so late, I meant to do it earlier today but I didn't get the chance.
Matchbox 0.1
Called it Matchbox since I figured that's what you use to start fires: this is what you use to start modding a game in which you make fires. Might change the name later on though, we'll see.
Anyway, this is basically a GUI for Daxar's tools. Nothing to get excited about (yet), but it'll let you decompress and such without having to hassle with command line or dragging.
I haven't written a ReadMe yet but here's the lowdown: as soon as you open it for the first time, it'll let you choose your custom LI executable. Then you can click "Make Little Inferno Moddable" to strip the .exe. This is basically how I think I'll have it once the modding scene is ready, so I decided to implement this "setup" bit already.
Everything else is in the Tools menu. You can decompress a .pak, compress a .pak, and (if you've selected your custom LI) merge a .pak, which merges it into the .paks of your selected LI. For the moment you can only do one at a time, even though modManage.exe supports multiple.
Just so you know, error checking and such is minimal and there are probably a lot of loopholes right now that can seriously confuse you if you don't know what your doing. For example if you try to compress a .pak without having a filelist.txt, it won't give an error or anything. I'll be adding lots of checks in the future though.
I'd be glad to hear any feedback or suggestions or anything. Obviously once a mod format is decided I'll make it much more user-friendly and have stuff like a mod list (similar to GooTool).
@puggsoy: Excellent! I'll give that a shot later. Got some company coming over now. As for the .filelist.txt, my own readme states this is a temporary measure; I may change it in the future... Any suggestions for an alternative method, or do you think it's good?
@Allan: Fantastic, thanks! And Merry Christmas to you guys as well!
So I got that parsed out to XML, looks like http://pastebin.com/j38ruUVG so far, which I think is a fairly decent XML format. Any comments on that?
In any case, it's cool to see stuff coming together. I don't suppose there's any sort of string-based names for the item IDs or sound IDs? Maybe we can make our own list if not...
Cheerio,
Daxar
EDIT: Just checked out Matchbox, puggsoy, and it looks like a great start. For now, of course, there's not much you CAN do, but it looks great so far. Would it be possible to make some sort of progress bar when decompressing that gives an estimate of the time left, and says the current time elapsed, or something like that? I can format output from liDecompress or such to make that easier, if need be. I'm not sure how Air works for that sort of thing; might be too difficult, I dunno.
Only a couple bugs I found:
1. It's possible to decompress more than one .pak file at a time, since the "decompressing" window doesn't block the main window, which runs more than one copy of liDecompress at a time. Because of the way liDecompress writes stuff to the temp/ folder before decompressing, this crashes horribly. Partly my fault, I know, but if you could make the decompress window block the main window, it'd be a good tempfix for now.
2. You can strip the exe multiple times. I know you probably know this already, and I made strip.exe check and see if the exe has been stripped already before it does anything, so it's not an issue, but worth pointing out.
Overall, though, very nice. Good show ol' chum. Let me know if you want to do something, and the way my tools are set up is blocking you somehow, because I can (hopefully) easily change my tools' behaviors to accommodate whatever you need.
And I think "Matchbox" is a fantastic name, btw.
i think the sound ids should match the resource ids of sounds in the pak file for sound that have only 1 take. the sound ids for sounds with multiple takes are the hash of the file path of a take with the numeric part removed so the sound id for 'data\SFX\cheer.01.flac' and 'data\SFX\cheer.02.flac' is the hash of 'data\SFX\cheer.flac' (which happens to be what the file name for that sound would be if it only had 1 take.) i'll have to post the hashing process i do to generate the ids from strings at some point but if you want to reverse the hash of a sound id with multiple takes (just use the res id map for single take sounds) you could look up the sound id in the sound manifest, look up the resource id of its first take, look up the id string for the first take resource, then strip the take number out of the path and that will be the correct sound id string.
item ids are the hash of the item id strings which are just the name of the item's directory in the tree (so 'AlarmClock', 'BombshellBetty', 'BFReplacementPills', etc.) there's no direct way to look up the strings for those but you can tell which id goes with which string by looking at the path for the item's animation resource in the item manifest. every item has an animation that is in its own directory so the format of the animation resource id string will always be data\items\ITEMID\ITEMID.anim.xml.
Nice. I have no time (and also nothing to do) these days so I will watch. : P
@Daxar: Thanks! About those bugs:
1. This actually bugged me a lot. I can't find any built-in property for this, the closest I got to was keeping it on top (which I've done here). I'll try and see if I can find anything, but if not I'll either 1) code it myself or 2) disable all buttons and menus while it's working.
2. Yeah, I noticed that last night but couldn't be stuffed to do anything about it yet. I'll make it check the .exe's size, if it's less than the normal size it won't be strippable.
I was also thinking of a progress bar, but since that'll probably take a bit of work I decided to just have a "please wait" dialogue for the moment.
By the way, just so you know: the only ways I can know what your programs are doing are through output and error streams (stdout and stderr). I can also give input through the input stream (like giving "y" when confirming that I want to strip). So what I could do is analyse the output your program gives when it says it's decompressing file number X out of Y, but that would just say the amount done, and I couldn't give a proper estimation of time left (although I could say time elapsed). If it's possible for your program to give a time estimation then I could do it, but if that's not possible or too much work then that's no problem, I think just saying how many out of the total files are done. I could probably say the name of each file too.
In any case I'll try and make a progress bar window. I haven't made one before so it'll be interesting.
If I need you to change anything else with your programs I'll say, but for the time being they're pretty much perfect.
@Allan: Fantastic, thanks! Eagerly awaiting the description of hashing.
In the meantime, that helps make this XML a lot more readable:
http://pastebin.com/Kk8bgFAY
Also helps out sndmanifest.dat.xml as well; still working on figuring out a way for all this to work properly if you add new resources... Right now I think my code would crash terribly...
@puggsoy:
2. You could also check strip.exe's output; I think it spits out an error message if the executable has been stripped.
As for the progress bar, what I was thinking was some way to estimate the amount of time left from the elapsed time and the percentage of items decompressed. So, for example, if the program has been running for 5 seconds and has decompressed 256 of 3796 resources, the time left would be 5.0/(256/3796) = X/(3796/3796) -> X = ~74.14 seconds. Just a quick calculation; it'll likely be wrong, but all progress bars are always wrong for time estimation, so it's nothing new. If you need me to format the output of my program so it's easier for you to parse, let me know.
A "time remaining" bar is better suited for a GUI than a commandline tool, which is why I didn't do it. I think there is a way to reformat text already drawn to cout-- maybe I'll look into that...
Seems to me that Adobe AIR is well-suited to being a GUI frontend for commandline tools, then. Cool!
EDIT: @puggsoy: Does this executable have parseable output as well? I may be changing my output to be more like this on next release: https://dl.dropbox.com/u/31816885/liDecompress-progress.exe
Well, if I were to check strip.exe's error stream to see if the .exe is strippable, I'd have to run it every time I wanted to check (i.e. whenever you change the chosen location), and if it were strippable then it would already have started to process.
Although, I will arrange an error message just in case it is run on an unstrippable .exe, so thanks for telling me about it.
I'll be able to do the estimation myself then. As you said programs are rarely accurate, I mean even Windows can mess it up terribly so it's really a minor issue.
As for parsing output, what you've already got is really enough. I can just detect the first and second numbers on each line, which are how far we are out of how much (which I can then use for calculations and whatever). Actually, I only need to get the second one the first time since that's always the same. And if I want I can also get the currently extracting file name as the end of the line after the first colon and space.
So yeah, no need to modify it. The new executable's output is actually less usable, since it appears to spit massive chunks instead of one line for each decompression. Obviously to display progress I need to know the progress more regularly, so yeah, the current output method is pretty much perfect for now. If I have any problems I'll say but for now I think it's all right.
Also yeah, AIR is pretty good for this. It's not really what it's made for (it's actually just Flash in desktop form), but it can do it well and I enjoy doing it so that's cool.
Oh, hmm, might be because I wasn't flushing cout...
Try this one: EDIT: See below
Thing is, I'm tired of it spitting out so much on the command line, so I'm trying to write over stuff already written. I hoping it's possible to do it in such a way that AIR applications can still read it, but if not I can always supply you with a slightly different version each release. I'm hoping this one will work, though.
Only drawback is there's a limited amount of space to write stuff on before it wraps to the next line, and then my writing-over code completely fails, so I can't write filenames. I reckon it's personal preference, but if you like it how it is better I can leave it.
EDIT: Scratch that, why not both? https://dl.dropbox.com/u/31816885/liDecompress-progress.exe
Run with the commandline argument "--overwrite-progress" to have the previous output overwrite the old progress, or without that commandline switch to spit out the regular output with full filename and all. Either one should (hopefully) work fine with your code (just with slightly different parsing), but the way I've been doing it until now will write the filenames, too, which is more helpful anyway, just clutters up the commandline a lot. But in your case, you're not showing the terminal window anyway, so it should be fine.
tl;dr Ignore me. It works like normal.
Also note this will fail to decompress a file with a hyphen in the beginning of the filename. But if anyone ever puts hyphens in the beginning of filenames, they had it coming.
Aha, right, I see now. Previously I hadn't actually run it from the actual command line, I just wrote a batch file and made it write the output to a text file. I ran it from the actual console this time though, and I can see what you mean: it overwrites the previous progress as it goes. Obviously writing it in a text file (or tracing it, which is what I'm currently doing in Matchbox) doesn't reflect this, but it is helpful when using the console.
Well, the new .exe works with both methods: i.e. the old way works, but if I add the overwrite argument I also get each line at a time. I'll use the old one though since, as you said, I'm not showing all the output so clutter isn't a problem. Plus I also have the filenames, which may be helpful somehow.
As for hyphens: you're right, I mean who does that?
EDIT: By the way, just figured out how to keep windows on top active (i.e. not allowing you to access the main window while a dialogue is up). I still had to do it manually (no built-in feature), but it was much simpler than I thought, and works perfectly.
EDIT2: OK, so I've managed to parse the compression and decompression output and make a loading window with a bar and everything. There's no time estimation yet but I'll get to that.
However, there's no progress output when merging, nor when stripping the .exe. (Can't recall what happens when simply pulling the .paks though). So yeah, do you want to add progress for those, or should it just say "please wait"?
Excellent! Glad that's working for you. Kudos and all that.
I haven't looked at a progress output or anything like that for merging yet. I was thinking today about how I want to overhaul the program anyway, and I'll probably add that sort of support in soon. Maybe along with some optimizations to speed the program up. I'd like to make it into a more full-fledged mod-managing program, with some sort of way to add and remove mods from a queue type of thing, like how GooTool manages it. Dunno yet. But yeah, I'll be working on it.
As for stripping, the problem there is that it's really just two huge file operations. First, I load the "Little Inferno.exe" resources into memory, which is a Windows API call that I can't split up into good progress-bar-sized chunks. Then, I save it all to disk (again a single Win32 function call). This second step is likely what takes all the time in the program (though the first may take a fair amount of time as well- I should probably test this). I could theoretically split the file-writing up into chunks, and spit out some kind of output at intervals, though I'm not sure how much that would gain. Do you think the time delay is enough of an issue? I can do that if need be.
EDIT: Just changed the program to spit out some debug output; seems that indeed the LoadLibrary() call to load the resources from the executable and the WriteFile() to save resource.pak to disk are the two major time-consumers for this program. The first is taking about 2.3 seconds, and can't be split into chunks for me to spit progress data out about. The second is taking about 2.6 seconds, and can. Writing embed.pak and frontend.pak to disk is taking somewhere between 1-10 milliseconds apiece, so they're not really any concern. If you want me to split the saving-resource.pak part of the program into progress-sized chunks, I can. Your call.
Oh, in case it helps you any, here's a version of the program that spits out some output along the way: https://dl.dropbox.com/u/31816885/strip-output.exe
Oh looks nice! Suggestion: Make the GUI version localizable, that will help much!
Oh it seems I forgot to tell you what those Chinese means. They are just the common program-crashed window in Windows 7. It is meaningless. No useful information provided.
@Mygod: When the GUI's mostly done I'll look into adding other languages. The only language I know other than English is Dutch though, and Google Translate will mess up grammar, so I'll probably need peoples' help with that (you can do Chinese if you want). But yeah for now I'll just try and get the main stuff done.
@Daxar: About the mod installing: I was planning to have a queue system similar to GooTool, which my program could probably manage. It'd basically take note of all mods that are in the queue, and every time you save it copies the original paks, then installs the mods in order of the queue (which you can customize of course). Just like GooTool, actually.
Unless you have a more efficient way of course, by all means do that. Just so you know you won't have to make your program keep track of a queue, if you don't want to.
The ouput-spitting version of the program right now isn't usable, but thanks anyway. If it's not too much trouble, giving some progress in the second step would be nice. The time it takes isn't really huge (much faster than decompressing resource.pak, for sure), but a progress bar might still be of some use. At the very least it'll let people know that something's happening (rather than a static "please wait").
If you do that, should I try estimating time on it as well?
@Mygod: Hmm, I can tell from your screenshot that you're running it in a folder with non-ASCII characters. I'll mess around and see if I can replicate the bug. Could be an error in my code. Strange that it would crash midway through, though. Anyhoo, thanks!
@puggsoy: Ok, if you want to make the queue in your GUI, that works too. The only potential issue is the possibility of wasting a lot of time if you call modManage multiple times with one .pak file each time, rather than once with all of them. So don't do that. Should be a fairly simple two-step process: Copy the original .pak files over, call modManage.exe [pakfile1]...[pakfilen]. The modManage program already merges the .pak files in order, so that should work fine. Sorry if you know all this already; I'm just kinda thinking out loud. Should work great, then.
Another thing, in case you didn't think of it: GooTool saves all the .goomod files to a central location when they're installed, in case the original files are deleted. It's probably a good idea to do that here, too.
And I'll get cracking on strip.exe progress. I think you don't need a time estimate on that one; a progress bar alone should be fine.
picking up with more item info - each item in the manifest has a binDataOffsetBytes field that is an offset in bytes into the itemsBinDataBytes section of the mainfest file where the rest of the data for the item is. once you find that data for the item, it's kind of like finding a whole new resource embedded inside the manifest resource (that's because each item was a separate resource right up until the last minute when i combined them all to help loading performance.) so, the binary data for each item starts with a header just like a normal resource would:
struct
{
BinHdrPtr skels;
BinHdrPtr joints;
BinHdrPtr bones;
BinHdrPtr boneShapes;
BinHdrPtr boneParts;
BinHdrPtr bonePartTreeVals;
BinHdrPtr rgnCells;
BinHdrPtr stringTableBytes;
BinHdrPtr burnGridUsedDataBytes;
u32 itemId;
i32 itemIdStrId;
u32 animResId;
vec2 iconAnimBoundsMin;
vec2 iconAnimBoundsMax;
BinLocStrKey name;
i32 costCoins;
i32 costStamps;
i32 valueCoins;
i32 valueStamps;
BinLocStrKey desc;
i32 unlisted;
i32 popsCoins;
i32 moneyItem;
f32 animThreshold;
f32 motorThreshold;
i32 absPosition;
f32 scaleVariance;
i32 quantity;
i32 shipTimeSec;
u32 initialBurnExportId;
i32 initialBurnPerGroup;
i32 mouseGrabbable;
f32 illuminate;
i32 enableHFlip;
i32 floorWallShadow;
i32 splitJumpLastFrame;
f32 purchaseCooldown;
i32 allowDirectionalLight;
u32 mouseGrabSoundResId;
i32 instantEffects;
i32 freezeOnCollide;
i32 enableFreezePostAnim;
u32 uniqueIgniteSoundResId;
i32 collideItems;
i32 collideEnvironment;
i32 orbitalGravity;
i32 allowExplodeStreaks;
i32 burnSlowsAnim;
i32 plagueOnCollide;
u32 spawnLimitBurnExportId;
u32 instAshSoundResId;
i32 canGetPlague;
i32 instAshDoesSplat;
f32 modXAmpMin;
f32 modXAmpMax;
f32 modXFreqMin;
f32 modXFreqMax;
f32 modXPhaseMin;
f32 modXPhaseMax;
f32 modXSpeedMin;
f32 modXSpeedMax;
f32 modYAmpMin;
f32 modYAmpMax;
f32 modYFreqMin;
f32 modYFreqMax;
f32 modYPhaseMin;
f32 modYPhaseMax;
f32 modYSpeedMin;
f32 modYSpeedMax;
};
it's going to take me a while to document all of the fields here so i'll do an overview of some stuff for now. the skeleton/joint/bone sections are related to the different ragdolls that you can get for each item. most items only have 1 but some have slight variations which would show up in the data as different skeletons, and the joints and bones define how to build the physical bodies in box2d for each skeleton. itemIdStrId is a key for looking up the items id as a string in the local string table for the item. iconAnimBounds controls how the item's catalog animation gets framed in the little window in the catalog when you view that item. (a vec2 is just a pair of 32bit floats) name and desc are the text you see in the catalog for the item (a BinLocStrKey is a u32 and an i32, the i32 being the key for the string table) cost and value stuff is related to how much things cost in the catalog and how much you get for them when you burn them. unlisted was just used for development and doesn't play any role in the shipping game. the "threshold" ones refer to things happening at different times in the burn cycle of the item. -1.0 is before you burn the item, 0.0 is when it catches on fire and larger values like 1.0 or 2.0 are after the item has been burning for a while. shipTimeSec controls how long the item takes to ship in case you want to mod everything down to 1 second : ) the mod X and Y stuff at the bottom is a rarely used bunch of data that causes the engine to give the item some non-physical movement (used only for things like bubbles and butterflies that fly up and away in a sin wave looking pattern.)
there's a bunch of stuff here (i haven't even gotten into the specifics of each section of data besides just the header) and i'm out of time again since i'm still on vacation for the holidays. i'll follow up again later with more info.
Woot, lotsa cool stuff to look at now! https://dl.dropbox.com/u/31816885/itemmanifest.dat.xml.txt
(also above pastebin character limit now. Derp)
A lot of these seem to be pretty self-explanatory values, such as costCoins, quantity, etc. But then again, there's a fair amount of terse-looking stuff too. But rather than stressing over every detail for what every field means, how about just documenting the file format stuff, so we can get all the values out and play with them? The stuff we don't know we can probably figure out by tweaking values and seeing the effect on the game. I don't want you to feel stressed with having to explain in detail what every field of every structure does. If we have questions about anything in particular, we'll ask. :)
Thanks, Allan! And enjoy your vacation. Tell your family we say hi. :)
Yeah when I move it into an English-only directory, the problem fixed! Thanks!
By the way about the itemmanifest.dat.xml, I think we'd better use attributes instead of elements for itemId, itemIdStrId, animResId, costCoins, etc.
Yeah, I was thinking about attributes, but decided on separate elements for readability. I'll toy with it, though. I'll probably split the sndmanifest.dat.xml up a bit for readability... We'll see.
Glad to hear an ASCII-only directory works. My code should theoretically work with UTF-8 fine, but I imagine I messed something up someplace. I'll have to look into it.
I'm willing to help translating the GUI in french if that's needed.
I can translate the GUI in Russian and Italian.
Ok, testing out XML with the itemdata information placed in XML attributes rather than separate elements: https://dl.dropbox.com/u/31816885/itemmanifest.dat.xml-attributes.txt
I personally think this is a bit less readable, but I'm open to suggestions/questions/comments/whatever. I don't think there's a way with tinyxml2 to add whitespace to make that any more readable; that'd probably have to be done by hand.
It IS a lot of data, but I think I can make reasonable defaults for some of the values, and cut down on the amount of XML that way.
I also tried to replicate the non-ASCII-directory bug, and I think because I'm using an English version of Windows, it's not feeding the proper UTF-8 string to the program, which is quite annoying. I don't think I can test this bug because of this issue, so for the time being just make sure your .pak files are in ASCII-only folders if you're clicking-and-dragging pakfiles into the executable. :/
EDIT: Totally forgot about seeing if I could use Puggsoy's program to test this...
EDIT2: Nope, that one seems to work fine. But then again, it can only decompress one .pak file at a time, and you only had trouble with multiple .pak files...
I'll allow you to select multiple files for decompressing and merging in the next release. Hopefully that should be tomorrow.
By the way, I was checking out compressing and found a couple of bugs. First of all, if it tries to compress a file that doesn't exist (i.e. you write a nonexistent file in the filelist.txt), liCompress crashes. Nothing special, just "liDecompress has stopped working".
Secondly, there are issues when one of the files you want to compress into the .pak has a space in the filename. It appears to compress OK, but when I decompress it again the file with the space doesn't appear, and the filelist.txt generated has a blank line instead of the file path. Obviously it's generally better not to have a space at all, but some people might do it so I thought to check it.
@momo & movildima: Awesome! I'll most likely have the languages in XML files, which you'll have to edit by hand. Is that OK?
k, cool.
@puggsoy: re/ the nonexistent file: Yes, I didn't begin to bother testing that. The whole .filelist.txt thing is still just a placeholder for now until I can think of something better. I haven't yet. I especially haven't done anything to finalize the format with error-checking and stuff. In any case, even if it didn't crash, it'd stop later, since it checks the number of items in the .filelist.txt and compares this against how many files it was able to compress, and halts if the number was different. I'm still trying to figure out the best way to do all this, and I'll address this once I've decided. But thanks for finding that; I'll be sure to keep it in mind.
re/ the filename space error: This actually has absolutely nothing to do with spaces in filenames and everything to do with a file not being in the original residmap.dat. Note this line in my README.md (line 84, where I'm talking about sound resources):
(come to think of it, I probably should have made this a lot clearer in the readme; put it in big bold letters someplace. I'll do that on next release)
So, if you're working on a GUI so people can select files to compress into .pak files, please stop now. It won't work at all. Right now we don't know how resource ID's are hashed from filenames, and until we do, there's no good way to get an ID from a filename if we don't have a filename-to-ID mapping already (i.e. unless it's in the original residmap.dat). For my code, the default ID generated for an unrecognized filename is undefined, and the default filename for an undefined ID is a blank string, hence the empty line in the .filelist.txt. Once we get the hashing algorithm, I'll make this all work, but for now it completely messes everything up. Long story short, If a file isn't in the original .pak files, it can't be compressed. Well, it can, but it ends up with a meaningless resource ID and can't be decompressed. For now. I probably should have added a test for that, too, but hey, alpha-stage modding tools and all that.
@all: I decided to find out if some of the item data (from the structure Allan just mentioned) could have default values, and so I made some debug code that spit out all the values for all the items. Seems to me I can simplify this XML down quite a lot in order to conserve space (and sanity, and all that kind of good stuff).
Here's my output: http://pastebin.com/Hwe2Bmtd
As you can see, a lot of the items have the same values for a lot of these. "costStamps," for example, is 0 for every item, and can be optimized out completely. Not everything can have a default value, of course, but I'm going to be going through this and figuring out what values I can make optional in the XML. I'll be sure to document it all, of course.
I think the game quotes everything with the ID so the hashing method is not important at all. You can create one yourself or let user enter IDs themselves.
You can try to extend the filelist.txt and make it hold another field called ID. (use : to separate maybe)
While running strip.exe or pullpakfiles.exe, generate a default filelist.txt. While running liDecompress.exe make it read the filelist.txt so that it knows what ID should be used.
Here are my thoughts, but I need a confirmation. What do you think, @allan?
While it is indeed possible for users to create an ID by hand, the chances can become quite high that IDs overlap, and the consequences for something like that could potentially be disastrous. This is why I'm holding out for getting the actual filename hashing algorithm, so we don't end up with a lot of random IDs that are potential conflicts floating around.
puggsoy wrote:
@momo & movildima: Awesome! I'll most likely have the languages in XML files, which you'll have to edit by hand. Is that OK?
I'm totally ok with that. Just send me that file on email
movildima
at
gmail
dot
com.
mygod is correct that ids could be anything in theory as long as there are no collisions and daxar is also correct that you probably want to derive them from file paths to make those collisions unlikely. here is the hash that we use (which itself is by no means a robust hash but surprisingly we never encountered any problems with collisions using it)
http://pastebin.com/fKcMKKk3
we expect the string that is being hashed to be UTF-16 encoded (i.e. what you would pass to the unicode version of a win32 api function)
Ah great! More exciting stuff is coming! : D
@puggsoy my email is my username at gmail dot com.
@momo & movildima: Heh, well, actually I'm gonna hold translation off until it's almost complete. That is, at least until we have a finalised mod format, and my program has a mod queue and all that. Sorry to get your hopes up, mostly I just wanted to check if XML was good with you guys for when the time comes (then I can sort of plan ahead). Good to know XML works though, so thanks.
@Daxar: No, I wasn't implementing any real interface for choosing files to compress yet. I was just running some error tests for compression (for example I made an alert dialogue pop up if my program can't find a corresponding filelist.txt). I'll definitely wait until you're done with compression, maybe even until we get a finalised mod format, before I allow people to choose files to compress. But for the time being the user just specifies a .pak, like your program.
Hopefully Allan's hash should help though!
I wouldn't like to drag all exe files to my game directory because that will mess up the directory. So I created a bat file like this. I think this will be a good idea. : P
@echo offtitle liTools Command Line
path %PATH%;"{liTools Path}";"{liTools Path}\util"
cmd
EDIT: Another bug found when I was trying to edit an data/animations/Intro/TomorrowCorporationLogo.png. Layer masks used. There may be something wrong with PNG parsing. Please check.
PNG: https://dl.dropbox.com/u/94637484/TomorrowCorporationLogo.png
PSD: https://dl.dropbox.com/u/94637484/TomorrowCorporationLogo.psd
Result: http://i.imgur.com/2fSKd.jpg
@Allan: Sweet, thanks! I'll get cracking on that, then.
@Mygod: Good idea for the batch file to add the tools to your PATH. I hope that works out good for you!
As for the images, after poking around some and fooling with GIMP a lot, I think I've figured it out. It's looking to me like Little Inferno is expecting any pixel that has an alpha = 0 to have RGB = 0,0,0 as well. I set GIMP to only modify the alpha channel of your image, and filled in the transparent areas without modifying the RGB values, and sure enough, here's what I got: http://i48.tinypic.com/p526w.png
I double-checked by compressing your image then decompressing it, but the same image gets spit back out, so I don't think it's my code. In any case, this is kind of a strange limitation imo. I've toyed around with low-alpha images, too, and it seems to behave in a strange manner with those, as well. For example, the "glassbulb" item has a low-alpha brown image that seems to display absolutely fine. However, if I try to make one of the chimneys in the main screen low-alpha, it treats it as if the alpha stays 255. Very strange to say the least.
Matchbox 0.2
You can just install this over the old version and it'll update it, no need to uninstall.
Here's what's changed:
-You can now decompress multiple .paks at once.
-You can now merge multiple .paks at once.
-You can now pull .paks from an .exe without stripping it.
-Added a bunch of checks and alerts (for example it won't let you try and strip an already-stripped exe). There's probably still a couple of things that I haven't checked for but I've done the most obvious stuff.
-Added a dialogue with a loading bar to show the progress of decompressing and compressing. For now it just says how many files out of the total it's done, but I'll make an algorithm for time estimation in the next release.
That's about it. I also fixed a small problem I had with the buttons being scaled down, but it's not a huge change and you probably won't notice it.
Note that unfortunately, when you select multiple files they're forced to be fed to the program in alphabetical order, even if you select them in a different order. This isn't really a problem when decompressing .paks, but it's not ideal when merging. Obviously once I get the whole queue system sorted you can drag them around and organise them by priority, but for now if you want to install them in a non-alphabetical order you should do them one-by-one.
By the way Daxar, I noticed that modManage doesn't give any usable progress output. Are you planning to add it or isn't it possible? No hurry of course (figuring out those IDs is probably more important), but just thought to ask.
@puggsoy: I'm planning on refactoring the modManage program sometime anyway, so for now there's no progress output. I'll put that on my TODO list though.
EDIT: Just testing out the new version of Matchbox. It works beautifully and looks brilliant. Great job!
@Allan: Works great, thanks!
You can try to set rgb values to zero when alpha is zero. I guess doing this might work.
@Mygod: Yep, that's what I did for now. Though it doesn't work for low-alpha portions of an image. It may have to do with normal images, and images without normal images don't work for low-alpha... I dunno. It's strange, to say the least.
@all: The new XML format with default values looks like this: http://pastebin.com/iH5qxjPT which looks fine with XML attributes rather than elements, so I stand corrected, Mygod.
Unless there are any objections, I'll stick with this format for the itemmanifest (until more is explained, of course. Maybe I'll end up splitting stuff into separate XML files like how it all was originally; I'll see). I parsed through all the values in itemmanifest.dat for all the fields of all the structures, and if any value for a field was used more than 50% of the time, I made that the default, which seems to work out great. The only field that was borderline was enableHFlip, which had the value 1 a total of 98 times out of 192 total objects. The rest of them were in the 180's or so.
Paste of the default values I settled on: http://pastebin.com/BSY4PpMy
Onward ho to adding support for new resources.
image data for the game is stored with premultiplied alpha. if you want to get back to the approximate values that would go into a PNG, you need to divide the color channels by the alpha channel when you extract the images from the game and multiply when you put them back in.
Ah, that would do it. Thanks, Allan! Works like a charm now.
EDIT: Ok, wow. I had no idea that this would make such a tremendous difference. I changed my modManage program to pull all the data from the .pak files and store it in RAM instead of in files in temp/ and tempmod/ before merging and recompressing. It's running somewhere between 7x and 20x faster compared to the original version, which is amazing. Time to refactor liDecompress and liCompress, I think.
@puggsoy: modManage is taking around 0.2 seconds now for 4 mods that each change one file in each original .pak file, so I might hold on a progress bar. XD
EDIT2: Done optimizing liDecompress to alleviate the need for a temp/ subfolder. I think it's somewhere around 40-60 seconds faster now. Good call, Mygod.
Wow. The effect is better than I expected. Nice job.
Do you mean your program store all the data in RAM? That will consume large amounts of memory. When it runs on a computer with low memory, it will produce large amounts of hard errors which will decrease the speed a lot. You can try to read one file a time (requires thread synchronization), decode it and write it to the disk. If not, that's great.
Anyway, I'm looking forward to your new version. : D
(Mygod and I chatted on IRC, and I answered his questions there)
Still gotta track down a new strange bug, but I think I'll just optimize liCompress in the same way I did liDecompress and maybe change the itemmanifest.xml format a bit before releasing. I'm going for more changes per release and less-often releases, hence the delay. Of course, you guys can always compile from source or come on IRC (irc.esper.net, channel #world-of-goo) and nag me if you want bleeding-edge builds. ; )
EDIT: Version 0.3.5 changelog:
* vdata/itemmanifest.dat is partially parsed to vdata/itemmanifest.dat.xml
* itemmanifest.dat.xml is NOT parsed back; it's incomplete
* Optimized liCompress and liDecompress to use RAM instead of hard disk, which means they should run ~40 seconds faster for resource.pak, just using more RAM in the process
* Commandline arguments for liDecompress and liCompress; see readme for details
* Optimized modManage program to merge mods within seconds
* residmap.dat now parsed to/from XML
* Images now properly converted to/from premultiplied alpha
* Initial Unicode support, which is still a bit buggy, but should help if you don't have an English localization of Windows
* Various bugfixes
Download: https://dl.dropbox.com/u/31816885/liTools0.3.5.zip
Source: https://github.com/meh2481/liTools/tree/3f8383a91a041ec1f65771b44595041f...
EDIT2: Fixed a major bug that would botch decompression of resource.pak and updated the links above accordingly. As a bonus, you can now specify compression level for liCompress via the commandline switch --level=n, where n can be 0 (no compression), 1 (fast compression) through 9 (full compression, slow but smaller filesize). All files are now compressed by default, and if the resulting compressed file is larger than the original, it is discarded and the uncompressed file is used. After running a couple tests, at level=9 my compressed resource.pak is about 2.1 MB smaller than the original, which takes an extra 30 seconds or so. At default compression level, it's still about 1 MB smaller. It's fun to toy around with, at least.
Looks like now I'll get cracking on adding new resources into pakfiles; I was thinking of a small residmap.dat-type file in each .pak mod so that the files can still be decompressed easily.
0.3.5 is working well for me. : D
By the way, @allan, if you have time, I would like to know the format of following files as well because I would like to make a non official translation for Little Inferno. (because kyle said the game cannot currently support Chinese and these will also help @Daxar's project sometime) : P
TwCen.font.xml
dialogmanifest.dat
letterdb.dat
@Daxar: Compression levels look interesting. Can the game still use the .paks if they're compressed than any level other than the default? Because if so I might add an option for it in Matchbox.
Hello people! You've done very nice work here!
I'd love to participate in modding this game too! I have previous experience in both C# and C++.
Check out my Portal 2 Extras Patcher (it's no longer needed though): http://forums.thinkingwithportals.com/downloads.php?view=detail&df_id=339
@puggsoy: I can translate your GUI to finnish.
@puggsoy: Just tested it and it works great. I think most compression schemes work this way; they don't care about compression level when decompressing, since it's just an optimized compression that takes longer.
@Haapavuo: Hi there, and thanks for the offer! If you have any ideas for what should be done, feel free to fork my git repo at https://github.com/meh2481/liTools and go to town with hacking on it. For the most part, I'm just basically madly coding tons of stuff whenever Allan describes something new, and I seem to be keeping up for the most part. Of course, once school starts back up again in a couple of weeks, all bets are off. You could also work with Mygod's codebase; it's a little bit behind, but may be organized better than mine. : )
@Haapavuo: Awesome! The more languages the better.
@Daxar: That's cool, just wondering. In that case I'll make a little dialogue to choose compression level (which will probably cause me to also create some sort of menu (maybe "Preferences" or something) to let you choose if it pops up every time you compress).
I am now doing some experiments with itemmanifest.dat file. I found out how item prices, names and descriptions are stored. Seems very interesting... I will update this post soon with a screenshot!
@Daxar: Thanks for the link! I'll look into that!
@puggsoy: Could you release the future versions of Matchbox rather zipped than with an installer? I'm pretty sure most people will like it that way. Thanks!
EDIT: OK! I've now struggled with the bits for a while and here are some of the results:
Legend:
RED: Item price in catalog
BLUE: Item cash drop when burn
MAGENTA: Item delivery time
GREEN: Item name
PINK: Item description
BROWN: Crash when edited (item ID maybe)?
BLACK: Didn't change anything noticeable.
GRAY: Interesting...
https://www.dropbox.com/s/2p4tfvrsjav4whp/liItemsManifest.png
https://www.dropbox.com/s/7wsvyjgqxuc0kuz/liItemsManifest4.png
Item names (and all texts) are always cut with 2 zeroes (0x00) so it should be pretty easy to write a parser for these dat files (or at least this one).
I'm still going to do some more research and updating the second image...
EDIT: There seems to be a lot of different burn types (these are not all): https://www.dropbox.com/s/uju42wq50kjlot0/liBurns.txt
Unfortunately it actually compiles into an installer, apparently it's essential. Personally I'd also prefer to release it as an archive, at least these beta versions.
I'll snoop around and see if it's possible to put it in a .zip, but if not it'll need to stay like this. I don't find it too bad, it makes installing new versions easy and I'd probably put it in an installer once it's complete anyway (since it's more user-friendly than .zips, and user-friendliness is the whole point of the GUI).
@Haapavuo: While I appreciate the effort, and you have found some cool stuff there, check the posts above. Allan is already describing the itemmanifest.dat format in great detail, and my liDecompress program already spits out a lot of that data in XML form. While finding new stuff is always cool, and reverse-engineering via a hex editor is good, perhaps you should do so on a file we don't know the format to already. ; )
@puggsoy: Interesting that it only spits out installers, but imo it's not that big of a deal. What I'd personally gripe about is that you have it hosted on some ad-ridden site like Mediafire. Have you looked into a hosting service you can direct-link to, like Dropbox?
Also seconded for having an installer rather than a .zip; makes it more easy-to-use and professional in my opinion. Have you looked into an icon for it yet? One of our awesome artists (or even me) could make you one.
@Haapavuo
Hello and welcome to this site! : D
If you want, here is my full source: https://dl.dropbox.com/u/94637484/LIRMFullSource.7z
The packer module is still buggy. Audio encoder (converted from @allan's C++ source by hand) often throws an AccessViolationException. Though my program will ignore the error and retry immediately, the output file will be probably messed up. (if luckily no exceptions are thrown, the program will works fine) You can decrease the core number to 1 to decrease the probability of the exception. If you wouldn't like to edit these ugly codes, I hope it will be of some value one day.
P.S. These code wasn't changed since 12/22/2012 so it doesn't support some new features that is included in liTools. I spent a whole day to try to solve that bug but failed. And after that I decided to use Daxar's liTools.
@Daxar: Sorry about that: with AdBlock it's always hard to remember which sites have ads and which don't. The thing is that someone recommended Mediafire a while back, and since I didn't see any problem with it I just chose it. Back then I didn't host much (just zips for video game sound effects), but now I'm doing much more and I can see how it'd be a problem.
I'll check out Dropbox, since it can do direct links that's probably best. I'll have to take the time to move all my other things over as well though (since I don't want my files spread over separate hosts).
I'm pretty rubbish at icons (or computer art in general: hoping to remedy that eventually), so I don't think I'll make my own. It's not so high on my priority list yet so I haven't really looked around or asked anyone, but if you or anybody here is willing, feel free to whip something up. Something like a Little Inferno-themed matchbox (heh) would be cool.
EDIT: Relinked my previous posts, they're now direct links from Dropbox.
Oh crap, how did I miss all that... I used the previous version of liTools which didn't have the itemsmanifest.dat.xml part. Got to try it out now! You've done more than I thought!
@Mygod: Thanks for your source. I'll look into that!
EDIT: Ahaa, I see. It does yet extract only some of the data, not yet the item names and descriptions etc. I'll see if I can crack some other files we don't know yet.
https://www.dropbox.com/s/acc8zs6yh68qf95/TCgotPwnd.jpg
Pages