Hmm, I didn't try running it on the Steam version before, myself, but it seems that you're right. Any idea what would cause it to not work, Allan? Seems that the Steam version runs perfectly fine, even when stripped and without any .pak files in the running folder.
@puggsoy: A GUI is a nice idea. It's currently quite low on my priority list, however. I'd rather have a bunch of commandline tools that are mostly finalized before I spend the time on a complicated GUI to use them. Definitely feel free to make a GUI if you want.
this is a bare bones win32 console app that takes a little inferno sound resource file (as extracted from a pak) and generates an ogg file. it's really just to demonstrate how to do it so error checking and other niceties have been omitted. the code depends on libogg and you MUST use version 1.1.4 if you want to compile the source because that's the version at which the original vorbis stream was created. using the newer version will result in an invalid bitstream. code to convert ogg files back into game sound resources is coming next...
also, a few people have asked about decrypting xml files or documenting the xml file format that the game uses but there actually isn't any xml data in any of these files. some of the resource ids have .xml at the end of them, but that's just because some of the data that was originally used to build them came from xml files and i didn't bother giving the resources a different file extension. trying to extract the xml data would be like trying to extract the c++ source code data from a compiled exe - the information that the source code represents is present in the exe, but its structure has been thrown away during the compilation process.
EDIT: is steam patching the steam version of the game back to its clean state before running it?
@Allan: Nope, Steam isn't patching it. In my case I copied the game somewhere else though, so it's possible that Steam ran the un-modded version, even though I started it from my copied version (although I don't know how that'd work).
@Daxar: Yeah I understand just decompressing and stuff is already difficult enough. I might try and whip up something simple over the next few days, just to make the whole deal a bit simpler.
part of the steam init process is to check that the game is running under the steam client and if not, launch the steam client and have it automatically launch the game. that may explain why you get the normal version of the game (the same thing you would get if you launched the game from the steam client yourself) when you try to run a modded version of the game from someplace else that is based on the steam game exe.
@Vials: Could you please stop asking about the TwCen.font.xml? We heard your request the first time, and nagging about it isn't going to help you any. IMO, fonts are the absolute least important part of modding, and we should save that until last, but that's just my opinion.
@puggsoy: I guess part of it is that I don't want a whole lot of people trying to mod up front, before any modding techniques or file formats are finalized. For me, having commandline programs to begin with alleviates the potential problem of newbies working hard to make mods before the modding scene is ready for them. But I suppose that isn't a huge issue anyway, since this isn't becoming the most active forum on the planet.
@Allan: Excellent, thank you! Plugged that source code in and got it running without a hitch. If I knew more about libogg I'd have tried making a recompressor today, but I ran out of time/energy to do so. And if XML would be a difficult resource type to work with next, fair enough. You know far more than we do what is tackle-able at this point, so I'm game for whatever you want to throw our way next. (I still gotta get wordPackDict.dat working, anyhow...)
Vials is nagging because he has to translate in Russian on his site. I'm interested in it too, but you're right Daxar, we should concentrate on principal resources.
@Allan
If the key is not in XML, what makes the toys act in different ways? Because, at least what I've seen, the toys folder is categorized the same way as World of Goo balls folder is.
And another question: I'm (still) a newbie in programming, but I read something onhow to make GUI's in Visual Studio. I have it actually. Can I try to make a GUI? The point is only to put some buttons and Browse fields and some calls to the programs Daxar made I think. Am I right?
in addition to libogg you'll also need libvorbis for this one (tested with libogg-1.1.4 and libvorbis-1.2.3 but the specific versions should be less important here.)
@Movildima: Well I'm using Flash's ActionScript and haven't had much experience with Visual Studio, but from what I know what you said is basically what you'd do, yeah. It would probably be good to have some general experience first though, because (at least in my idea), there'd be some copying and moving around of files involved. It'd be more than just simply calling Daxar's programs, since those basically do the decompressing and recompressing, while the GUI would probably organise everything into a simple and manageable space.
Don't let me stop you of course, feel free to make a GUI. Just thought I'd say how my view of the whole situation is, since it might be more complex than you expect.
@Daxar: I can completely understand that actually, although it didn't cross my mind before. I'll just try and make it and see how it goes, but if I feel like releasing it will unleash a premature modding frenzy then I don't have to release it right away. I also enjoy making GUIs for command-line programs in general (I did it before for a video game sound conversion tool), so I thought it'd be cool to do so for this since I was getting so many ideas.
Craaap, I accidentally deleted your three posts when I was trying to merge them, Mygod. Sorry. Here it is:
Mygod wrote:
My GUI version still does not work. The game keeps crashing. Can't figure out why.
EDIT: Finally got my stupid program worked! MUAHAHAHA!
It is just because I made a number of stupid mistakes. Now my program replaced an image in the game successfully! (I replaced data/_common/coinFrame.png)
Screenshot: The mod is so stupid so I'm not going to show you some screenshots.
Download: It is all magic so no downloads!
There's still something wrong with my program about reading 64bpp PNG. Maybe there's something wrong with the class library.
I will use my computer (the better ones) this weekend so I'm going to publish my program this weekend. (after I fix a lot of bugs and add some cool features such as ogg, compression level, more xml support, and so on)
@Allan
OkTHANKS! Your program works peeerrrrffffeeeccct!
Okay let's now discuss the mod format. (weirdly I have no edit button while using IE8, moderator please combine these posts)
I think the best format to use is also .pak that contains everything (this will make much less work to be done) and an index.xml.xsl (to modify the index.xml file I mentioned before, we can even delete vdata/residmap.dat with this file). What are your opinions?
@ allanblomquist
I didn't understand what to do, but I tried: LISndExtract.exe resource.pak embed.pak frontend.pak
and the result is this : http://i48.tinypic.com/m3odi.jpg
with the "Dont send" Error
@Allan: Awesome, thanks so much! Question, though: Is there some particular OGG compression level or file size that I should be using for each file? I've tried replacing the original title music with a new song, and it loads and plays fine. However, when the sound resource is unloaded (If I click on one of the chimneys to start the game, or press Esc and end the game), I get a message box saying there was an unhandled exception, with no other details. It could well be a mistake in my own code, but in case it wasn't, I was wondering if I need to use some particular settings for OGG compression in order for the output to be correct (If I use the original generated OGG, it works fine).
@Mygod: I agree that some kind of .pak-based format would be an ideal candidate for modding. I personally, however, would shy away from XSL, since it provides a lot of difficult barriers to entry for new modders. XML is difficult enough for some modders as it is, and XSL is fairly terse in comparison. I'm not certain we need to worry about adding resources to the game for the moment, however, since we don't know enough about how the game handles resources to be able to add our own currently. Heck, we don't even know how resource IDs are generated from filenames yet. But once we do know how it all works, a standardized format will definitely be ideal.
@puggsoy/movildima: If either or both of you want to build GUIs on top of my commandline programs, great! To be honest, GUI design isn't my area of expertise. I will mention that it's probable that these tools will be ported to Mac/Linux as Little Inferno is ported, so it may be a good idea to develop the GUI accordingly, with minimal platform-dependent overhead. Quite easier said than done, I know, but just something to keep in mind. Thanks!
there is one more piece to the puzzle before you'll be able to use new sound files in the game. the resource vdata\sndmanifest.dat is a manifest that holds information about all of the sound resources in the game. it is used by the resource loader to resolve resource dependencies and allocate memory up front without having to wait for any individual resource to finish loading first. that manifest will also have to be updated with info about your new sound before the runtime will be able to handle it correctly. the manifest starts with a header
struct
{
BinHdrPtr sounds;
BinHdrPtr takes;
};
which sets out a section for sounds and a section for takes. in this context, a take refers to a single ogg file (which is processed into a single resource in a pak file by the code i posted before.) a "sound" in this context is a handle for a group of one or more takes and is not something that is in a pak file anywhere - it's just a way to refer to a collection of takes. the sounds section of the manifest is an array of structures that look like this:
struct
{
u32 logicalId;
i32 firstTakeIdx;
i32 numTakes;
};
where logicalId is an id that is used as a handle for that logical sound, firstTakeIdx is the index of its first take record in the takes section of the manifest, and numTakes is the number of take records for that logical sound. the final section of the manifest is an array of all the take records:
struct
{
u32 resId;
i32 channels;
i32 samplesPerSec;
i32 sampleCountPerChannel;
i32 vorbisWorkingSetSizeBytes;
i32 vorbisMarkersSizeBytes;
i32 vorbisPacketsSizeBytes;
};
where resId is the resource id of the audio data for this take in the pak and the rest is meta data that needs to be in sync with the info in the header of the resource data.
i'm guessing that last part is why your new sound doesn't work correctly. you'll need to find the take record for that resource in the sound manifest (the one with the matching resId field) and update its meta data to match that of the new resource.
modifying the sound manifest is also how you would change the set of random sounds you hear for a lot of in game actions. when you grab a goo ball item in the fireplace for example, the game randomly plays 1 of 5 different goo grab sfx (data\SFX\grab\goo.01.flac - data\SFX\grab\goo.05.flac) because there is a single "sound" record in the manifest for grabbing a goo ball and it has 5 "takes." the logicalId of the sound record is how the goo ball item connects to that logical sound, but that's a story for a future gigantic post. : )
EDIT: about mod format - my original thoughts on this were that mods could be distributed as pak files and then they could be run by dragging the pak for the mod you want to play onto the game exe. the game would then mount that pak along with its standard paks and preferentially load resources from the mod pak before it goes looking for them in the base game paks. that functionality doesn't actually exist in the game yet though, it would have to be added in a future update. anyway, just saying that making a simple change to the game itself like that is an option, even if that's not actually a good idea for how to do mods.
Okay, let's use XML instead. Here are my thoughts.
An modder can create/modify some files in the data and vdata directory, and then make a {modName}.xml to describe the mod.
<?xml version="1.0" encoding="utf-8"?>
<mod name="{Mod Name}" version="{Mod Version}">
<resource>
<!--
And then, a lot of file elements. Here are the attributes.
id : Required : u32
path : Required : string (relative path, such as vdata/residmap.dat)
compressed : Optional (true as default) : boolean
type : Optional (raw as default) : special
raw: Just compress it (or not) and copy it to the pak
png, ogg and so on: Convert it into the original format before copying
e.g. <file id="635769077" path="data\_common\0.png.png" type="png" compressed="false" />
-->
</resource>
<!--and an embed element and a frontend element-->
</mod>
We can use an program to compile it to a pak file. (maybe we can use the magic resource id for this xml file)
Another program/module can read the mod.pak file and combine it into the three pak files. After that, modding is done.
@Allan
If you want to make the game a bit moddable, make it be able to read outer files if vdata/residmap.dat exists. That will make much less work! (of course your idea is good too if it can read multiple pak files at the same time)
The problem I see with that, Mygod, is a lot of the information in that XML is extraneous. An id can be determined from filename, type and compression can be determined from file extension, that sort of thing.
@Allan: Hmm, that's an interesting idea, so long as you don't mind the work involved. Maybe a sort of modlist.txt (or XML, or some binary format, whatever format would be easiest for you to work with) with a list of .pak files. Resources with IDs in the pakfiles higher in the list are given higher priority over resources further down the list. Just an idea.
Time to start parsing that .dat file for ogg data...
Multi-platform support shouldn't be a problem for me, I'll be using Adobe AIR so it shouldn't be too difficult to make Mac and Linux versions. I haven't actually done it before myself so it will take some work, but I mean it'll probably be easier than if I were working in some other language like C# or Visual Basic.
EDIT: I've started working on it and I was thinking, whatever the mod format ends up being, it should contain information about which .paks it edits. That way if you want to install a mod that effects only embed.pak or frontend.pak, you don't have to decompress and recompress the massive resource.pak.
Also, do you think it's possible to only decompress/recompress part of a .pak? resource.pak really does take a while, it'd be a pain to do the entire .pak whenever you mod the game.
I was thinking about that earlier; if you replace a resource with a resource of identical size, it would be a simple file seek and write operation. However, if it was of differing size, you'd have to shift the remaining contents of the file around afterward.
However, most of the time spent with unpacking and packing the resources is spent with the zlib compression and decompression. If all you're doing is updating a few files, none of the rest of the resources would have to be decompressed; they'd just be pulled out of the .pak file and stuck back in as-is. It'd make it quite a lot faster, and would be far more ideal. Currently, as I'm just trying to get data in the right format, I haven't added support for this type of resource updating yet, but it's definitely a good idea.
I tried making a mod by decompressing the resource.pak, and overwriting the arena.png file but as I attempted to recompress it said error wrong file size, help please? Also, are we going to be able to edit the xml files?
I don't know the exact error message because it flashed very quickly at the end of the compression and the resource file doesn't change. Something along the lines of "Error: file size 38xx different than original file size 37xx". I'm going to try MyGod's now because it has a GUI, but frankly I would rather have a program that doesn't depend on the .NET framework. Why don't you guys work together to make the mod manager?
EDIT: I was able to use MyGod's one, it's a bit more complete I must admit and it conserves the .normal.png images of the items. Check out the metal chimney.
Because he's using C++ while I'm using C#. And C# programs must depend on the .NET Framework.
Sorry about that. I'm not familiar with C++ (I can only read C++ codes and convert them to other languages, but I have never written a C++ program) and I guess Daxar isn't familiar with C# either.
Also, my program keeps Daxar from being lazy and his keeps me from being lazy too. I spent almost the whole weekend making my program.
About that error message, I guess Daxar is forcing to use the same file size so that the program can run faster. (while mine is slow)
@Daxar
I tried multi-thread packing (making .pak files), it was much faster on my computer. (though it made the game crashed)
P.S. I'm a student so I won't have much time on workdays. So if you find something wrong with my program, please wait patiently on workdays.
EDIT: @momo1526 Please tell me as soon as it works for you (or not).
EDIT:
V1.0.4.0 has been published.
Download & Source: See #74
What's new: Improve the speed of encoding & decoding images.
Here is another mod I made: https://lh5.googleusercontent.com/-dK0CvDxhGdQ/UM2rpn_B7QI/AAAAAAAAAC4/o... . I don't really know if they're woth uploading here, but it sure is fun
Also, if with MyGod's we can just replace and recompress, can't we just have a renamed zip system (like gootool) which indicates what goes where? Something like .limod or .infernomod? Using the index a little bit differently somehow maybe.
Next step, messing with xml!!! (what's the progress on that?)
I don't know the exact error message because it flashed very quickly at the end of the compression
Hence why you should run it from the commandline like I do instead of clicking and dragging.
Could the error message have possibly been "Error: size of file list: xxxx differs from size of files to pak: xxxx"? This would mean that your .filelist.txt generated by liDecompress isn't there or is malformed somehow. Did you run the new version of liDecompress before running liCompress?
momo1526 wrote:
it's a bit more complete I must admit and it conserves the .normal.png images of the items
Not sure what you're talking about here; I convert the .png.normal images to PNG as well. I just leave the file extensions as-is. Same with the "coloritemicon", "colorbgicon", and "greybgicon" images.
@Mygod: Good idea with the multithreaded compression. I should look into that.
What does the .normal do exactly? I thought it was there for collision purposes.
Here's exactly what the console shows. This only occurs while compressing resources.pak, I tested my own frontend.pak experiment and it worked. (this was all done on a fresh LI exe)
This error only occurs when the .filelist.txt has a different number of entries than the number of files in the temp/ subfolder created while compressing, which means that either the .filelist.txt was malformed, something didn't compress right, or something got left over in the temp/ subfolder, which is deleted if that loop terminates normally, and if an error occurred before... apparently it doesn't...
Thanks for finding this bug. In the meantime, try deleting the temp/ folder and running again.
EDIT: Looking back at my code, this may or may not help, since I'm deleting that folder each loop anyway. Hermph.
EDIT2: No, that isn't the issue, since you end up with less files than are in the .filelist.txt. May well be that there's a problem with compressing some file. Could you run it as
liCompress.exe [pakfiles] > out.txt
and paste the generated out.txt here for me to look at?
Oh, and to better keep track of my source code, I've created a public GitHub repo, which can be found here: https://github.com/meh2481/liTools
I'll post my code changes there rather than in .zip files, to save time. That'll also have more up-to-date code than my releases, since I have the habit of committing rather frequently.
@momo: I'm working on a GUI for Daxar's tools, which doesn't require the NET Framework. It does require the Adobe AIR runtime, but it'll automatically prompt you to do it when you install the program anyway (and I think it'll do it for you too if you're connected to the internet).
Also about the modding formats, we're waiting until we know how to edit everything (or almost everything) before we start designing something like .limod or .infernomod files. Most likely they'll just be .paks with a bit of addin data but we'll discuss it more later on.
I think C++ also requires Microsoft Visual C++ Redistributable, doesn't it? And Adobe AIR would be another dependence.
The main goal of this program is to make modding fast and decrease the waiting time. (not OS independence) So I merged all modules into one single executable and some dlls instead of running another executable because I saw an article say creating a new process is slow. (it will take about 1 second, if you do it repeatly, that would be a large number)
As far as I know C++ doesn't require any external runtime/framework downloads. This might be different for Visual C++, but pure C++ doesn't. You do need to download Adobe AIR for my program but it does it automatically during installation if needed. It's also backwards compatible (latest version works no matter which version you compiled for), which iirc isn't the case with .NET programs.
The downside is that I can't use .dlls and do fancy stuff like .NET can, it can only run external programs (although it can specify arguments), so my program may be slower or less efficient than yours.
Which is one of the reasons I use GCC rather than Visual Studio. GCC executables require no runtime libraries unless you specifically link them in, and I'm statically linking what I need.
As a general rule, yes, processes have a relatively large startup time, since you have to allocate memory for an address space and that sort of thing. In fact, I ran into this problem with the initial version of my decompression program, which called another program to decompress each file. However, the end user is generally the slow cog in the machine when it comes to program design, and they don't generally run programs back-to-back quickly enough for it to be a noticeable issue.
As a general rule, I'm doing my best to stick to the Linux design strategy: Each program should do one specific task, and should do it well. This way, each program is solid and has little room for error, and GUIs can easily be built on top of them. This is how a lot of GUI-based programs are designed. For development, I commonly use an IDE called Code::Blocks, which is a GUI interface that calls the GCC compiler- a commandline program. Opening up my Microsoft Office folder, I find 35 executables. Some of these are likely called via a similar method. Perhaps not via the command line, but it is fairly common for large programs to be split into smaller programs just for simplicity's sake.
EDIT: liTools Minor Version 0.3.1 changelog:
- Multithreaded decompression/compression for speed
- OGG streams are now left uncompressed
Mostly this is just an attempt at optimizing the compression/decompression routines. The programs now poll your PC for the number of processor cores and spawn that many threads to work on decompression or compression, so hopefully now they'll run several times faster. On my quad-core lappy here, it cut the resource.pak decompression time from 3 minutes 8 seconds down to 1 minute 43 seconds. If you have a single-core processor, it probably won't make that much of a difference, but hopefully this'll be a good deal faster for most people.
EDIT2: liTools Minor Version 0.3.2 changelog:
- modManage program for merging in mods
A fairly straightforward program to add in mods in the form of .pak files. See the readme for details on how to run it and how to create mod files. Rather than decompressing the original .pak files, decompressing the mod .pak files, then recompressing the original .pak files, it cuts out a lot of the hassle by pulling everything out of the .pak files without decompressing, then sticking the changed files in the right places. It also only updates files that were changed, so if your mod only changes frontend.pak, then it only worries about changes to that file. Even if your mod changes the gigantic resource.pak, it takes only about 30 seconds to merge changed files in (On the two times I tried it on my computer here). Definitely a lot faster than a two-minute decompress followed by a two-minute compress.
Wow, that's great stuff Daxar! It'll definitely play a big part in the final mod format.
This'll also simplify my GUI, since it can just merge in one go, instead of the whole un-and-re-compress method you mentioned.
It's README.md because that's the first thing git slapped into the repo. GitHub parses .md files differently than .txt; it adds some markup stuff to it, which is nice (You can see how it looks on the GitHub page). It's still a standard text file, so just open it with Notepad++ (or whatever) like normal.
How's the GUI coming, btw?
EDIT: Version 0.3.3 changelog:
- on decompression, vdata/wordPackDict.dat is parsed and written to vdata/wordPackDict.dat.xml
- on compression, wordPackDict.dat.xml is compressed back into .dat form
Finally got around to parsing the vdata/wordPackDict.dat file, and made a fairly simple XML specification for it. If you do modify the xml, keep in mind that the probabilities of all the words should add up to 1.0, as Allan said. I dunno if anything bad will happen otherwise, but it might. Hopefully an up-and-coming language update won't make this work entirely moot...
Cheerio!
-Daxar
GUI's coming along OK. Right now I'm pretty much just making it more user-friendly to decompress, compress, and merge .paks, using a file selection dialogue. Pretty much just making it easier to use your tools, instead of having to drag or use the command-line (plus it doesn't have to be in the same folder). I'm not gonna implement anything that really assists in making or organising mods yet though, since I think it'd be better to wait until we've created a mod format.
Anyway I might get this "beta" done today, otherwise probably tomorrow.
Ok, cool. Let me know if there's any working directory issues; like it's decompressing in the wrong spot or not finding residmap.dat or stuff like that, and I'll take a look at it.
Working on getting this sndmanifest.dat parsed and updating properly...
At the moment it's actually working very well, AS3 has this cool thing where you specify a "working directory", and somehow it runs the .exe as if it were in that directory. So I just do that in the folder where the file is and it can extract right next to it.
The only little snag is that for liDecompress.exe, I need to copy over residmap.dat over to the directory as well (otherwise I'll get an error), and then when it's done decompressing I delete it again. This isn't a huge problem since the file's tiny and everything works smoothly, but in the rare case that someone already has their own file named residmap.dat in the folder (and it isn't the same as yours) then we'd get some problems.
Even though that's really unlikely, if it's possible to somehow put residmap.dat into the .exe, or something similar, that would be convenient. Not essential though.
My program sometimes fails at vorbis_synthesis_blockin (access violation) especially while multithreaded while compiling ogg files.
Using:
libogg 1.1.4
libvorbis 1.2.3
vorbis_synthesis sometimes returns -136 but sometimes it returns 0 before access violation.
Do someone know what the issue could be?
By the way, I know when the pak files are loaded now. (because I have got a lot of problems)
embed.pak is loaded once the game starts to run.
frontend.pak is loaded after the warning animation is faded out.
resource.pak is loaded after the user clicks a building and the background is faded out.
@puggsoy: Excellent! I figured working directories were causing that trouble earlier with some people clicking-and-dragging. I think there's no real benefit to reading residmap.dat from a file, like I originally thought there would be, so I'll be sure to make it an array inside the executable on next release. Good call.
@Mygod: Sorry, I haven't seen that before. I do know I was having some crash issues before because I wasn't properly mutex'ing a global list before writing to it, but as far as I know, Allan's code is threadsafe. Sorry, I dunno.
I actually was running into a problem with the OGG parsing for my program for the past three days; kept crashing on me and I couldn't figure out why. I think I got it fixed now, though.
liTools Version 0.3.4 changelog:
* On decompression, vdata/sndmanifest.dat is parsed and written to vdata/sndmanifest.dat.xml
* On compression, sndmanifest.dat.xml is compressed back into .dat form
* Added recalcSoundManifest.exe program to update sndmanifest.dat.xml if .ogg files are changed
* Removed residmap.dat; contents are now compiled into the compress/decompress executables
Be sure to check the new section in the readme about sound modding for this one. If you're installing over an old version, be sure to run liDecompress on embed.pak, since vdata/sndmanifest.dat is now parsed out to XML.
This is about all I can do for now with the info Allan has given us so far. I've looked at the other .dat files with a hex editor, and they're all in a similar format, but I need to know what represents what in order to parse it all out to XML.
Awesome, works like a charm.
OK so the GUI can now strip the .exe, compress/decompress .paks, and merge .paks (only one at a time for now). It's getting late but tomorrow I'll make sure everything's working OK and upload it so you guys can check it out.
0.3.4 didn't work for me when I dragged three pak files onto liDecompress.exe at the same time.
EDIT: When I dragged them one by one, it worked.
Screenshot: http://i.imgur.com/8JPFy.png
Suggestions:
1. Move files to data or vdata directory directly instead of moving them from temp directory. I guess this will speed up a lot. (20-40 seconds)
2. In most cases trying to merge into pak files will fail because there are only 64 bytes padding or less. You can try to enlarge the minimum padding, e.g. a half of the original file size. Undoubtedly, it will make the file larger but it can make merging speed up. (Enlarge all files' padding when there is one enlargement requested)
Users should be able to adjust the size of padding if they feel their hard disk remaining space is not enough or they would like to make merging faster.
3. Uninstall mod: Keep original pak files and original mods, replace the new files with old one. If the old one is in the remaining mod file list, use it. If not, extract it from the original pak files or delete them directly if it is not in the original pak files either.
4. Remove the redundant numsounds, numtakes, channels, samplespersec, samplecountperchannel, vorbisworkingsetsizebytes, vorbismarkerssizebytes, vorbispacketssizebytes attributes in the sndmanifest.dat.xml. I think they can be found in the encoded flac files, can't they?
Hmm, strange error you're getting there. Could you translate that to English and tell me what it says? May have something to do with you running a non-English localization of Windows, and some of the file path stuff failing.
Anyhoo, for the suggestions:
1. Definitely a good idea; I'll have to refactor a fair amount of code and such to do that, but the hard disk is generally the biggest performance bottleneck, and I could tell for sure that it was becoming an issue after I made the code multithreaded. Good call.
2. For my purposes, I'm completely ignoring padding in the pakfiles. I'm not sure of a good way to determine how much padding is ideal, and if someone creates an image slightly larger in file size than the padding, the entire pakfile would have to be rewritten anyway. In my opinion, merging mods is fairly fast as it is, and I'm not sure how much performance could be gained by coming up with an "ideal" amount of padding. Best-case scenario, yes it could help a lot, but worst-case scenario it would help none, it would make the program more complex, and would waste a lot of disk space.
Reminds me, there was this podcast I listened to where they talked about optimizing file I/O... I should listen to that again sometime.
3. Yep, it's on my to-do list. See "TODO.txt." I think I included that file in the latest version...
4. Originally, that was my plan: to pull all the data from the ogg files on recompression. But unfortunately, it's impossible to guarantee that this will work. The way my program is set up, it compresses and decompresses one pakfile at a time, and I did this intentionally so that it's as generic as possible. Unfortunately, in order to get all the vorbis header data, you have to read the entire .ogg file, which takes a fair amount of time for all 613 sound resources. So reading in all the data when I'm compressing sndmanifest.dat.xml is out of the question. It made compressing embed.pak take far too long (about 45 seconds on my first tests).
But wait, you're compressing all the ogg files anyway-- No, you're not. Most of the sound resources are in resource.pak, while sndmanifest.dat is in embed.pak. If someone starts on a fresh install of liTools and runs liDecompress on embed.pak, then liCompress on embed.pak, all the vorbis data the game needs to play sounds has been completely lost, unless I preserve it in the xml. This is why I made recalcSoundManifest a separate program, since there's no guarantee at any time that all the sound resources are there and are readable. This program will skip over sounds it can't open and continue on it's merry way, only updating the sound manifest where it can. This way, the original sound manifest (which is hopefully correct!) is preserved as much as possible. It takes a bit of time to recalculate the manifest, yes, but this way it's always correct, so long as the user runs recalcSoundManifest if they change a sound and are sure to compress embed.pak again. And the only way to do this properly that I can think of is to keep the vorbis header data in the xml.
EDIT: As for the "numsounds" and "numtakes", they're just there for personal reference. They aren't used by the compression code at all. They're just good numbers to know for the future when we're working with sound stuff.
i figured we might as well do the item data next. even though it's huge and scary, the item data is really what makes up the core of the game. the file that we're interested in is vdata\itemmanifest.dat which contains the data for all the items in the game. it starts off with a header:
struct
{
BinHdrPtr itemsManifest;
BinHdrPtr normalDeps;
BinHdrPtr soundDeps;
BinHdrPtr effectDeps;
BinHdrPtr itemDeps;
BinHdrPtr itemsBinDataBytes;
};
where itemsManifest is a section with the main list of all items in the game, normalDeps is a section for all of the normal map texture dependencies, soundDeps is for sound dependencies data, effectDeps for particle effect dependencies, itemDeps for data on the dependencies that items have on other items and finally itemsBinDataBytes which is a section of all the nitty gritty data about each item.
there is a record for each item in the game in the itemsManifest section and each one looks like this:
struct
{
u32 itemId;
u32 animResId;
i32 recentlyModifiedRank;
i32 firstNormalDepends;
i32 numNormalDepends;
i32 firstSoundDepends;
i32 numSoundDepends;
i32 firstEffectDepends;
i32 numEffectDepends;
i32 firstItemDepends;
i32 numItemDepends;
u32 catalogIconColorItemTexResId;
u32 catalogIconColorBGTexResId;
u32 catalogIconGreyBGTexResId;
i32 binDataOffsetBytes;
};
where itemId is the id of the item, animResId is a reference to the animation resource that the item uses for both the animation on its catalog page as well as in the fireplace, recentlyModifiedRank is just a sort order that is used by a debug cheat that we used to be able to quickly access all of the most recently modified items in the game during development (the cheat isn't accessible in the final shipping version of the game), firstNormalDepends is the index of the first normal map texture dependency record for this item in the normalDeps section of the file and then numNormalDepends is how many of those records this item has, then there is a similar pair of values for sound, effect and item dependencies, then there are 3 texture resource ids for the catalog icons for this item for when you can afford it, you can't afford it, and when it is still locked, and finally binDataOffsetBytes is the byte offset into the itemsBinDataBytes section of the file where the main item data record for this item can be found.
the next section in the file is the normalDeps section which is a bunch of records that look like this:
struct
{
u32 normalTexResId;
};
so each record is just the id of a texture resource. the soundDeps, effectDeps and itemDeps sections follow and they are all pretty much the same - each is just a collection of ids for sound resources, particle effect resources and items respectively.
finally the itemsBinDataBytes section is all of the data for things like item name and description, cost, ship time, physics shapes, collision settings, effect settings, lighting, etc. i'll leave the format of that section for another post since it's probably the biggest by far and i'm out of time at the moment. hope everybody is having a happy holiday!
Catalog
Combos
Forum
Get the Game
News
Hmm, I didn't try running it on the Steam version before, myself, but it seems that you're right. Any idea what would cause it to not work, Allan? Seems that the Steam version runs perfectly fine, even when stripped and without any .pak files in the running folder.
@puggsoy: A GUI is a nice idea. It's currently quite low on my priority list, however. I'd rather have a bunch of commandline tools that are mostly finalized before I spend the time on a complicated GUI to use them. Definitely feel free to make a GUI if you want.
here is the first part of the sound stuff:
bin: https://dl.dropbox.com/s/3rnmdhq7lt7mt3z/LISndExtract.exe
src: http://pastebin.com/8XVa1W9A
this is a bare bones win32 console app that takes a little inferno sound resource file (as extracted from a pak) and generates an ogg file. it's really just to demonstrate how to do it so error checking and other niceties have been omitted. the code depends on libogg and you MUST use version 1.1.4 if you want to compile the source because that's the version at which the original vorbis stream was created. using the newer version will result in an invalid bitstream. code to convert ogg files back into game sound resources is coming next...
also, a few people have asked about decrypting xml files or documenting the xml file format that the game uses but there actually isn't any xml data in any of these files. some of the resource ids have .xml at the end of them, but that's just because some of the data that was originally used to build them came from xml files and i didn't bother giving the resources a different file extension. trying to extract the xml data would be like trying to extract the c++ source code data from a compiled exe - the information that the source code represents is present in the exe, but its structure has been thrown away during the compilation process.
EDIT: is steam patching the steam version of the game back to its clean state before running it?
@Allan Interested only internal structure in TwCen.font.xml
@Allan: Nope, Steam isn't patching it. In my case I copied the game somewhere else though, so it's possible that Steam ran the un-modded version, even though I started it from my copied version (although I don't know how that'd work).
@Daxar: Yeah I understand just decompressing and stuff is already difficult enough. I might try and whip up something simple over the next few days, just to make the whole deal a bit simpler.
part of the steam init process is to check that the game is running under the steam client and if not, launch the steam client and have it automatically launch the game. that may explain why you get the normal version of the game (the same thing you would get if you launched the game from the steam client yourself) when you try to run a modded version of the game from someplace else that is based on the steam game exe.
Ah OK, that explains it. I guess that means modding isn't an option for people who only got it on Steam, unless they mod the original files directly.
@Vials: Could you please stop asking about the TwCen.font.xml? We heard your request the first time, and nagging about it isn't going to help you any. IMO, fonts are the absolute least important part of modding, and we should save that until last, but that's just my opinion.
@puggsoy: I guess part of it is that I don't want a whole lot of people trying to mod up front, before any modding techniques or file formats are finalized. For me, having commandline programs to begin with alleviates the potential problem of newbies working hard to make mods before the modding scene is ready for them. But I suppose that isn't a huge issue anyway, since this isn't becoming the most active forum on the planet.
@Allan: Excellent, thank you! Plugged that source code in and got it running without a hitch. If I knew more about libogg I'd have tried making a recompressor today, but I ran out of time/energy to do so. And if XML would be a difficult resource type to work with next, fair enough. You know far more than we do what is tackle-able at this point, so I'm game for whatever you want to throw our way next. (I still gotta get wordPackDict.dat working, anyhow...)
Vials is nagging because he has to translate in Russian on his site. I'm interested in it too, but you're right Daxar, we should concentrate on principal resources.
@Allan
If the key is not in XML, what makes the toys act in different ways? Because, at least what I've seen, the toys folder is categorized the same way as World of Goo balls folder is.
And another question: I'm (still) a newbie in programming, but I read something onhow to make GUI's in Visual Studio. I have it actually. Can I try to make a GUI? The point is only to put some buttons and Browse fields and some calls to the programs Daxar made I think. Am I right?
second half of the sound code - this is to go from ogg files to game sound resources:
bin: https://dl.dropbox.com/s/bxsvttl3fr7bz2j/LISndCompile.exe
src: http://pastebin.com/YUutYty0
in addition to libogg you'll also need libvorbis for this one (tested with libogg-1.1.4 and libvorbis-1.2.3 but the specific versions should be less important here.)
@Movildima: Well I'm using Flash's ActionScript and haven't had much experience with Visual Studio, but from what I know what you said is basically what you'd do, yeah. It would probably be good to have some general experience first though, because (at least in my idea), there'd be some copying and moving around of files involved. It'd be more than just simply calling Daxar's programs, since those basically do the decompressing and recompressing, while the GUI would probably organise everything into a simple and manageable space.
Don't let me stop you of course, feel free to make a GUI. Just thought I'd say how my view of the whole situation is, since it might be more complex than you expect.
@Daxar: I can completely understand that actually, although it didn't cross my mind before. I'll just try and make it and see how it goes, but if I feel like releasing it will unleash a premature modding frenzy then I don't have to release it right away. I also enjoy making GUIs for command-line programs in general (I did it before for a video game sound conversion tool), so I thought it'd be cool to do so for this since I was getting so many ideas.
Craaap, I accidentally deleted your three posts when I was trying to merge them, Mygod. Sorry. Here it is:
@ allanblomquist
I didn't understand what to do, but I tried: LISndExtract.exe resource.pak embed.pak frontend.pak
and the result is this : http://i48.tinypic.com/m3odi.jpg
with the "Dont send" Error
You have to run LISndExtract on the .flac files extracted by the .paks, not the .paks themselves. You can extract them using Daxar's tools.
Wow, I missed a lot, you guys are awesome! Shouldn't that build be a page visible up there? You should talk to david about that.
@Allan: Awesome, thanks so much! Question, though: Is there some particular OGG compression level or file size that I should be using for each file? I've tried replacing the original title music with a new song, and it loads and plays fine. However, when the sound resource is unloaded (If I click on one of the chimneys to start the game, or press Esc and end the game), I get a message box saying there was an unhandled exception, with no other details. It could well be a mistake in my own code, but in case it wasn't, I was wondering if I need to use some particular settings for OGG compression in order for the output to be correct (If I use the original generated OGG, it works fine).
@Mygod: I agree that some kind of .pak-based format would be an ideal candidate for modding. I personally, however, would shy away from XSL, since it provides a lot of difficult barriers to entry for new modders. XML is difficult enough for some modders as it is, and XSL is fairly terse in comparison. I'm not certain we need to worry about adding resources to the game for the moment, however, since we don't know enough about how the game handles resources to be able to add our own currently. Heck, we don't even know how resource IDs are generated from filenames yet. But once we do know how it all works, a standardized format will definitely be ideal.
@puggsoy/movildima: If either or both of you want to build GUIs on top of my commandline programs, great! To be honest, GUI design isn't my area of expertise. I will mention that it's probable that these tools will be ported to Mac/Linux as Little Inferno is ported, so it may be a good idea to develop the GUI accordingly, with minimal platform-dependent overhead. Quite easier said than done, I know, but just something to keep in mind. Thanks!
there is one more piece to the puzzle before you'll be able to use new sound files in the game. the resource vdata\sndmanifest.dat is a manifest that holds information about all of the sound resources in the game. it is used by the resource loader to resolve resource dependencies and allocate memory up front without having to wait for any individual resource to finish loading first. that manifest will also have to be updated with info about your new sound before the runtime will be able to handle it correctly. the manifest starts with a header
struct
{
BinHdrPtr sounds;
BinHdrPtr takes;
};
which sets out a section for sounds and a section for takes. in this context, a take refers to a single ogg file (which is processed into a single resource in a pak file by the code i posted before.) a "sound" in this context is a handle for a group of one or more takes and is not something that is in a pak file anywhere - it's just a way to refer to a collection of takes. the sounds section of the manifest is an array of structures that look like this:
struct
{
u32 logicalId;
i32 firstTakeIdx;
i32 numTakes;
};
where logicalId is an id that is used as a handle for that logical sound, firstTakeIdx is the index of its first take record in the takes section of the manifest, and numTakes is the number of take records for that logical sound. the final section of the manifest is an array of all the take records:
struct
{
u32 resId;
i32 channels;
i32 samplesPerSec;
i32 sampleCountPerChannel;
i32 vorbisWorkingSetSizeBytes;
i32 vorbisMarkersSizeBytes;
i32 vorbisPacketsSizeBytes;
};
where resId is the resource id of the audio data for this take in the pak and the rest is meta data that needs to be in sync with the info in the header of the resource data.
i'm guessing that last part is why your new sound doesn't work correctly. you'll need to find the take record for that resource in the sound manifest (the one with the matching resId field) and update its meta data to match that of the new resource.
modifying the sound manifest is also how you would change the set of random sounds you hear for a lot of in game actions. when you grab a goo ball item in the fireplace for example, the game randomly plays 1 of 5 different goo grab sfx (data\SFX\grab\goo.01.flac - data\SFX\grab\goo.05.flac) because there is a single "sound" record in the manifest for grabbing a goo ball and it has 5 "takes." the logicalId of the sound record is how the goo ball item connects to that logical sound, but that's a story for a future gigantic post. : )
EDIT: about mod format - my original thoughts on this were that mods could be distributed as pak files and then they could be run by dragging the pak for the mod you want to play onto the game exe. the game would then mount that pak along with its standard paks and preferentially load resources from the mod pak before it goes looking for them in the base game paks. that functionality doesn't actually exist in the game yet though, it would have to be added in a future update. anyway, just saying that making a simple change to the game itself like that is an option, even if that's not actually a good idea for how to do mods.
Okay, let's use XML instead. Here are my thoughts.
An modder can create/modify some files in the data and vdata directory, and then make a {modName}.xml to describe the mod.
<?xml version="1.0" encoding="utf-8"?>
<mod name="{Mod Name}" version="{Mod Version}">
<resource>
<!--
And then, a lot of file elements. Here are the attributes.
id : Required : u32
path : Required : string (relative path, such as vdata/residmap.dat)
compressed : Optional (true as default) : boolean
type : Optional (raw as default) : special
raw: Just compress it (or not) and copy it to the pak
png, ogg and so on: Convert it into the original format before copying
e.g. <file id="635769077" path="data\_common\0.png.png" type="png" compressed="false" />
-->
</resource>
<!--and an embed element and a frontend element-->
</mod>
We can use an program to compile it to a pak file. (maybe we can use the magic resource id for this xml file)
Another program/module can read the mod.pak file and combine it into the three pak files. After that, modding is done.
@Allan
If you want to make the game a bit moddable, make it be able to read outer files if vdata/residmap.dat exists. That will make much less work! (of course your idea is good too if it can read multiple pak files at the same time)
The problem I see with that, Mygod, is a lot of the information in that XML is extraneous. An id can be determined from filename, type and compression can be determined from file extension, that sort of thing.
@Allan: Hmm, that's an interesting idea, so long as you don't mind the work involved. Maybe a sort of modlist.txt (or XML, or some binary format, whatever format would be easiest for you to work with) with a list of .pak files. Resources with IDs in the pakfiles higher in the list are given higher priority over resources further down the list. Just an idea.
Time to start parsing that .dat file for ogg data...
Multi-platform support shouldn't be a problem for me, I'll be using Adobe AIR so it shouldn't be too difficult to make Mac and Linux versions. I haven't actually done it before myself so it will take some work, but I mean it'll probably be easier than if I were working in some other language like C# or Visual Basic.
EDIT: I've started working on it and I was thinking, whatever the mod format ends up being, it should contain information about which .paks it edits. That way if you want to install a mod that effects only embed.pak or frontend.pak, you don't have to decompress and recompress the massive resource.pak.
Also, do you think it's possible to only decompress/recompress part of a .pak? resource.pak really does take a while, it'd be a pain to do the entire .pak whenever you mod the game.
I was thinking about that earlier; if you replace a resource with a resource of identical size, it would be a simple file seek and write operation. However, if it was of differing size, you'd have to shift the remaining contents of the file around afterward.
However, most of the time spent with unpacking and packing the resources is spent with the zlib compression and decompression. If all you're doing is updating a few files, none of the rest of the resources would have to be decompressed; they'd just be pulled out of the .pak file and stuck back in as-is. It'd make it quite a lot faster, and would be far more ideal. Currently, as I'm just trying to get data in the right format, I haven't added support for this type of resource updating yet, but it's definitely a good idea.
I tried making a mod by decompressing the resource.pak, and overwriting the arena.png file but as I attempted to recompress it said error wrong file size, help please? Also, are we going to be able to edit the xml files?
Hmm, what was the exact error message? I can't find the place in my code that would spit that message out.
And Allan hasn't told us yet how the xml files are set up. Soon enough.
My first mod generated!
http://i.imgur.com/g8zVt.png
(okay this is actually a bug of my program)
EDIT:
My first mod generated again!
http://i.imgur.com/CCDNt.jpg
And finally, my working Little Inferno Resource Manager V1.0.3.5!
It can almost do everything...
But before using it, you NEED to install .NET Framework 4 first: http://www.microsoft.com/en-us/download/details.aspx?id=17718
Screenshot: http://i.imgur.com/1o2mI.png
Download: https://dl.dropbox.com/u/94637484/LittleInfernoResourceManager.7z
Source: https://dl.dropbox.com/u/94637484/LittleInfernoResourceManager_source.7z
Third party libraries: libogg, libvorbis, FreeImage
Help file is in the executable! Welcome to try!
I don't know the exact error message because it flashed very quickly at the end of the compression and the resource file doesn't change. Something along the lines of "Error: file size 38xx different than original file size 37xx". I'm going to try MyGod's now because it has a GUI, but frankly I would rather have a program that doesn't depend on the .NET framework. Why don't you guys work together to make the mod manager?
EDIT: I was able to use MyGod's one, it's a bit more complete I must admit and it conserves the .normal.png images of the items. Check out the metal chimney.
Because he's using C++ while I'm using C#. And C# programs must depend on the .NET Framework.
Sorry about that. I'm not familiar with C++ (I can only read C++ codes and convert them to other languages, but I have never written a C++ program) and I guess Daxar isn't familiar with C# either.
Also, my program keeps Daxar from being lazy and his keeps me from being lazy too. I spent almost the whole weekend making my program.
About that error message, I guess Daxar is forcing to use the same file size so that the program can run faster. (while mine is slow)
@Daxar
I tried multi-thread packing (making .pak files), it was much faster on my computer. (though it made the game crashed)
P.S. I'm a student so I won't have much time on workdays. So if you find something wrong with my program, please wait patiently on workdays.
EDIT: @momo1526 Please tell me as soon as it works for you (or not).
EDIT:
V1.0.4.0 has been published.
Download & Source: See #74
What's new: Improve the speed of encoding & decoding images.
Here is another mod I made: https://lh5.googleusercontent.com/-dK0CvDxhGdQ/UM2rpn_B7QI/AAAAAAAAAC4/o... . I don't really know if they're woth uploading here, but it sure is fun
Also, if with MyGod's we can just replace and recompress, can't we just have a renamed zip system (like gootool) which indicates what goes where? Something like .limod or .infernomod? Using the index a little bit differently somehow maybe.
Next step, messing with xml!!! (what's the progress on that?)
Could the error message have possibly been "Error: size of file list: xxxx differs from size of files to pak: xxxx"? This would mean that your .filelist.txt generated by liDecompress isn't there or is malformed somehow. Did you run the new version of liDecompress before running liCompress?
Not sure what you're talking about here; I convert the .png.normal images to PNG as well. I just leave the file extensions as-is. Same with the "coloritemicon", "colorbgicon", and "greybgicon" images.@Mygod: Good idea with the multithreaded compression. I should look into that.
What does the .normal do exactly? I thought it was there for collision purposes.
Here's exactly what the console shows. This only occurs while compressing resources.pak, I tested my own frontend.pak experiment and it worked. (this was all done on a fresh LI exe)
@momo1526: Allan said earlier what the .png.normal files do. See his post: http://infernofans.com/comment/289#comment-289
This error only occurs when the .filelist.txt has a different number of entries than the number of files in the temp/ subfolder created while compressing, which means that either the .filelist.txt was malformed, something didn't compress right, or something got left over in the temp/ subfolder, which is deleted if that loop terminates normally, and if an error occurred before... apparently it doesn't...
Thanks for finding this bug. In the meantime, try deleting the temp/ folder and running again.
EDIT: Looking back at my code, this may or may not help, since I'm deleting that folder each loop anyway. Hermph.
EDIT2: No, that isn't the issue, since you end up with less files than are in the .filelist.txt. May well be that there's a problem with compressing some file. Could you run it as
liCompress.exe [pakfiles] > out.txt
and paste the generated out.txt here for me to look at?
Oh, and to better keep track of my source code, I've created a public GitHub repo, which can be found here: https://github.com/meh2481/liTools
I'll post my code changes there rather than in .zip files, to save time. That'll also have more up-to-date code than my releases, since I have the habit of committing rather frequently.
Actually, there is no temp folder 0.0
Yah, I just figured that out. See the edits to my post.
Happy to help! Is it on it's way to be fixed?
@momo: I'm working on a GUI for Daxar's tools, which doesn't require the NET Framework. It does require the Adobe AIR runtime, but it'll automatically prompt you to do it when you install the program anyway (and I think it'll do it for you too if you're connected to the internet).
Also about the modding formats, we're waiting until we know how to edit everything (or almost everything) before we start designing something like .limod or .infernomod files. Most likely they'll just be .paks with a bit of addin data but we'll discuss it more later on.
Great, can't wait for the GUI. I also can't wait to get my hands on that xml!
I think C++ also requires Microsoft Visual C++ Redistributable, doesn't it? And Adobe AIR would be another dependence.
The main goal of this program is to make modding fast and decrease the waiting time. (not OS independence) So I merged all modules into one single executable and some dlls instead of running another executable because I saw an article say creating a new process is slow. (it will take about 1 second, if you do it repeatly, that would be a large number)
As far as I know C++ doesn't require any external runtime/framework downloads. This might be different for Visual C++, but pure C++ doesn't. You do need to download Adobe AIR for my program but it does it automatically during installation if needed. It's also backwards compatible (latest version works no matter which version you compiled for), which iirc isn't the case with .NET programs.
The downside is that I can't use .dlls and do fancy stuff like .NET can, it can only run external programs (although it can specify arguments), so my program may be slower or less efficient than yours.
Which is one of the reasons I use GCC rather than Visual Studio. GCC executables require no runtime libraries unless you specifically link them in, and I'm statically linking what I need.
As a general rule, yes, processes have a relatively large startup time, since you have to allocate memory for an address space and that sort of thing. In fact, I ran into this problem with the initial version of my decompression program, which called another program to decompress each file. However, the end user is generally the slow cog in the machine when it comes to program design, and they don't generally run programs back-to-back quickly enough for it to be a noticeable issue.
As a general rule, I'm doing my best to stick to the Linux design strategy: Each program should do one specific task, and should do it well. This way, each program is solid and has little room for error, and GUIs can easily be built on top of them. This is how a lot of GUI-based programs are designed. For development, I commonly use an IDE called Code::Blocks, which is a GUI interface that calls the GCC compiler- a commandline program. Opening up my Microsoft Office folder, I find 35 executables. Some of these are likely called via a similar method. Perhaps not via the command line, but it is fairly common for large programs to be split into smaller programs just for simplicity's sake.
EDIT: liTools Minor Version 0.3.1 changelog:
- Multithreaded decompression/compression for speed
- OGG streams are now left uncompressed
Download: https://dl.dropbox.com/u/31816885/liTools0.3.1.zip
Source: https://github.com/meh2481/liTools
Mostly this is just an attempt at optimizing the compression/decompression routines. The programs now poll your PC for the number of processor cores and spawn that many threads to work on decompression or compression, so hopefully now they'll run several times faster. On my quad-core lappy here, it cut the resource.pak decompression time from 3 minutes 8 seconds down to 1 minute 43 seconds. If you have a single-core processor, it probably won't make that much of a difference, but hopefully this'll be a good deal faster for most people.
EDIT2: liTools Minor Version 0.3.2 changelog:
- modManage program for merging in mods
Download: https://dl.dropbox.com/u/31816885/liTools0.3.2.zip
Source: https://github.com/meh2481/liTools
A fairly straightforward program to add in mods in the form of .pak files. See the readme for details on how to run it and how to create mod files. Rather than decompressing the original .pak files, decompressing the mod .pak files, then recompressing the original .pak files, it cuts out a lot of the hassle by pulling everything out of the .pak files without decompressing, then sticking the changed files in the right places. It also only updates files that were changed, so if your mod only changes frontend.pak, then it only worries about changes to that file. Even if your mod changes the gigantic resource.pak, it takes only about 30 seconds to merge changed files in (On the two times I tried it on my computer here). Definitely a lot faster than a two-minute decompress followed by a two-minute compress.
Wow, that's great stuff Daxar! It'll definitely play a big part in the final mod format.
This'll also simplify my GUI, since it can just merge in one go, instead of the whole un-and-re-compress method you mentioned.
EDIT: Why does the readme have an .md extension?
It's README.md because that's the first thing git slapped into the repo. GitHub parses .md files differently than .txt; it adds some markup stuff to it, which is nice (You can see how it looks on the GitHub page). It's still a standard text file, so just open it with Notepad++ (or whatever) like normal.
How's the GUI coming, btw?
EDIT: Version 0.3.3 changelog:
- on decompression, vdata/wordPackDict.dat is parsed and written to vdata/wordPackDict.dat.xml
- on compression, wordPackDict.dat.xml is compressed back into .dat form
Download: https://dl.dropbox.com/u/31816885/liTools0.3.3.zip
Source: https://github.com/meh2481/liTools
Finally got around to parsing the vdata/wordPackDict.dat file, and made a fairly simple XML specification for it. If you do modify the xml, keep in mind that the probabilities of all the words should add up to 1.0, as Allan said. I dunno if anything bad will happen otherwise, but it might. Hopefully an up-and-coming language update won't make this work entirely moot...
Cheerio!
-Daxar
GUI's coming along OK. Right now I'm pretty much just making it more user-friendly to decompress, compress, and merge .paks, using a file selection dialogue. Pretty much just making it easier to use your tools, instead of having to drag or use the command-line (plus it doesn't have to be in the same folder). I'm not gonna implement anything that really assists in making or organising mods yet though, since I think it'd be better to wait until we've created a mod format.
Anyway I might get this "beta" done today, otherwise probably tomorrow.
Ok, cool. Let me know if there's any working directory issues; like it's decompressing in the wrong spot or not finding residmap.dat or stuff like that, and I'll take a look at it.
Working on getting this sndmanifest.dat parsed and updating properly...
At the moment it's actually working very well, AS3 has this cool thing where you specify a "working directory", and somehow it runs the .exe as if it were in that directory. So I just do that in the folder where the file is and it can extract right next to it.
The only little snag is that for liDecompress.exe, I need to copy over residmap.dat over to the directory as well (otherwise I'll get an error), and then when it's done decompressing I delete it again. This isn't a huge problem since the file's tiny and everything works smoothly, but in the rare case that someone already has their own file named residmap.dat in the folder (and it isn't the same as yours) then we'd get some problems.
Even though that's really unlikely, if it's possible to somehow put residmap.dat into the .exe, or something similar, that would be convenient. Not essential though.
My program sometimes fails at vorbis_synthesis_blockin (access violation) especially while multithreaded while compiling ogg files.
Using:
libogg 1.1.4
libvorbis 1.2.3
vorbis_synthesis sometimes returns -136 but sometimes it returns 0 before access violation.
Do someone know what the issue could be?
Quote LiSndCompile Line 122-123:
vorbis_synthesis( &vb, &op );
vorbis_synthesis_blockin( &vd, &vb );
@Daxar
Have you ever seen this exception?
By the way, I know when the pak files are loaded now. (because I have got a lot of problems)
embed.pak is loaded once the game starts to run.
frontend.pak is loaded after the warning animation is faded out.
resource.pak is loaded after the user clicks a building and the background is faded out.
@puggsoy: Excellent! I figured working directories were causing that trouble earlier with some people clicking-and-dragging. I think there's no real benefit to reading residmap.dat from a file, like I originally thought there would be, so I'll be sure to make it an array inside the executable on next release. Good call.
@Mygod: Sorry, I haven't seen that before. I do know I was having some crash issues before because I wasn't properly mutex'ing a global list before writing to it, but as far as I know, Allan's code is threadsafe. Sorry, I dunno.
I spent a whole day but still didn't solve the problem... Well I think I will give up my project and help yours. : P
I actually was running into a problem with the OGG parsing for my program for the past three days; kept crashing on me and I couldn't figure out why. I think I got it fixed now, though.
liTools Version 0.3.4 changelog:
* On decompression, vdata/sndmanifest.dat is parsed and written to vdata/sndmanifest.dat.xml
* On compression, sndmanifest.dat.xml is compressed back into .dat form
* Added recalcSoundManifest.exe program to update sndmanifest.dat.xml if .ogg files are changed
* Removed residmap.dat; contents are now compiled into the compress/decompress executables
Download: https://dl.dropbox.com/u/31816885/liTools0.3.4.zip
Source: https://github.com/meh2481/liTools (I added a makefile this time around for ease of compiling)
Be sure to check the new section in the readme about sound modding for this one. If you're installing over an old version, be sure to run liDecompress on embed.pak, since vdata/sndmanifest.dat is now parsed out to XML.
This is about all I can do for now with the info Allan has given us so far. I've looked at the other .dat files with a hex editor, and they're all in a similar format, but I need to know what represents what in order to parse it all out to XML.
Awesome, works like a charm.
OK so the GUI can now strip the .exe, compress/decompress .paks, and merge .paks (only one at a time for now). It's getting late but tomorrow I'll make sure everything's working OK and upload it so you guys can check it out.
0.3.4 didn't work for me when I dragged three pak files onto liDecompress.exe at the same time.
EDIT: When I dragged them one by one, it worked.
Screenshot: http://i.imgur.com/8JPFy.png
Suggestions:
1. Move files to data or vdata directory directly instead of moving them from temp directory. I guess this will speed up a lot. (20-40 seconds)
2. In most cases trying to merge into pak files will fail because there are only 64 bytes padding or less. You can try to enlarge the minimum padding, e.g. a half of the original file size. Undoubtedly, it will make the file larger but it can make merging speed up. (Enlarge all files' padding when there is one enlargement requested)
Users should be able to adjust the size of padding if they feel their hard disk remaining space is not enough or they would like to make merging faster.
3. Uninstall mod: Keep original pak files and original mods, replace the new files with old one. If the old one is in the remaining mod file list, use it. If not, extract it from the original pak files or delete them directly if it is not in the original pak files either.
4. Remove the redundant numsounds, numtakes, channels, samplespersec, samplecountperchannel, vorbisworkingsetsizebytes, vorbismarkerssizebytes, vorbispacketssizebytes attributes in the sndmanifest.dat.xml. I think they can be found in the encoded flac files, can't they?
Hmm, strange error you're getting there. Could you translate that to English and tell me what it says? May have something to do with you running a non-English localization of Windows, and some of the file path stuff failing.
Anyhoo, for the suggestions:
1. Definitely a good idea; I'll have to refactor a fair amount of code and such to do that, but the hard disk is generally the biggest performance bottleneck, and I could tell for sure that it was becoming an issue after I made the code multithreaded. Good call.
2. For my purposes, I'm completely ignoring padding in the pakfiles. I'm not sure of a good way to determine how much padding is ideal, and if someone creates an image slightly larger in file size than the padding, the entire pakfile would have to be rewritten anyway. In my opinion, merging mods is fairly fast as it is, and I'm not sure how much performance could be gained by coming up with an "ideal" amount of padding. Best-case scenario, yes it could help a lot, but worst-case scenario it would help none, it would make the program more complex, and would waste a lot of disk space.
Reminds me, there was this podcast I listened to where they talked about optimizing file I/O... I should listen to that again sometime.
3. Yep, it's on my to-do list. See "TODO.txt." I think I included that file in the latest version...
4. Originally, that was my plan: to pull all the data from the ogg files on recompression. But unfortunately, it's impossible to guarantee that this will work. The way my program is set up, it compresses and decompresses one pakfile at a time, and I did this intentionally so that it's as generic as possible. Unfortunately, in order to get all the vorbis header data, you have to read the entire .ogg file, which takes a fair amount of time for all 613 sound resources. So reading in all the data when I'm compressing sndmanifest.dat.xml is out of the question. It made compressing embed.pak take far too long (about 45 seconds on my first tests).
But wait, you're compressing all the ogg files anyway-- No, you're not. Most of the sound resources are in resource.pak, while sndmanifest.dat is in embed.pak. If someone starts on a fresh install of liTools and runs liDecompress on embed.pak, then liCompress on embed.pak, all the vorbis data the game needs to play sounds has been completely lost, unless I preserve it in the xml. This is why I made recalcSoundManifest a separate program, since there's no guarantee at any time that all the sound resources are there and are readable. This program will skip over sounds it can't open and continue on it's merry way, only updating the sound manifest where it can. This way, the original sound manifest (which is hopefully correct!) is preserved as much as possible. It takes a bit of time to recalculate the manifest, yes, but this way it's always correct, so long as the user runs recalcSoundManifest if they change a sound and are sure to compress embed.pak again. And the only way to do this properly that I can think of is to keep the vorbis header data in the xml.
EDIT: As for the "numsounds" and "numtakes", they're just there for personal reference. They aren't used by the compression code at all. They're just good numbers to know for the future when we're working with sound stuff.
i figured we might as well do the item data next. even though it's huge and scary, the item data is really what makes up the core of the game. the file that we're interested in is vdata\itemmanifest.dat which contains the data for all the items in the game. it starts off with a header:
struct
{
BinHdrPtr itemsManifest;
BinHdrPtr normalDeps;
BinHdrPtr soundDeps;
BinHdrPtr effectDeps;
BinHdrPtr itemDeps;
BinHdrPtr itemsBinDataBytes;
};
where itemsManifest is a section with the main list of all items in the game, normalDeps is a section for all of the normal map texture dependencies, soundDeps is for sound dependencies data, effectDeps for particle effect dependencies, itemDeps for data on the dependencies that items have on other items and finally itemsBinDataBytes which is a section of all the nitty gritty data about each item.
there is a record for each item in the game in the itemsManifest section and each one looks like this:
struct
{
u32 itemId;
u32 animResId;
i32 recentlyModifiedRank;
i32 firstNormalDepends;
i32 numNormalDepends;
i32 firstSoundDepends;
i32 numSoundDepends;
i32 firstEffectDepends;
i32 numEffectDepends;
i32 firstItemDepends;
i32 numItemDepends;
u32 catalogIconColorItemTexResId;
u32 catalogIconColorBGTexResId;
u32 catalogIconGreyBGTexResId;
i32 binDataOffsetBytes;
};
where itemId is the id of the item, animResId is a reference to the animation resource that the item uses for both the animation on its catalog page as well as in the fireplace, recentlyModifiedRank is just a sort order that is used by a debug cheat that we used to be able to quickly access all of the most recently modified items in the game during development (the cheat isn't accessible in the final shipping version of the game), firstNormalDepends is the index of the first normal map texture dependency record for this item in the normalDeps section of the file and then numNormalDepends is how many of those records this item has, then there is a similar pair of values for sound, effect and item dependencies, then there are 3 texture resource ids for the catalog icons for this item for when you can afford it, you can't afford it, and when it is still locked, and finally binDataOffsetBytes is the byte offset into the itemsBinDataBytes section of the file where the main item data record for this item can be found.
the next section in the file is the normalDeps section which is a bunch of records that look like this:
struct
{
u32 normalTexResId;
};
so each record is just the id of a texture resource. the soundDeps, effectDeps and itemDeps sections follow and they are all pretty much the same - each is just a collection of ids for sound resources, particle effect resources and items respectively.
finally the itemsBinDataBytes section is all of the data for things like item name and description, cost, ship time, physics shapes, collision settings, effect settings, lighting, etc. i'll leave the format of that section for another post since it's probably the biggest by far and i'm out of time at the moment. hope everybody is having a happy holiday!
Pages