Coding Request For Daxar

18 posts / 0 new
Last post
puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25
Coding Request For Daxar

Ehh, so there's this compression algorithm library which I'd love to use to decompress some data. Unfortunately the library is in C and, as a result, must be used using C or C++. I haven't the slightest idea on how to do this, so I was wondering if Daxar would be able to whip something up? Nothing too fancy, simply a command-line program that takes an input and output (maybe stdin and stdout) would be more than sufficient.

The library is wfLZ, which can be found here. I realise you've probably got a bunch of other things to do, but it would mean a great deal if you'd be able to do this.

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

GitHub (If you're interested in seeing the code): https://github.com/meh2481/wfLZEx
Download: https://dl.dropboxusercontent.com/u/31816885/wfLZEx.exe

Works fine for me for my test cases. Let me know if you have any troubles.

$ echo "I like your face" | wfLZEx.exe compress > out.data
$ wfLZEx.exe < out.data
I like your face

Reads from stdin, writes to stdout. If you want to compress, pass along the commandline parameter "compress", otherwise it decompresses.

Cheers!
-Daxar

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

First of, thanks a bunch! This is really awesome of you.

However, I haven't really been able to test it yet. First, I got a missing DLL error "libgcc_s_dw2-1.dll", then after I got that there was also one for "libstdc++-6.dll". Once I had both of those, it simply said "The application was unable to start correctly (0xc000007b)."

Any ideas? If it matters at all, my OS is a 64-bit version of Windows 8 Pro.

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

Interesting. I haven't seen these errors before, so I'm interested in a couple different solutions to see what works and what doesn't. Bear with me please.

1) This one should "just work." This one has the original executable and the DLLs in question (As opposed to the ones you found, one or both of which were 64-bit, which can't be called from a 32-bit program like this one): https://dl.dropboxusercontent.com/u/31816885/wfLZEx.zip

2) This one (should) have the DLLs statically linked: https://dl.dropboxusercontent.com/u/31816885/wfLZEx.exe

3) Another test in case #2 doesn't work: https://dl.dropboxusercontent.com/u/31816885/wfLZEx-static.exe

If you could try all three, and let me know which ones work and which ones don't, that'd be sweet. One of them ought to, at least. ; )

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

Ah fantastic; they all work! Of course, I don't have a proper test to use, but compressing and decompressing as with your example works. I also used it on the data I initially wanted to decompress, and checking out the bytes it looks like a valid result.

So should I use #2 or #3 from now on? #1 is a bit of a hassle with the DLLs. Or will you make a sort of "final" version?

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

#2 is the one I was hoping would work, so it's about as final as it gets. Let me know if anything breaks or doesn't work as intended.

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

OK, so apparently the data I want to decompress is actually compressed in multiple chunks. A set of chunks is preceded by a ZLFW header, and each chunk then has the usual WFLZ header. Here's some code I've been given that should be able to decompress these chunks:

const u32 decompressedSize = wfLZ_GetDecompressedSize( ptrToZFLWHeader );
u8* dst = ( u8* )malloc( decompressedSize );
u32 offset = 0;
while( uint8_t* compressedBlock = wfLZ_ChunkDecompressLoop( info.src, &chunk ) )
{
wfLZ_Decompress( compressedBlock, dst + offset );
const u32 blockSize = wfLZ_GetDecompressedSize( compressedBlock );
offset += blockSize;
}

Where "dst" holds the output data. I assume this is for when you're decompressing from within the actual file, hence the "ptrToZFLWHeader". In your program I think you'd just replace that with "vData.data()".

Sorry, this is getting a bit complex. I really should learn C/++ myself sometime so that I can do these things on my own.

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

Hmm...
So you're saying this is in a non-standard wfLZ format?
Unfortunately, I'm not certain how I would go about testing this. I can still write the code, there's just no way for me to know if it's working or not...
Do you have a slightly larger code snippet? I'm not certain exactly what this "chunk" variable is on the line while( uint8_t* compressedBlock = wfLZ_ChunkDecompressLoop( info.src, &chunk ) ) is. I can poke around the Interwebs if need be and figure it out if you can't.

EDIT: Looking back over the code, I think I'm following what they're doing here, aside from the "chunk" variable; wfLZ seems to be quite lacking in documentation as to what this is or how I should use it.

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

I asked the person who gave me the snippet what "chunk" should be, so I'll say once I get an answer. If you want I could give you the file I'm trying to decompress, but I'm not actually sure what kind of result to expect.

EDIT: So apparently it should be defined as such:

u32* chunk = NULL;

Here's what they said about it, if it helps at all:

You never use it directly, it's basically a state variable that the decompressor uses internally, and you just need to make sure it gets passed to the next call to wfLZ_ChunkDecompressLoop(), so it can iterate through the chunks properly.

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

No to nag or anything, I mean you seem busy (since you just updated liTools as well), but bumping just in case you didn't see the edit on my last post.

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

Yeah, sorry about that. Nearly forgot. I'll work on it when I get a chance.

EDIT: Try https://dl.dropboxusercontent.com/u/31816885/wfLZEx.exe
You'll need to pass along the commandline parameter "chunk".

I have no idea if this'll work or not, since I don't completely understand what I'm doing. If this doesn't work, I may need some example data or some way to talk to whoever's giving you this information. What's this for, btw?

My code so far for this looks like follows:

vector vData;
while(!cin.eof())
{
int val = cin.get();
if(val != EOF)
vData.push_back(val);
}


uint32_t* chunk = NULL;
const uint32_t decompressedSize = wfLZ_GetDecompressedSize( vData.data() );
uint8_t* dst = ( uint8_t* )malloc( decompressedSize );
uint32_t offset = 0;
while( uint8_t* compressedBlock = wfLZ_ChunkDecompressLoop( vData.data(), &chunk ) )
{
wfLZ_Decompress( compressedBlock, dst + offset );
const uint32_t blockSize = wfLZ_GetDecompressedSize( compressedBlock );
offset += blockSize;
}


for(int i = 0; i < decompressedSize; i++)
cout << dst[i];
free(dst);

With the most relevant stuff being the block in the middle; the rest is just I/O.

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

Hmm. Still not working, it just crashes ("stopped working").

What I want to achieve with this is to be able to extract (and possibly re-insert) textures from Ducktales Remastered. Initially this would be near-impossible to do without any info, but 2 things were (officially and legally) released about the format; this decompression library, and a document talking about how they encoded the textures. I think that I'll be able to decode the textures myself, but the decompression needs to be done in C.

Unfortunately, the person who's helping me with this wants to stay anonymous. I will ask him if he'd mind contacting you about this though, since that would be much easier for you. Should I give him the email you used when you sent me the LILCC prize?
Here's the file I'm currently trying to decompress. This specific file is actually an extract from this file, which apparently contains Uncle Scrooge's sprites. What I did was, starting after the first ZLFW header, I manually copied "compressedSize" (as specified by that same header) bytes into a new file, then stuck the header onto the start. The reason I did this is because I don't know the ANB format (which is apparently quite complex), and at this point I just want to be able to decompress the wfLZ compressed files. It's possible that I'm doing something wrong in this respect, although I'm not really sure what.

EDIT: Right, so I can send you his email address, how should I do so? I'd like to avoid posting it here for obvious reasons.

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

You can either send him my email address (via email would work) or vice versa; I don't particularly care. I'll take a stab at seeing if I can figure out the problem behind the crash, but I'm guessing that not everything is lining up as perfectly as I'd like in terms of variables and memory and such. Either that or the file itself has problems, but so long as you copied it right and such, that shouldn't be the case.

EDIT: It's crashing on the first loop of decompression, which tells me that I'm probably passing something along incorrectly, or something in the compression header is malformed. I'm assuming the info.src variable should be the same as ptrToZFLWHeader, but if it is, something else is amiss.

EDIT2: Double-checking with a hex editor, and everything seems ok. Probably something at my end, then.

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

Righto, sent. Hopefully that should make things go a tad smoother!

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

Oh my goodness, thanks!!! This is totally awesome! I'll have to find out where the alpha images are stored (I don't own the game myself so I'll have to ask for some ANBs) but this is fantastic! You certainly deserve that sleep : )

There are, however, a couple of things. Firstly, it seems that Scrooge is the only one that comes in one piece; all other characters (that I know of) are made up of weird chunks like this. Did he explain the ANB format? Even if it doesn't contain any info on organising the chunks, I'd be interested in checking it out.

Secondly, what I would prefer is to have the program decompress input into the output, as it did initially. That way I can make a program to take the ANBs, decompress the data with your program and convert it to textures (or have your program produce PNGs to output if you prefer), then merge alpha, organise (if possible), and kind of show in a GUI. That way it'll be simpler for people to use, and they won't have to merge the alpha and organise chunks themselves - this is one of the reasons I looked into the format, since you could already get the textures via RAM dumps but had to arrange everything by hand.

Also yeah, since it's in the Requests forum, new members can't post right away, to prevent people joining purely to request stuff. To be honest someone should actually move that (or make a new thread), since it started out as a request but has now turned into more of a project.

Anyway, once again a huge THANK YOU and hope you sleep well! Looking forward to making more progress : D

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

Apparently I can't even post in the requests section without having had an account there for months, so I reckon I'll keep posting updates here.

This sort of output what you're after? ; ) https://dl.dropboxusercontent.com/u/31816885/output.zip

Executable: https://dl.dropboxusercontent.com/u/31816885/wfLZEx.exe
Source: https://github.com/meh2481/wfLZEx
Usage: wfLZEx.exe [file1.anb] [file2.anb] ... [filen.anb]

Pulls all images from the .anb files and dumps them in the folder "output" that'll be created in the same folder as the executable.

Oh, the artifacts around the sides of the images are not a glitch; they're correct as far as the image data is concerned. The engine itself defines UV coordinates in pieces that it uses to create objects, which is the same way the game can properly display images that are split up into tons of tiny fragments. More on this later; gotta sleep now. Suffice it to say that the images are split up into tons of smaller fragments for file size concerns, and it's physically possible to create an automated way to piece them back together. I haven't done it yet, but from the conversation I had with the dude, I now know how. And I'm insanely busy for the next couple of weeks, I'm afraid. This should be enough to get you guys started, though.

Have fun!
-Daxar

puggsoy's picture
Offline
Joined: 11/20/2012 - 04:25

You are brilliant, this is really more than I expected. I do kind of wish I could've done more, it looks so interesting. Looking at your source code it looks like you've got most of the ANB format sorted.

What I was actually expecting was something that wrote each image's data to stdout, so that I could pipe it through a wrapper and manage the textures through that. To be honest, I'd still like something like this, just so that I can make it a tad more user-friendly. At this point you've done most of the work though, so just an option to choose the output directory would be enough. Ideally I'd have a full-blown explorer that can add and remove stuff from ANBs, but at this point simply getting the images is what matters. I dunno, I just feel like I haven't done enough work myself.

Anyway, as you said you're busy, and actually I am too. There's no hurry with any further work on this, what you've done is more than enough already. Once we both have more free time though I would really like to make some sort of GUI though, just to help in some way.

Daxar's picture
Offline
Joined: 11/18/2012 - 14:09

Yeah, so basically once you get it wfLZ decompressed, there's still a lot more work that needs to be done with the image. The image width and height are stored right above the ZLFW header, so I backtrack a bit to get those. (If I backtrack a bit more, I can get image pieces, and hopefully piece them back together correctly) Then, there's two DXT1 images in the decompressed data, which I extract with libsquish. And finally, once you have the color image and the inverted multiply image, you have to follow a careful algorithm (described in the page you linked earlier) to reconstruct the image while keeping the alpha. As one of your friends noticed, if you just multiply, you'll end up with no alpha channel. I'm not sure how easy these steps would have been for you to do, and the guy kept explaining stuff, so I just kept coding. : P

My program does all this for you and streams the result via libpng to a file. If you want, I can pipe the final image's width & height, then raw RGBA data (which will be of size width*height*4) to stdout so you can use it easier. The problem I ran into, and that the other guy caught, is that my program wasn't reading correctly from stdin. I'm not sure if this was me reading from stdin incorrectly, or if there was some kind of error in stdin itself or the data that prevented me from piping it. In any case, I can try giving it a shot if need be. Or alternatively you can call my program with a particular filename or output folder or such to decompress to.

The information for pieces are stored later in the anb file, and I think I know how to use that to reconstruct an image. This would also get rid of the artifacts around the edges of the images, as far as I know.