D3 RoE - extracting pk4's in order to improve FPS

[Joe Forster/STA]

Guys, .PK4 (and .PK3 and many "big files" in other games, for that matter) files are actually plain ZIP archives. There are three possibilities to deal with lots of game data files:

1. Have each file as a separate file in the file system. Very stupid idea as it eats up a lot of space (cluster slack, even with small cluster sizes, as with NTFS) and opening files takes relatively lot of time because the OS has to find the file in the file system. And this is especially true today when many games have their data in thousands of files.

2. Have all files put into a single, large, uncompressed "big file". (This "big file" can also be a ZIP archive, created with the "store" - that is, don't compress - method.) This is much faster as there's only one file for the OS and, when scanning the big file upon initialization, the game can build an internal list for all the files inside the "big file". This means that opening a file only means seeking inside the "big file".

3. Have all files put into a single, large, compressed archive. This gives you the advantage of the previous implementation and more... If you're using ZIP, you can use Info-ZIP's free implementation. And ZIP gives you a relatively good compression ratio at an extremely high decompression speed. Actually, extracting files from ZIP archives is not significantly slower than simply reading (or copying) files as the bottleneck is rather the hard disk reading (and writing) data, not the CPU running the decompression algorithm.

So, if you ask me, the best way is keeping game data inside a ZIP archive. As for preloading data, the way how files are stored doesn't matter, if you have enough RAM: all data will be loaded when the level is loaded so data doesn't need to be loaded while you're playing! (And RAM is cheap as hell anyway...!) So, I think, CrAcKo is experiencing some kind of optical illusion when he says uncompressing the .PK4 files improves anything at all...!