You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen,
maybe he can introduce the second phase in compression, then he can store forward reference count for a stream that should ideally decrease requirements memory size beyond window size and that should increase compression too.
HDDs are very slow, I recently upgraded to nvme based storage and I can feel the speed.
23-05-2020, 22:40
Threaded Mode