43gb of BeamNG's 46gb file size comes from the zip files that hold all of the game's assets. Almost all of that data is composed of uncompressed DDS bitmap texture files. Crucally: All of those zip files have compression turned OFF. They're zips all right, but compression is disabled, so all those bitmaps are stored as raw data. This is extremely inefficient. My suggestion is to compress all of the game's zipped assets using normal zip compression. What does this do? 1: It reduces the game's file size from 45gb to 22gb - meaning the game takes half the disk space. 2: It does not impact loading time on fast drives, and on slow drives it will actually speed up loading. 3: From all of my testing, it doesn't introduce any bugs or require any modification to the source code of the game in any way. I tested this idea out by writing a script that used 7Zip to extract and re-compress all of the game's asset files. It took a little while to load, but it worked. File size was halved, and I timed the loading speed and found it was exactly the same, if not faster. I did not change any of the game's code, scripts, or assets at all. I just extracted and re-compressed the asset files. This does not require any modification to the game's code, since BeamNG uses the zlib module which is already able to decompress zip files encoded using the normal DEFLATE algorithm. DEFLATE is already used to compress most of the community's modded game assets, so there is no compatibility reason not to use it on the vanilla game assets themselves. I was able to run my modified client just fine, with no compatibility errors or crashes, even when combining the modified client with other community modifications. Using compressed assets should speed up loading times because the compression allows for files to fit through the disk drive's bottleneck faster. For example, a 1Gb uncompressed file on a 500Mb/s SSD should take two seconds to read, but compressing the file to ~550Mb should reduce reading time to just over a second. In theory, compressing the assets should allow the client to load up to twice as quickly as before. In practice, though, the actual unpacking of assets seems to be the real bottleneck for my system, but any hard disk with a throughput of less than 500Mb/s should see a significant benefit from compressing the assets. In my testing, the game worked exactly as before, with no overhead and no performance drops or lag spikes. My game did not crash at all during the two weeks I tried this out, even when using mods. The module that handles decompression is already installed in the game! I didn't need to change anything! Unfourtunately, Steam replaced my assets with the uncompressed versions because they didn't verify. I've tried enabling NTFS compression instead, but it severely bottlenecks my loading times because it's single thread only, and can only compress BeamNG to ~75% of its normal size (7Zip can take it all the way down to 44% - nice!). NTFS compression also causes Steam to break, because it can't figure out how a 45 gigabyte game can fit into a 32 gigabyte drive. I used the DEFLATE compression algorithm with a word size of 128 bytes (7Zip's "zip, 9 - Ultra" preset) in my tests. TLR - Enabling compression on the game's zip files cuts the game's size in half and speeds up loading on slower systems. Please add this, or explain if/why it's not possible!
Using compression is not necessarily free and beneficial. If your computer is bottlenecked by CPU, then using compression will slow it down. Also, compression can make life harder for Steam's update delta generation system, potentially leading to larger downloads on each update (a possible concern in countries where data caps exist). The answer is not as clear-cut as it might seem, and would require us to deviate development resources to this topic, so we can properly research the true pros/cons under a variety of circumstances.
The delta generators should be okay, in theory. Since each file in the zip is compressed individually before being appended to make the zip file, a small change to a single subfile should still keep the rest of the zip more or less identical in terms of delta values, though. It'll still have to rewrite the rest of the subfile, but that's still an order of magnitude below rewriting the file. Adding new files to the zip will not mess up the delta. You're right, though - I hadn't thought of that problem. gzip (and thus zlib) decompress uncompressed and compressed zip files at the same speed using the same CPU power, for some reason. Try it! Uncompressed zip files are faster to create, but all zip files decompress at the same speed. It's just the nature of DEFLATE. It still seems a little strange to waste 20gb, but I understand there's a mountain of QA that would get in the way. Thank you for your response!
Okay, a longer answer. But I need to get back to work There are many factors at play. You have game loading times, update download sizes (internet data costs or limits, and also download times), update patching work (wear on people's storage drives, and of course patching times), runtime overhead (the use of zip files is not free, the system will be faster with zero zip mods, than with 5000 active zip mods), incorrect modding concerns (the use of zip files can act as a deterrent against modding the game incorrectly (people lose their modifications more easily, and to make things worse it increases the workload of people who do support tasks, reducing their time for other tasks). There's probably more factors I forgot. Depending on how exactly you balance the priorities of all the factors, then the optimal solution can change from having no zip files at all, to having a single huge zip file for everything, or having a bunch of zips for groups of content (current solution), or even having one dedicated zip file per individual file. On top of that, you also gotta choose how much compression to use in cases where chose to use zip files (atm we chose to use no compression). This is a topic we're reevaluating easily 2/3 times a year. The current solution is not merely a coincidence, but the result of careful evaluation and re-evaluation of all those factors. If we only cared about game loading speed, it would be an one-off easy decision. In any case, we are humans and can always be wrong in our assessments though, specially with complex topics like this, so we do value additional input Edit: if this post seems at points contradictory with my previous post, it's because I've had to completely switch context, from <unannounced feature> to the topic of update headaches. I didnt' remember all the details at first
@stenyak Thank you for the informative post. I would have two further questions if you dont mind: Your posts suggests that zipped mods can have an more negative performance impact than non zipped ones. So from a performance standpoint one could say: - No mods - highest performance (obviously) - Unzipped mods - less performance - Zipped mods - lowest performance Correct? Seing that Repo mods come zipped automatically I would assume that unzipping those would be a bad idea as it would likely break automatic mod updates which probably compares zip archive checksums. But unzipping non Repo mods could possible have some slight performance gains? Thank you for your time and Happy Holidays.
Performance assesments will vary depending on hardware specs. Slow HDD but very fast CPU? using zips with maximum compression might be fastest. Slow HDD and slow CPU? Maybe zips without compression. Fast SSD and slow CPU? Maybe no zips. Etc. Still, that's if all we cared about was loading time performance, and we ignored all the other factors like how long it takes to patch the game on each update, etc.
The delta generation system is far from perfect and it comes with a drawback: wearing down the SSD As shown, the "patching" is 33.3GB. If the files are compressed patching won't be anywhere near 20GB, also zip compression by it's nature, is deterministic and lossless, therefore less file size, less delta size. I upgraded my hardware and now using a PCIe gen 4 SSD with more than 6000MB/s write speed. Only then I'm able to cut the "patching" time significantly. However this is an overkill and playing (and updating) the game shouldn't require this high-performance disk in the first place EDIT: However, this is relatively less concern as Modern Standby on Windows 11(and some machine, 10) is known to wear down the SSD much faster. I guess we should focus more on Linux supports.........
