Oh this opens other questions, if the same movement in the game leads to different outcomes is that proof that the calculation are off by a little bit or maybe that's a limitation of the game, for example the xyz position can only be calculated up to a certain decimal, maybe 3.6645434 and the same movement next time would be a decimal off, the last 4 will be a 5
(sigh). I've already put too much time and effort to explain why this primal hack is a completely unreliable approach. I've also approximately put the same amount of effort to show that it was only the tip of the iceberg, and that without a deterministic engine, it just cannot work reliably that way, even if the engine did it itself. But hey, I won't write a wall each time, so good night all
Very interesting conversation, and very interesting programming stories First of all, yes the internal state of the simulation is huge. The vehicle's LUA state should also be included into this, which is very big too. The initial approach that we were looking into following for replay and multiplayer was the deterministic state synchronization that has been discussed above. The problems of it have already been stated, but the most serious ones that blocked the whole thing are the multithreaded nature of the physics core the floating point differences between different CPUs and the need to anyway have some kind of state saving to be able to have "keyframes" for replays (so as to be able to seek). Our current approach is to record only the state that is needed for visual representation. Think of it as recording 3d video. In this approach, the biggest % of the total data that will need to be saved are the node positions. Even so, recording 1000 node positions at 60 fps (for a single vehicle) is 720kb/sec uncompressed. That amount of data will barelly fit a single vehicle in an average internet connection. Due to that we have been researching ways to compress the data as much as possible. Our current compression prototypes suggest, that real-time compression ratios above 7x are possible for the worse vehicle movement cases. Nevertheless, compression ratios are not the whole story, because we cannot dedicate too much CPU time in encoding/decoding the vehicle streams (CPUs are needed by the physics too), so this is still an active research topic. So to sum up, our current research prototypes suggest that replay, in a 3d movie like way, can be done. Increased CPU load is a problem. We are looking into how much that can be reduced without affecting compression ratios. A final note. Due to our approach of physics simulation, we have to constantly push the limits of current technologies. For example, for replay we have and are doing a considerable amount of research in 3d compression. Due to the unpredictable nature of research, it is very hard to have a concrete timeline of when things will be ready. This applies even more to us, because we are still a small team trying to do big/complex things. So it make take some time until you see any of above commentary being released. So keep your expectations low .
I agree that on modern hardware, if the overall mechanics of multithreading in the physics simulation has not been though of with determinism in mind from the ground, then it can become a risky challenge to cope with it afterward. Only someone close to the actual engine could enlighten me on this subject, so I thank you for this. That's not to say you did it wrong, mate. You did it, that's huge enough already. It's only a crossplatform issue actually. But it's not the nature of floating-point calculations which makes it non-crossplaftorm-deterministic, it's the lack of reliable normative ways to cope with FPUs inconsistencies. This will be eventually fixed by the industry at some point both for CPUs and GPUs (I'm just angry that it has not been done already as floats is far from being a new feature in computing world). So in the meantime, one can only cope with the crossplatform issue by using other types (like integers), and I perfectly understand it can be a big deal that also should be done from the ground (in physics at least). Floats makes life so easier overall, even though they have their drawbacks too. I may add there is also a language normative issue about the guarantee of FPUs local consistency (ie. even for a single machine), but in practice this works on all gaming platforms. In short, the float issue is only a crossplatform issue. Consequently, for deterministic multiplayer it means that only players with close enough architectures can take part in the multiplayer session (I agree that's still a shame, but not that uncommon in the game industry, often for completely different reasons, and not necessarily technical). For deterministic replay, that means that a player record cannot be replayed on another "not close enough" player's rig (a relatively minor issue). Until the industry copes with it (many actors are involved, roughly hardware manufacturers and OS makers, common languages norms can wait... that should first practically work and spread), we hardly have any other choice than using other types (like integers) to completely solve the crossplatform issue. Obviously, granted a game company can't wait for this to happen. Also granted it has good reasons to use floats over integers (for their helpful properties), especially for such a degree of complexity. I already talked about that. There are some workarounds to consider, like post-processing the record file, caching states in RAM or even write them on disk every few seconds or so, or a combination of these and other tricks. Anyway, "instant T state snapshot" is a useful internal working feature of any game engine (even for a deterministic one), it has many usages. Very interesting details, I did not ask that much. I'm glad to see you're on the track. It's always a lot of work anyway. At least, now we know that replay is seriously considered (I did not except it otherwise for such a game, but we always need devs feedbacks to make sure). I guess we can safely say "it's on the track", even though we still don't know if this will be ready for the release (you neither I guess). But once released, I'm sure you're aware that players pressure will rise a little bit Thank you for the details, and anyway thank you for this game. What you're bringing there is an old childish unreachable dream of breaking things all over the place without the need to spend insane amount of money to enjoy it. That's worth the price in its current state, and a fantastic fundation for more
First of all thanks for the kind words About the determinism. The physics core has actually been developed to work in a deterministic way (it was our initial approach to replays). Most of the other subsystems (e.g. gfx engine) have not, which is problematic. It could be solved by having determinism "boundaries" between the physics/other_subsystems but it isn't easy. Post-processing the record file would take approximately as much time as it took to play it, which for long recordings is problematic. Also the "keyframes" that would need to be cached/recorded, would have to be the full physics state which is enormous. Apart from all the nodes/beams/tris it'll also have to include the LUA state which is many times more than the core physics data structures. In addition the starting world conditions would have to be exactly the same. So this would create problems with replaying recordings between different versions of the game. I completely agree with your comment about the state snapshoting being extremely useful for game engines, but that would require to have somewhat stable code/data structures. Due to the nature of the physics core, even small optimizations (in the range of 1-2%) might require to radically reorganize the base data structures (to optimize cache utilization). And changes like these happen all the time in the core...
I don't get it well. Why do you need determinism for the gfx part of the engine? In my eyes this part precisely doesn't have to be determinist (in fact, it simply can't pixel wise & time wise). You should only need to reproduce internal physics, the renderer is not related. Is it somewhat related to what you say below about the state snapshoting? Of course, that's what I said too. But it's a one time process only (GPU's power can also be used with OpenCL to accelerate physics steps calculations for that specific purpose, granted you can scale your physics to such a degree of parallelism, though beware of FPUs differences between CPU's one and GPU's one). And it would only be needed in case occasional (we need not much than few seconds apart) snapshoting during gameplay is too much overhead. Maybe it's too soon to consider it... deterministic engines games company (like Blizzard) coped with that kind of issue by always providing older versions of their game releases, so players can make use of their old replay when needed. I can understand you'd prefer to avoid it, but that's one of the weight of a determinitic system: if the game fixes bugs, these bugs obviously cannot be reproduced the same way in old replays triggering them.
Input and other information comes from the gfx side, also some heavy physics/lua computations are being done on gfx frames. GPU based physics is not something that can be done just to serve replays. Actually, moving the physics on the GPU side is a huge research/development project. The whole physics core will need to be rewritten to suit the GPU hardware (architecture, caches, scheduling and so on). Also, if it is to be done, Vulkan seems to me a better candidate than OpenCL. This is why i've noted that these things can happen in games that are in a released state (and not early access like .drive is). It would be problematic if we broke replays every other week...
I won't ask more on the subject I agree for Vulkan. I named OpenCL mainly because I've read somewhere that you were doing some R&D on it. Thanks again for details. I sincerely wish you the best success.
Nevermind... found the answer already reading on... .. Sorry for the non-contributing post. I'd remove it but that's not possible (?) --- Post updated --- This I wouldn't mind at all.. I'd make a fraps from the angles that I wanted to save, and save the movie. After this the replay could be discarded/broken.. If I know this happens every so often, I would find other ways to "keep" the replays. Positive is that you're saying it is possible... so I will be patient for now. See how it develops..
Interpolation or rubberbanding could be used to keep replays lighter. In essence it saves keyframes of node positions (and velocities if preferred, double the data though) and interpolate (basic estimation) what happens between the keyframes. It could cycle trough the nodes and only store the information about a few of them each frame and then interpolate or "smooth" between them. Maybe the replay accuracy could be adjusted with an option so that the more beefy computers could make a more accurate replay. I don't think you need a high update rate in the replay to keep it interesting to watch. 5 fps would be enough for me with interpolation to make it smooth and it wouldn't translate too bad to slowmotion either. Interpolation allows for any timescale and still keeping it smooth. A crash is generally somewhat linear too and five updates a second should be sufficient, no? Sorry about the bump.
When I said that keyframing could be recorded only few seconds appart, it implies interpolating between them (especially without determinism, of course). I'm sure devs already considered it anyway. Slow motion would likely exacerbate the trick though. Why? Most people would be glad to have a replay in this simulation! AFAIK, this is the only thread were devs answered to the question: do you plan to implement replay? Yes, they clearly do. They certainly have other priorities for now, but replay is on their TODO list. People who need an official answer can find it here, thus this thread deserves to be bumped from time to time, or referenced at least.
