PC Alpha 21 Discussion Overflow

Just an idea: One way he could achieve this would be moving calculations to the clients. For example **IF** zombie pathing currently is all done on the server, in his mod each client might do the pathing for the zombies summoned by that clients player.
Zombie pathing is already cheap and runs off the main thread. I did fix a bug last week with only 1 path per frame being sent to the threading system and since it typically only takes 1 frame to make a path, it was a bottleneck for say 100 zeds making paths each second. A fairly simple change now has it requesting up to 8 paths per frame of which the threading system can provide about 6. This tested well with around 150 zeds on a server spread across 6 players.

 
Just an idea: One way he could achieve this would be moving calculations to the clients. For example **IF** zombie pathing currently is all done on the server, in his mod each client might do the pathing for the zombies summoned by that clients player.
I was thinking something like that, except rather than handing off to the client, put it on a brand new process (their own) which would replace the inbuilt process.

The problem with moving to the client then, is "which client if you have two in proximity"? Now you've gone from just patallel processing to distributed computing! Fun and games for the whole family! 

Oh well, time will tell. I am in no rush, as I have a very powerful dedicated server for 4 players, but if the patch worked as "advertised", (or really, just reduced load without EAC disable/heavy client side requirements etc.) I'd make all 3 of my servers public (vanilla, DF, and Sorcery).

Edit: sorry, missed that bit about controlling the client's summoned Zeds. Still stands that you need to send the info back and reassemble the info and resolve any conflicts before sending updates back out to the other players. With the 50+ players he's claiming, that's still quite a feat.

 
Last edited by a moderator:
Yeah, I can't imagine they do it except between traders.  I wonder how they would manage vehicles... Do you have to pick it up to take it with you, or having it on the helipad is enough?  We shall see.  Since they did remove it, that means it is probably guaranteed to appear.  No reason to remove it otherwise.  Of course, maybe they just don't want to have false hope out there. 😁
It should be no surprise to anyone that we discuss fast travel as many games have it, hence the helipads. The vehicle teleport is one of my main issues. We think the vehicle should go with you without it being annoying. I'm thinking it teleports to an open parking like area and finding a free spot. I'd prefer it be a very limited system, but it could end up as whatever makes sense or not doing it at all. It could easily not be in A22, yet show up later.

 
It should be no surprise to anyone that we discuss fast travel as many games have it, hence the helipads. The vehicle teleport is one of my main issues. We think the vehicle should go with you without it being annoying. I'm thinking it teleports to an open parking like area and finding a free spot. I'd prefer it be a very limited system, but it could end up as whatever makes sense or not doing it at all. It could easily not be in A22, yet show up later.
That makes sense.  You would need a good sized area where the vehicle can spawn because with a lot of players for servers allowing more than the normal max of 8, you could have a bunch of stuff in the way.

A few possibilities:

You could make a temporary holding spot for the vehicle that requires you to put it back down before leaving the trader's compound.  If you don't put it down, you're stuck in the compound.  And the temporary holding can hold a vehicle that has things in inventory, but can only be used automatically during fast travel so it doesn't allow filling a vehicle and then carrying it around with you.  You could tag a single vehicle to take with you that is within the helipad area.

Perhaps a better option is that when you arrive at the new location, you are automatically placed on your vehicle so you're taking about the same space (a little more) as if you don't have a vehicle.

Another option is to have the radio or some other device include an "inventory" that shows your vehicle that you chose to bring with you only to you and you would then click on it and click a place to put it on the ground.  It would never go into your inventory and would not need to find its own placement when arriving.

 
That makes sense.  You would need a good sized area where the vehicle can spawn because with a lot of players for servers allowing more than the normal max of 8, you could have a bunch of stuff in the way.

A few possibilities:

You could make a temporary holding spot for the vehicle that requires you to put it back down before leaving the trader's compound.  If you don't put it down, you're stuck in the compound.  And the temporary holding can hold a vehicle that has things in inventory, but can only be used automatically during fast travel so it doesn't allow filling a vehicle and then carrying it around with you.  You could tag a single vehicle to take with you that is within the helipad area.

Perhaps a better option is that when you arrive at the new location, you are automatically placed on your vehicle so you're taking about the same space (a little more) as if you don't have a vehicle.

Another option is to have the radio or some other device include an "inventory" that shows your vehicle that you chose to bring with you only to you and you would then click on it and click a place to put it on the ground.  It would never go into your inventory and would not need to find its own placement when arriving.


I would guess those solutions are all too much special code for something only sometimes relevant in unsupported configurations. I think the teleport will just not work if all spots are filled. Or the car is teleported out of the compound just like players at night (because the code for that is already there, it just needs to be turned on for vehicles as well)

 
It should be no surprise to anyone that we discuss fast travel as many games have it, hence the helipads. The vehicle teleport is one of my main issues. We think the vehicle should go with you without it being annoying. I'm thinking it teleports to an open parking like area and finding a free spot. I'd prefer it be a very limited system, but it could end up as whatever makes sense or not doing it at all. It could easily not be in A22, yet show up later.


What the....

image.png

 
I would guess those solutions are all too much special code for something only sometimes relevant in unsupported configurations. I think the teleport will just not work if all spots are filled. Or the car is teleported out of the compound just like players at night (because the code for that is already there, it just needs to be turned on for vehicles as well)
Turning that on for vehicles would be nice.  Especially if you have a larger trader.  Will be nicer to drive into it with it being larger, but it's a pain if you get kicked out at night with your vehicle still inside.

 
We think the vehicle should go with you without it being annoying. I'm thinking it teleports to an open parking like area and finding a free spot.


If you are talking about teleporting vehicles outside the merchant's base when night falls, then in this case the parking spaces should be located outside the merchant's base.

If you are talking about moving between merchants along with vehicles, then parking spaces should also be located outside the merchant's base, since the teleport can occur a minute before the merchant closes.

One way or another, the helipad is not suitable for these purposes, since it is located in the area of the trader's base.

And why then do we need a helipad, this is a big question )

 
Parking spaces? The entire world is your parking space. Park wherever you want inside or outside, on the street or in the ditch. I don't think there are any formal parking spaces anywhere.

 
Parking spaces? The entire world is your parking space. Park wherever you want inside or outside, on the street or in the ditch. I don't think there are any formal parking spaces anywhere.
For the correct teleport of the player's vehicle so that he is not lost, this is probably necessary)

And no one will write a fine for wrong parking )

 
Zombie pathing is already cheap and runs off the main thread. I did fix a bug last week with only 1 path per frame being sent to the threading system and since it typically only takes 1 frame to make a path, it was a bottleneck for say 100 zeds making paths each second. A fairly simple change now has it requesting up to 8 paths per frame of which the threading system can provide about 6. This tested well with around 150 zeds on a server spread across 6 players.
Ohhh?  What I noticed was your animation rate versus physics updates and transforms are all done OOP. This can be spread out to help the runtime load. Making this multi thread is very difficult with the way it is coded however I was able to separate phases for players in specific so far. I took over the gmupdate from the unity update tick and separated some of the work. Specifically the entity update tick. Your thread pool is set too high. Way too high. If most of the game is single threaded, then having a thread pool do that will toss the threads into the pool waiting for work that almost never shows up. This can cause IO handling problems. This is my own opinion but the pool should be slightly larger than the work orientation within the code. If the game is only coded to use 2 cores in general, then 1200 threads in a pool will be an issue.

I also like to run work simultaneous along side the main, rather than isolated to only threads. This keeps a better work flow and the data has a better potential to be closer in the data stack when passed off to the other threads by the main which then runs part of the operation.

I am isolating the Entity.OnUpdate from the EnityAlive.OnUpdateLive which contains the heaviest calculations. Both can be multi threaded by dividing the work evenly with the players being split among them. If the Entity.OnUpdate runs at the same time as EnityAlive.OnUpdateLive, then the overlaps can cause issues. Isolating them into phases prevent this and I highly recommend it for all of the entity. I run the Entity.OnUpdate phase multi threaded, then the live section multi threaded. There are a few things you have to pull out of this process and run isolated to avoid overlaps with multiple players updated simultaneous. 
updateCurrentBlockPosAndValue and checkForTeleportOutOfTraderArea. Ideally these would be inside the position update phase. The buffs, inventory, bag, stealth and updateprefab sections need to be isolated, plus those must be done single thread. If some of these can be written for thread safety, it can all go multi thread while keeping your OOP process. This would avoid a total rewrite of the game while getting a giant advantage out of the multi thread.

Here is how I mapped the isolation phases

Positions > Chunks > Buffs-Inventory-Bag-Stealth-Prefab > Entity.OnUpdate > EntityAlive.OnUpdateLive > One offs that must avoid overlaps like CheckSleeperTriggers
4x               1x             1x                                                         4x                            4x                                         1x
1x = single thread
4x = four threads

Not sure if I measured this improper but your runtime with a single player in game on the dedicated servers was around 0.05-0.06 which is honestly atrocious. The unity update tick rate is 0.01. 5 runtimes missed from a single player. I hope I measured it wrong. After taking over the gmupdate and doing some tweaks for dedicated and taking over the entity update phase entirely, I have it down to below 0.01 with one player in game. Actually it was closer to 0.001.

I can go into a bunch of other things if you are interested.

Due to the runtime you are averaging and how much faster I can run the update, I can run multi runtime ticks in the same time frame, so I split the work across them. It actually was running so fast at first when I left it running every tick, everything was moving around like the flash. Chickens were red blurs. Breaking the chicken sound barrier. Beak neck speeds. Now I am updating across ten ticks. Zombies are in very nice animation flow while keeping their tracking more accurate. I also split them across four ticks. I update the players twice to keep them in better animation, network and update state with others around them.

You can reduce the updates across multiple ticks while keeping your OOP but splitting it to phases to allow multi threading too :)

You can also use flat buffers for the entity update phase which helps avoid some of the garbage collection.

I did some work on block collapse to make it rapid and look like it crumbles.

Chunk sync to unity has four ticks which dramatically helps in with loading large volumes and high player counts.

I run the entire gmupdate every tick like normal, it is just the entities update and chunk sync to unity that can be spread out across the ticks/frames
Tick 1 and 6 update players. Tick 2 3 4 5 is for non players and single thread for now(Yes it can be multi threaded but more difficult)
Tick 7 8 9 10 is for chunk sync to unity.

Keep in mind, I do that in 0.1 where the normal game right now is updating one player in 0.05-0.06. I do what I can to keep the runtimes under 0.01 to match the unity tick rate. The closer to that you can keep it, the better the overall fps. 

We were able to achieve 300 zombies and 70 fps on a high end server with multi threading. There are huge race conditions and overlaps that make non players extremely hard to multi thread.

You have a few sections using static lists inside of the class which is the only thing stopping multi thread. To avoid the garbage you need more lists allocated upfront or allocate them at runtime if not worried about the gc.

The AI is, well I will be honest its a bit ugly. It must be done single thread currently. I can achieve multi if I alter those static lists, however it hits a duplicate path id error and then spams the console. I suspect it is an issue in the library you guys are using for the pathing algo but I could be wrong. The good thing is its fast. Spreading the work load across more runtime ticks allows the ai to keep up despite being single thread and the changes you guys have made for a21 is a big deal. That single path check instead of eight will help a lot too. I am avoiding touching ai since you guys are updating it but I will let you know if I find anything

 
Last edited by a moderator:
Tell us what you really, think, don't hold back! 🤪

Wow, does seem like you've really done some really serious work reverse engineering this. I hope the devs can use some of this to improve game performance. 

 
Tell us what you really, think, don't hold back! 🤪

Wow, does seem like you've really done some really serious work reverse engineering this. I hope the devs can use some of this to improve game performance. 
I think they are doing a great job and have taken on a behemoth task. Probably not a huge consideration early on in the dev cycle but this late, I imagine not using multi thread is hampering the game potential. I would be stripping out things and 'streamlining' as a way to get the fps higher at this point just to fit in content. In reality, its a combination of things. They started this game when Unity was single thread and had no support for multi threading. Unity stopped supporting proper netcode for years and mid development of the game so the Pimps have done a really good job. Bravo Pimps.

I hope this can help out in some manner, maybe a small change in the approach of application from the strict OOP. They can keep the pattern but try to adjust it towards phases which would allow multi threading those particular phases as OOP processes. Each core gets a split of the entity list to update and handle on its own for each phase. Differ the work when possible. Utilize the runtime ticks to spread out updates. Not everything needs to be updated every single tick so simple pacers forcing it to only trigger every second or third tick is already 50% less work and retains their current pattern. If it keeps the game flow in tact, then they are really blowing their load a bit as they reprocess things over and over with no real purpose. That is the nature of OOP, so it is not their fault. What I am doing breaks the rules of OOP.

It is really hard to use the modern unity features of the engine like the job system since that really should be coded using ECS patterns, not OOP. They have also missed using pooling in some cases. Whenever they can be used, they really should be. What can you do at this stage though? You cant rewrite it all without breaking everything. I know optimization is not their focus but I feel they have missed the boat a little bit. Some of this is not really optimization so much as engineering and coding patterns. Although I am calling this an optimizer since that is what it achieves and does indeed have optimizations for ram reduction, in reality it is borderline an engineerizer lol and using harmony injections to achieve it makes it 10x more difficult.

It will be made public once I can sort out some interesting bugs that happen due to multi threading. This post is already too long. Maybe I will write up an entire process of how I did it and the complications along the way. A programmers journey to hell
 

 
Hypothetically speaking, if a mod like this did exist would it work on SP games also.

I have only seen it mentioned having to do with multiplayer servers.
While it is possible, it is another layer of headache. The local player aka local host when self hosting or client while only single player must be updated on the main thread. This is due to some rules in Unity such as mesh changes, chunk syncs, ui alterations and a few other things must be done strictly inside the Unity engine. The main thread is the only thread capable of passing data in and out of the Unity engine. However that means the other players can still be multi thread and get all of the advantages.  They should be able to handle a lot more than 8 players if the work load is spread and multi threaded using some phasing techniques.

Non players are another beast I am still working on. Some of it can be safely isolated and multi threaded. In particular the entities that are considered alive such as zombies, animals, vehicles, traders. These can all get isolated and multi thread for their physics sections among some other tricks. Non live entity, not so much. The way the game is coded would be very difficult to do that. I really want to avoid anything that alters their form and pattern too much while still getting significant multi thread boosts.

I would need to code a version for single player users in particular but I don't see why it could not work. That is something I can look at if people are interested. I am sure you want to see some results first with the dedicated optimizer. It is a little hard to believe it is possible or that the results are as good as stated. All I can say is they are real. You might not see the same level of improvements seen on a dedicated but you will definitely see a dramatic stability increase.

Here I am self hosting a dedicated and running a client on the same hardware. This impacts the total potential since the server starts to lose fps when the client is having trouble with hundreds of zombies all at once. Still managed over 200 zombies and a super high fps. This is before client side optimizations and just zombie and players using multi threading. I could not get the zombie stable enough on multi thread yet but this gives an idea of the potential. It is very much plausible to do. I can only get player multi thread stable so far. Once all the hiccups are sorted out I will push it further and see what is possible

OptimizerSampleDedicatedOver200Z.jpg

I've been around and I have no idea of whom you're talking about. Why not name the mod author and post a link? Maybe @faatal would like to see what they did and whether it can be utilized? Why keep it a secret? That's not good news....that's just saying there is a secret.
psst, its a secret. You know me. It is not ready for public release yet. I want to squash some more bugs before making it public. It has run on a few of the biggest american and euro servers on and off the last few months. I had to do some work on ServerTools first but I am back to work on the Optimizer. If you don't remember me, you called me an arm chair developer a few years ago

 
I think they are doing a great job and have taken on a behemoth task. Probably not a huge consideration early on in the dev cycle but this late, I imagine not using multi thread is hampering the game potential. I would be stripping out things and 'streamlining' as a way to get the fps higher at this point just to fit in content. In reality, its a combination of things. They started this game when Unity was single thread and had no support for multi threading. Unity stopped supporting proper netcode for years and mid development of the game so the Pimps have done a really good job. Bravo Pimps.

<snip>
Kudos to you for the effort you put in, and the recognition of where the inefficiencies and legacy constraints come from. 

If you ever do get around to documenting the journey, I'd be keen on reading it. There are a few topics that are off the edge of my area of expertise, but close enough to still be fascinating to me. I speak regularly with kernel programmers and hardware designers, and specialist algorithm designers. I understand little of what they do, but enough so they aren't speaking gibberish, and I generally find it fascinating. This would be one of those areas I'd be keen on reading up on.

 
Back
Top