Multithreading

[zebany]

As i know, RunUO uses a single core thread to process input in linear order and uses threads for networking and timers. As timer and network io threads are more lightweight threads, core thread does almost everything. I understand that's a design choice and works great so far.

But since cpus becomes to have more cores every single day, developers are forced to implement parallel execution as much as possible. I mean, not simple as "a gui thread, and an everything else thread".

I was thinking about how RunUO would be coded to use threading as much as possible and i could not find a solution. I mean, let's say we have lock statements on every single property on our mobile and item classes, and use a thread from pool for each client packet. There would still be lots of deadlock issues, and even the performance would decrease because of all that synch code. There might be seperated threads for AI or something. Or maybe world partitioning like seperated main loops for each facet.

I don't think its possible to make RunUO fully multithreaded at this point, but just wondering how it could be done if you guys were coding RunUO today, from scratch. Not asking if it would worth doing it (i don't believe it would), just how it could be done

I learned too much stuff about c# from RunUO and i'm trying to learn about multithreading now. And i appreciate RunUO community's ideas. That's why i'm asking.

[Jason]

RunUO is multi-threaded....

It would not be effective to break up the core thread, it would cost more cpu time to merge the thread, and you would experience perhaps worst performance because of that fact.

Making a thread, and keeping track of it, and merging x threads together costs cpu time.

[zebany]

Hi Jason,

Yes i know it would not worth doing it. And yes RunUO "uses threads" but it runs on mostly one thread. It is not an http server that can use a thread pool and process every single request on its own thread.

So, let me change my question;
When cpu vendors starts selling 4core, even 16 core cpus, developers will need to write servers with parallel processing capabilities. Because then, there will be an advantage. So what would be the easiest way to achieve multithreading?

[Malaperth]

RunUO does use thread pool...

[zebany]

Originally Posted by Malaperth

RunUO does use thread pool...

Where??

Talking about async io? Threaded save method? What? RunUO uses thread pool, not for %1 of what it does.

No, it doesn't use thread pool for game loop.

Timer system runs in a thread, matches expired timers in a seperated thread and passes them in a synch queue to be executed in "main thread". So all timers "OnTick" methods runs in the main thread.

Network system uses threading. It uses socket's beginread and beginwrite async methods, so they are executed in async io threads automatically, reads data, passes them to message pump and it parses them in "main thread" and executes packet handlers in "main thread".. Server generates a packet in "main thread" and writes it to client socket in async io thread.

Still, much of that work is done in main thread. It's not possible to use blocking sockets for a server like runuo, so its not an option not to use threads for network. Timer system on the other hand, thought well and works great on a seperated thread.

I'm not talking about how runuo's single thread model is bad. Its efficient and easy to debug / write. No deadlocks, no synchronisation problems and stuff.

I just wanted to ask for your opinions for a multithreaded system like runuo.. How could it be done easily? Using an rdbms on backend? Or synchronising everything manually?

[Ray]

Originally Posted by zebany

Where??

Talking about async io? Threaded save method? What? RunUO uses thread pool, not for %1 of what it does.

• Timer thread
• Networking
• Compression (ZIP)
• Database (MyRunUO)
• External Systems (Pandoras Box Server)
• Threaded (Async) save

Just what I've seen so far. You're right, thats not 1% of the game code, but it's the most delaying part.

Originally Posted by zebany

I just wanted to ask for your opinions for a multithreaded system like runuo.. How could it be done easily? Using an rdbms on backend? Or synchronising everything manually?

Usually, parallelism and 'doing it the easy way' exclude each other
A massive parallelism would make it vary hard to keep the code threadsave. Imagine how many scripts out there would cause problems with that, just because (i.e) there's an overloadable method in the Timer class that allows you to execute code in the timerthread instead of the main thread. A Beginner can hardly understand the difference of accessing references like items in their main thread or in another one.


Even with multicore systems, performance is not all about parallelism in one application. There are still other processes running on a system and the main goal of a server is to get interrupted as little as possible. A too massive use of multithreading and involved locking would cause the other way. A developer has to decide if a thread is worth to be created. That means: how much is the cost of the operation itself, compare it to the cost of creating the thread handle, the synchronization to and from the handle, the locks that can happen due to this and even the possible contextswitch of the OS that can block out the entire operation.

There are not that many operations that are worth to have their own thread, most of them are to decrease delay of networking, as this is the biggest bottleneck in every/most server system. Another aspect is to check if the operation is standalone or not, as standalone operations have very little synchronization costs. Say, the A* pathfinding algorithm could be excluded to another thread, you just have to define a start and an ending. As long as the map is unchanging, only using map and statics as obstacles, this would be a standalone operation that greatly benefits from multithreading because theres only a synchronization for input and output.
Unluckily, there are dynamic obstacles too, so there are has to be more synchronization on the whole world which renders this to uselessness.

Anyway, i've never had problems with runuo where i was lacking cpu power on 1 thread. Mostly it idles 'til the next worldsave even with a hundred of players

[Jason]

My small understanding of multiple threads, would tell me that there is no easy way to make RunUO act like Apache where each user is a seperate thread for the simple fact they have to intereact so often.

Like I said the cost of talking to another thread would make it not worth it. What will happen is saving will get faster, and other parts of the the backend can be improved. But the main thread will always be limited to the one thread you see, at least thats how I see it.

main thread = players walking around talking to each other.

The only thing that might happen, which would make since, is when instance dungeons is added to the distro. This would be a perfect time to add "threads" where group of players might talk to one another. They would then have to recenter the main thread, it would have to be fast enough( faster then 60 ms ) since loading times do not exist in this game.

The there is also the fact multiple threading is really a new programming design, and when the compilers automaticly handle multilple threads, since we have 64 CPUs on every machine thats when you would see what your talking about.

* 64 is just an extreme case, used to point out at some point we won't even "program" with multiple threads, it will just happen. Since we would have near unlimited resources for everything. We got awhile to go, although 2008 is suppose to introduce 16 cores

[Zippy]

I would disagree with the 1% statement. Maybe < 1% of all code is run in different threads, but as someone else already said the majority of "slow" statements (saves, networking, compression).

I would not change a lot about how RunUO was designed. The reason that RunUO is significantly single threaded is because the vast majority of code is user produced. Not to be negative to the user base, but I would say (liberally) < 5% of people who script for RunUO would be capable of understanding the complexities involved in a massively threaded system. It would be hugely limiting the what novice users could accomplish. Given the language of the scripts is well defined and well outside of our control we cannot implement entire systems to handle threading/sync for the scripter "under the hood" like other systems (POL, UOX, Sphere) can do (but, as far as I know, largely do not).

That said, a significant bottleneck in the current core is map access, particularly multiple accesses involved in movement (including boat movement). Rewriting the map code to be reentrant and switch to a call back sort of model (like "StartCheckingLOS, and call this function when you're done") would be beneficial.

But I don't think its a huge deal or I would have done it already. The fact is huge shards like UOGamers even with their massive numbers use a low amount of CPU the vast majority of the time. Especially with 2.0's optimized main loop. There doesn't seem to be a huge need for more threading because RunUO scales extremely well without it.

[Zippy]

Originally Posted by Jason

My small understanding of multiple threads, would tell me that there is no easy way to make RunUO act like Apache where each user is a seperate thread for the simple fact they have to intereact so often.

Like I said the cost of talking to another thread would make it not worth it. What will happen is saving will get faster, and other parts of the the backend can be improved. But the main thread will always be limited to the one thread you see, at least thats how I see it.

main thread = players walking around talking to each other.

The only thing that might happen, which would make since, is when instance dungeons is added to the distro. This would be a perfect time to add "threads" where group of players might talk to one another. They would then have to recenter the main thread, it would have to be fast enough( faster then 60 ms ) since loading times do not exist in this game.

The there is also the fact multiple threading is really a new programming design, and when the compilers automatically handle multiple threads, since we have 64 CPUs on every machine thats when you would see what your talking about.

* 64 is just an extreme case, used to point out at some point we won't even "program" with multiple threads, it will just happen. Since we would have near unlimited resources for everything. We got awhile to go, although 2008 is suppose to introduce 16 cores

Its important to note that a multiple threaded application running single processor machine will still usually see a lot of benefit.

In computer science, tasks (things threads do) can be classified as "processor/processing bound" or "I/O bound." Processing bounded applications are usually only mathematical computations and simulations, stuff that uses a relatively small amount of information to compute something complex (like an FFT). Most everything else is I/O bound. I/O bound means most of the time the application is waiting for an external input device (a hard drive, main memory, network card). While it's waiting for that result the thread cannot do anything. A multithreaded application can effectively suspend that thread until the I/O operation is complete and continue to do other things.

So, multithreading is well understood and has been around for a long time (a multitasking operating system is a close cousin to this, and these OSes have been around for tens of years).

Still, multiple processor systems help raise the issue of multithreading in the minds of developers. Also, multiple processor systems allow for processor bound multithreaded applications. So yay for that, too.