Server Freeze (99% CPU use) - Runuo 1.0RC0

[Ingvarr]

When server freezes, the network utilization is 0%

This problem is "mistery"... really.

The bug iitself not a "mystery" to me. Just a RunUO core bug (in one of the threads execution stucks somewhere behind Server.NetState.OnSend() )

Hovewer I have no idea how to solve this problem since we have no source code for this core part, and this bug doesnt happen on dev's shard.

No idea what hardware/software/network condition triggers it either.

 

[seph]

The bug iitself not a "mystery" to me. Just a RunUO core bug (in one of the threads execution stucks somewhere behind Server.NetState.OnSend() )

Hovewer I have no idea how to solve this problem since we have no source code for this core part, and this bug doesnt happen on dev's shard.

No idea what hardware/software/network condition triggers it either.

:-(

This problem is very very very very dificult to work fine.
Please, open de CORE!

Thanks

 

[Phantom]

:-(

This problem is very very very very dificult to work fine.
Please, open de CORE!

Thanks

This won't happen, not like you would be able to fix it anyhow.

I have reported this thread, as its going towards the crying and bitching of people who want RunUO to be open source which will never happen.

 

[Ryan]

This is probably a piss poor script customization somewhere, this is not a RunUO core bug. Back in the previous betas (to RC0) we had an LoS bug that caused 100% CPU spikes but it was fixed in RC0.

Thanks,
Ryan

 

[seph]

This is probably a piss poor script customization somewhere, this is not a RunUO core bug. Back in the previous betas (to RC0) we had an LoS bug that caused 100% CPU spikes but it was fixed in RC0.

Thanks,
Ryan

Please, I am here searching for a help...

How debug this ?

The RunUO dont have any way to debug scripts ???

 

[cward]

300MB Ram useage with 130 online players? I find that hard to believe. I use that much with 40 players and 4 fully spawned facets. I think you either have some cheap RAM, a Cheap processor ( Celeron or low grade AMD ), or not enough RAM. I would also check the pagefile size and delete inactive accounts.

I had the same problem that you are having and had to get more RAM to fix it.

*Edit*
Heres the problem.

- Looking about memory and CPU: 80~99% CPU, 657MB RAM in use

.Net will only use about 60% of the RAM after thats gone it eats your CPU and freezes your shard. You are useing 64% at shard freeze. Get more RAM and your problem will be fixed. Read this thread.
http://www.runuo.com/forum/showthread.php?t=42163

This is the exact same thing that happened to me, if you don't want to get more RAM the clean your shard up. If you don't want to do either then you'll keep freezing.
If you can keep your shard RAM useage a lil below 600MB then you shouldn't get lag or shard freezes.

 

[seph]

300MB Ram useage with 130 online players? I find that hard to believe. I use that much with 40 players and 4 fully spawned facets. I think you either have some cheap RAM, a Cheap processor ( Celeron or low grade AMD ), or not enough RAM. I would also check the pagefile size and delete inactive accounts.

I had the same problem that you are having and had to get more RAM to fix it.

*Edit*
Heres the problem.

.Net will only use about 60% of the RAM after thats gone it eats your CPU and freezes your shard. You are useing 64% at shard freeze. Get more RAM and your problem will be fixed. Read this thread.
http://www.runuo.com/forum/showthread.php?t=42163

This is the exact same thing that happened to me, if you don't want to get more RAM the clean your shard up. If you don't want to do either then you'll keep freezing.
If you can keep your shard RAM useage a lil below 600MB then you shouldn't get lag or shard freezes.

Hmmm interesting...

But... Can I change the amount of Memory or set Virtual Memory to use of .Net ??
I have 1Gb RAM and 1.5GB Virtual memory...

 

[cward]

Thats not going to help much, I'm sure you have at least 1000 accounts that haven't been used in months. You'll be amazed by how much RAM wiping inactive accounts will clear up. Either way you have got to free up some RAM or get some more RAM.

 

[Ingvarr]

This is probably a piss poor script customization somewhere, this is not a RunUO core bug. Back in the previous betas (to RC0) we had an LoS bug that caused 100% CPU spikes but it was fixed in RC0.

Thanks,
Ryan

Heh thats why I say it will never be fixed.

Really Ryan, I've located the thread that stalls the server in such occassions long ago. This is not the main thread and not the timer thread, this is one of the packet handling threads. The thread that stalls everything does some omnious processing inside Server.Network.NetState.OnSend - how can this be the custom script? I cant even look what exactly it does and never quits doing, because there is no source code for this method. And this because this is part of the core.

Also I removed all custom scripts besides absolutely elementary ones (that just tweak some constants as gain speed and such). But as I said, I am not a stranger to debugging (to put it mildly), and if this was the ordinary custom script bug, I'd caught it long ago.

Also about memory. It uses only 400-450 Mb, we have 1Gb, this is not the "no memory condition", even for .NET. Maybe adding more memory can postpone this bug, but I hope that source of it can be eliminated in first place.

 

[seph]

Heh thats why I say it will never be fixed.

Really Ryan, I've located the thread that stalls the server in such occassions long ago. This is not the main thread and not the timer thread, this is one of the packet handling threads. The thread that stalls everything does some omnious processing inside Server.Network.NetState.OnSend - how can this be the custom script? I cant even look what exactly it does and never quits doing, because there is no source code for this method. And this because this is part of the core.

Also I removed all custom scripts besides absolutely elementary ones (that just tweak some constants as gain speed and such). But as I said, I am not a stranger to debugging (to put it mildly), and if this was the ordinary custom script bug, I'd caught it long ago.

Also about memory. It uses only 400-450 Mb, we have 1Gb, this is not the "no memory condition", even for .NET. Maybe adding more memory can postpone this bug, but I hope that source of it can be eliminated in first place.

I dont have any chance to remove accounts, items or statics from my shard! This is strange, because a BUG em .net, never will be done about it ?

Well... I think this is a serious BUG and a big limitation to RunUO. This is very BAD to US, users of RunUO, and we dont do anything to solve or "workaround" this... however... the server needs more memory ? This is ridiculous! 40% free memory (RAM Memory) and 1.5Gb FreeMemory (Virtual Memory/Swap), and Shard Freezes...

This is lazy!

Well.. I am still waiting another tips or other way to debug this problem... Or then, how tunning memory utilization about .net....

Waiting...

Best wishes

 

[cward]

I guess your shard is fucked then if you don't want to fix it. I'm not sure if this is a bug, Maybe Microsoft intended for it to only use 60% of RAM. You may be waiting a long time.

 

[seph]

I guess your shard is fucked then if you don't want to fix it. I'm not sure if this is a bug, Maybe Microsoft intended for it to only use 60% of RAM. You may be waiting a long time.

This is very very bad!

I need more memory then... this problem is "unsolved"...

This is the problem using NO OpenSource Software (and Microsoft plataform).

OK.
Thanks to all for tips.

Best wishes.

 

[cward]

Has nothing to do with it not being opensource. It has to do with you failing to meet the hardware requirements you need.

 

[seph]

Has nothing to do with it not being opensource. It has to do with you failing to meet the hardware requirements you need.

I dont need more hardware! I have 40% of RAM free and 1.5GB Virtual Memory Free!
You accept all conditions and limitations from software ?

I dont understand why .net dont use 1.5Gb Free Virtual Memory.

 

[cward]

This is a problem with .Net not with RunUO. If you want .Net opensource contact microsoft, I'm sure they have a reason why the only permit it to use 60% currently. If you want to fix your shard get more ram. You've been told how to fix it, if you don't want to fix it then its your problem.

I want to say all I see from this is you making some lame excuse to make RunUO opensource when the problem is .Net not RunUO. If you don't want to upgrade your RAM or clean up unused accounts then rot in the lag or build your own emulator. This problem is resolved, you just don't want to do what it requires to fix it. The problem that is not resolved is with the .Net framework. Go contact microsoft if you want that resolved.

 

[seph]

This is a problem with .Net not with RunUO. If you want .Net opensource contact microsoft, I'm sure they have a reason why the only permit it to use 60% currently. If you want to fix your shard get more ram. You've been told how to fix it, if you don't want to fix it then its your problem.

I want to say all I see from this is you making some lame excuse to make RunUO opensource when the problem is .Net not RunUO. If you don't want to upgrade your RAM or clean up unused accounts then rot in the lag or build your own emulator. This problem is resolved, you just don't want to do what it requires to fix it. The problem that is not resolved is with the .Net framework. Go contact microsoft if you want that resolved.

OK. thanks.
Best wishes...

 

[Ingvarr]

Tell me then cward why:

a) It freezes with 400 Mb used on our shard (1 GB total, so this is just 40% not 60%) And this is only if you count only physical memory. Btw its omnious to me why do you exclude virtual memory - .NET can use virtual memory.

b) Ive located the thread and procedure it freezes inside, and this is a core packet handling thread and core procedure (Server.NetState.OnSend). This is not .NET system thread and not garbage collection. And shard will exit this "frozen" state if Ill forcefully place this "stalled" thread into the "suspend" state. If there were the out-of-memory I wouldnt able to resume normal execution no matter what I would do.

If there was a bug with low memory for you, this does not mean that this is a same one. So please dont make an universal excuse for any freezing bugs is upgrading memory.

Also I dont rant about "making RunUO opensource" or whatever. I just hope this bug will be really fixed. Even if it does not happen on developers hardware/software setup.

 

[TheOutkastDev]

b) Ive located the thread and procedure it freezes inside, and this is a core packet handling thread and core procedure (Server.NetState.OnSend). This is not .NET system thread and not garbage collection. And shard will exit this "frozen" state if Ill forcefully place this "stalled" thread into the "suspend" state. If there were the out-of-memory I wouldnt able to resume normal execution no matter what I would do.

I seriously doubt its in the OnSend method. All OnSend does is get the next packet in the queue and begin the asynchronous send call. If it's locking up from the Asynchronous call, then your network card has some issues.

 

[Ingvarr]

Well this is the best I can get from call stack.

Remember, I have no source nor debug info for this code, so I can base only on assumptions about location made by debugger.

But lets not stray to "faulty hardware" again. If this would be faulty network card (and stall in the driver therefore) it would stall entire network communication, remember, network card and driver have no idea about RunUO threads. But this is not the case, it stalls only one thread, all other network communications work fine.

Lowest it can be below RunUO for such behavior - at WinSock level.
Also dont know if it helps, this "freeze" is not instant. For some time shard still keeps running for other people but with gradually increasing lag (server CPU is already at 100%), and after at least 10 minutes, it freezes completely.

Here is call stack for thread in question (it changes sometimes, but in majority of time it such)

System.Net.Sockets.OverlappedAsyncResult.OverlappedAsyncResult(socket, asyncstate, asynccallback)
System.Net.Sockets.Socket.BeginSend(buffer, offset, size, socketFlag, callback, state)
Server.Network.NetState.OnSend(asyncResult)
System.Net.Sockets.OverlappedAsyncResult.CompletionPortCallback(errorCode, numBytes, nativeOverlapped)

 

[seph]

Well this is the best I can get from call stack.

Remember, I have no source nor debug info for this code, so I can base only on assumptions about location made by debugger.

But lets not stray to "faulty hardware" again. If this would be faulty network card (and stall in the driver therefore) it would stall entire network communication, remember, network card and driver have no idea about RunUO threads. But this is not the case, it stalls only one thread, all other network communications work fine.

Lowest it can be below RunUO for such behavior - at WinSock level.
Also dont know if it helps, this "freeze" is not instant. For some time shard still keeps running for other people but with gradually increasing lag (server CPU is already at 100%), and after at least 10 minutes, it freezes completely.

Here is call stack for thread in question (it changes sometimes, but in majority of time it such)

System.Net.Sockets.OverlappedAsyncResult.OverlappedAsyncResult(socket, asyncstate, asynccallback)
System.Net.Sockets.Socket.BeginSend(buffer, offset, size, socketFlag, callback, state)
Server.Network.NetState.OnSend(asyncResult)
System.Net.Sockets.OverlappedAsyncResult.CompletionPortCallback(errorCode, numBytes, nativeOverlapped)

Well, this is a limitation of emulador... because, if dotnet limit to use 60% of RAM memory of the server, if shard grows up, more and more memory is required, and this "memory upgrades" never stops. Plesae... I think this problem can more attetion, for a workaround.
My Shard haves 800k items and 60k mobiles, and this is not a "large" shard yet! This freezes reduce the uptime of server (2 restarts by day, 12/12 hours) and causa a lot of problems with it. Then, i need your attention for a workaround, for sollution for it.

I dont have any mobiles or items to remove :-(
Then, the only way if ADD MORE MEMORY to server... because .net... but.. this is a limitation for ME, and I think, its a limitation for RunUO too... Never can be a large shard with RunUO without a "LARGE" amount of memory (RAM > 2Gb).

This is problem for me... and i think, for others shard admins in this community, then... :-(

Thanks..