NetState crash - Too much data pending

[_Epila_]

My RunUO NetState is crashing, seems to be something related to a huge amount of data being sent to the client, but what is too much data? 100kb?

First of all, it generated some crash reports that points to different parts of the code each time it crashes
I've started the server with VS on debug mode and asked it to handle EVERY thrown exception, and I've found an interesting behavior

Shouldn't the exception be catched at "throw"? Ok, it's not the point...
-Is it safe to increase the pending cap?
-Since NetState uses Asynchronous methods, this cap should not be easy to reach, am I right?
-Is it safe to ignore this cap?

EDIT: It does not only disconnect the client, it does crash the server!

 

[Soteric]

Posting crashlog could be helpful

 

[_Epila_]

The crash log points to a different part of the code each time it crashes

Server Crash Report
===================
 
Operating System: Microsoft Windows NT 6.1.7600.0
.NET Framework: 4.0.30319.233
Time: 14/01/2014 22:27:20
Mobiles: 12140
Items: 156453
Clients:
- Count: 1
+ 127.0.0.1: (account = Feeh) (mobile = 0x1F 'Feeh')
 
Exception:
System.NullReferenceException: Referência de objeto não definida para uma instância de um objeto.
   em Server.Network.NetState.Send_Completion(Object sender, SocketAsyncEventArgs e) na D:\UO\Conquest T2A\Server\Network\NetState.cs:linha 772
   em System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
   em System.Net.Sockets.SocketAsyncEventArgs.ExecutionCallback(Object ignored)
   em System.Threading.ExecutionContext.runTryCode(Object userData)
   em System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   em System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   em System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   em System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   em System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
   em System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   em System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Server Crash Report
===================
 
Operating System: Microsoft Windows NT 6.1.7600.0
.NET Framework: 4.0.30319.233
Time: 14/01/2014 23:19:35
Mobiles: 12140
Items: 156451
Clients:
- Count: 0
 
Exception:
System.NullReferenceException: Referência de objeto não definida para uma instância de um objeto.
   em Server.Network.NetState.Receive_Process(SocketAsyncEventArgs e) na D:\UO\Conquest T2A\Server\Network\NetState.cs:linha 728
   em Server.Network.NetState.Receive_Completion(Object sender, SocketAsyncEventArgs e) na D:\UO\Conquest T2A\Server\Network\NetState.cs:linha 705
   em System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
   em System.Net.Sockets.SocketAsyncEventArgs.ExecutionCallback(Object ignored)
   em System.Threading.ExecutionContext.runTryCode(Object userData)
   em System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   em System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   em System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   em System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   em System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
   em System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   em System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

 

[Soteric]

What RunUO version do you use?

 

[_Epila_]

I am using 2.3 Revision 1080 with every Mark's (GitHub) update on supporting newer clients (packets + Boat movement) and Rebirth cliloc handling
The only changes I've made to NetState.cs was to support newer clients, nothing on the communications was changed on the core

 

[fwiffo]

maybe you should remove the capacity exceeded exceptions, as shown in your screenshots taken from visual studio, and log who and what data is sending, ignoring the data to send, just log it in another, new file, for you eventual consideration.

Knowing who is generating so much data is the right way to solve this problem, because to generate such amount, there is someone who is sending it through netstate

 

[_Epila_]

It is a good start fwiffo, I'l try that.
But NetState shouldn't crash the server, it should disconnect the client, am I right ?

 

[Soteric]

Post NetState.cs

 

[fwiffo]

even if the exception is generated through netstate, the subsequent section could cause a crash at the parent thread.

I don't know really well how multithreading is structured in this service, but for what I remember in reading the sources, when I tried my way for implementing new netcode, it's a single thread that passes all data to asynch sockets, so this is BEFORE data is sent (you are still building the packets here, no?), thus it's not in another thread, but still in the main thread.

 

[_Epila_]

The packets are already built, the exception seems to be thrown when the thread is filling the buffer, and seems it is filling faster than the socket can send to the client

I've tested without the limited capacity buffer, seems that the data got sent, but the client freezes, I don't know if the client is processing the data or there was an overflow

Here is my NetState.cs

 

[Soteric]

This file doesn't correspond to crash log lines. Line 728

            m_SendQueue = new SendQueue();

can't produce null reference exception. Please repost crashlog or attach correct NetState.cs

 

[_Epila_]

Sorry, I was trying to log what and when then problem occur, and VS formatted the file
line 772 -> 960
line 724 -> 905 (this one happened when RunUO was paused at VS and disconnected the client due inactivity, the exception was thrown inside the packet (en/de)cryption process)

 

[Soteric]

I think you have to update to the latest Mark version. As far as I see he has solved this issue.

 

[_Epila_]

Yeah, seems that it's solved, not it display on the console and disconnect when there is too much data pending...
Anyway, thanks for the attention Solteric and fwiffo