MegaSpawner is back in development.

[Morxeton]

3.25 years since the last version... I decided to bring this system back to life. I know, I know... I come and go like a cheap hooker around these parts, but I'm back, for a while at least.

At first I didn't know if I would come back or not, until I got some network code down that actually worked. I stress tested it to hell and back and the code is pretty solid, granted it's more than likely not the best way to achieve this. Why am I worried about network code you say?... because:

v4 will be rewritten from scratch and the operation behind it will be different. The heart of the spawning system will be run in a seperate application that communicates with the system on the RunUO server via TCP sockets. It utilizes serialization over sockets, encryption, and compression (you can always toggle encryption and compression on/off, encryption is more or less for if your MSS server is exposed to the internet). This allows you to run the MSS core on another machine (a machine on the same LAN as the RunUO server) and free up resources on RunUO.

You may be concerned about bandwidth consumption, but the data that will be sent back and forth is minimal as possible. For example, if no one is even logged on the shard, the system wouldn't even be communicating between the two other than a ping check every now and then. When a monster is killed, that data is sent to MSS for running the timers and keeping track of what to spawn next. When it's time to spawn, it will send the command back to RunUO. You can also run the MSS on the same machine as RunUO and connect through the loopback.

There will be no gumps, except for the connection config gump to uplink to the MSS system. There will also be a client app that communicates with the RunUO server via TCP and this app will be used to configure all of your spawners.

Yay? or go jump off a cliff?

[Johabius]

Sounds very intriguing. And really 3.25 years isn't that long. Look at the development of something like, um, Duke Nukem Forever. I look forward to seeing this new iteration of MegaSpawner.

[Kamron]

I like the idea, sounds great.

[Sythen]

I can't wait, this sounds like an innovative idea! lol.

*birdy comes whistling in your ear*

"remember the x,y,z spawning coordinate positioning system" lol...

Good luck with the project and hope to see it soon.

well not too soon! The longer it takes you to work on it, the better quality I 'think' it will be. So take your time and try not to get discouraged.

Well thats all she wrote. :P

[SiENcE]

Please make an interface to use it with runuo1/sunuo under linux mono too :-).

[TheRockstar2253]

Can't wait to see the new System! Wtg!

[Morxeton]

This new system is still in the early design stages. I've redesigned the entire network core even from what I had over a week ago.

I'm utilizing a two socket design where one socket is used for packet traffic (which I call CommSocket) and the other is used for serialized object traffic (DataSocket). The sender sends a packet over to the receiver via the CommSocket with length information of the incoming serialized object to be read from the DataSocket. The DataSocket is regulated by a command queue so that once an object starts to send over the stream on the sender's end, the sender's queue is blocked until the receiver sends a packet to the sender stating that the entire stream was received. This allows the handler on the receiver's side to determine that the object was 100% received and to go ahead and toss it into the deserializer then into the data handler for processing.

The CommSocket is not regulated and simply allows packets to get tossed back and forth at free will. Each packet contains two main parts and the third part is optional depending on the packet. The first 5 bytes of each packet are static. The first byte is the ID and the next 4 bytes are for the 32-bit integer representing the length of the third part (if that packet has a third part, otherwise the length would be 0). If the length is greater than 0, then it attempts to read all of the bytes (determined by the length) for the third part from the stream until reaching the next packet. This data is read into a buffer in which the packet handler determines how to process it based on the packet's type.

Originally the CommSocket was simply tossing strings back and forth with delimiters and in the event of a buffer overflow, it would take a crap. I'm finally starting to learn some network code that is worth something, granted I'm sure it could be done better...

I also figured out how to deserialize an object in a different assembly. .NET normally does not allow you to take an object from one assembly, serialize it, then deserialize it in a different assembly, even if both assembly's objects are completely identical right down to the properties, fields, and the namespace they are in. I added a deserialization binder that basically fools the deserializer into believing that an object from assembly A is really from assembly B. And it's dynamic... meaning that it works on any assembly, versus hard coding it to replace assembly A's name with assembly B's name when A is the sender and B is the receiver, and vice versa for when B is the sender and A is the receiver.

In case anyone is interested in this dynamic deserialization, here is a code snippet:

        private static PacketObject GetFrom_Stream(Stream stream)
        {
            BinaryFormatter binaryFormatter = new BinaryFormatter();
            binaryFormatter.AssemblyFormat = FormatterAssemblyStyle.Simple;
            binaryFormatter.Binder = new DynamicBinder();

            return (PacketObject)binaryFormatter.Deserialize(stream);
        }

        internal sealed class DynamicBinder : SerializationBinder
        {
            public override Type BindToType(string assemblyName, string typeName)
            {
                string currentAssemblyName = Assembly.GetExecutingAssembly().FullName.Split(',')[0];

                string[] typeNameArray = typeName.Split('[');

                StringBuilder sb = new StringBuilder();

                for (int i = 0; i < typeNameArray.Length; i++)
                {
                    string field = typeNameArray[i];

                    if (field.IndexOf("Version") > -1)
                    {
                        string[] fieldArray = field.Split(',');

                        StringBuilder replaceAssembly = new StringBuilder();

                        for (int j = 0; j < fieldArray.Length; j++)
                        {
                            if (j > 0)
                            {
                                replaceAssembly.Append(',');
                            }

                            if (j == 1 && !IsExcluded(fieldArray[1])) // Index of assembly name.
                            {
                                replaceAssembly.Append(" ");
                                replaceAssembly.Append(currentAssemblyName);
                            }
                            else
                            {
                                replaceAssembly.Append(fieldArray[j]);
                            }
                        }

                        field = replaceAssembly.ToString();
                    }

                    if (i > 0)
                    {
                        sb.Append('[');
                    }

                    sb.Append(field);
                }

                return Type.GetType(sb.ToString());
            }

            internal bool IsExcluded(string assembly)
            {
                for (int i = 0; i < Settings.ExcludedAssemblies.Length; i++)
                {
                    if (assembly.IndexOf(Settings.ExcludedAssemblies[i]) > -1)
                    {
                        return true;
                    }
                }

                return false;
            }
        }

ExcludedAssemblies is a string[] that only contains the assembly name "mscorlib" for the time being. I don't think I will need to include any others but I designed it this way to make it easier.

What the DynamicBinder is doing is simply taking the assembly name from the sender and replacing it with it's own assembly name, then returning the object type into the deserializer as it's processing the stream.

I could have simply designed it the following way, but then it wouldn't be as dynamic since it would require changes if the assembly name changed in either the sender or receiver (it's just one less thing to worry about):

        internal sealed class DynamicBinder : SerializationBinder
        {
            public override Type BindToType(string assemblyName, string typeName)
            {
                return Type.GetType(typeName.Replace("Scripts.CS", "MegaSpawner System"));
            }
        }

This is much simpler, but I wanted to make it as dynamic as possible. If I named the assembly "Scripts.CS" for the MegaSpawner System, then this trick wouldn't be necessary. But who would want to click on Scripts.CS.exe to launch their MSS server?

This code snippet could be useful if say, you serialize an object to a file, then load that file and deserialize it into a completely different application that contains the same object. You could also just create a DLL file that would be commonly used between the two apps, but I didn't want to create a Scripts.CS.dll to use with the MSS server, like that wouldn't be confusing or anything...

I hope this code could prove useful to someone who's interested in serialization...

Back to the progress of this system, I'm about 85-90% done with the core network code then I have to get started on the scripts for the RunUO side of the server, which includes creating the MegaSpawner item, handling for all of the updates and such to pass between the systems, and whatnot. There's still quite a lot to accomplish.

[cjpainter1]

, I would like to Jack this Thread to say Thanks to everyone for the scripts and the chance to play. Thanks....

[Morxeton]

Interesting you picked my thread to jack, heh... no big deal but you can create your own threads if you'd like. I want this one to stay on topic which is of the development of the next MegaSpawner System.

[Phoric]

Sweet, thats good news, this was my favorite custom spawner back then.

[arul]

The first byte is the ID and the next 4 bytes are for the 32-bit integer representing the length of the third part

Why Int32? Int16 should be enough. I might be an evil user that sends a packet with int.MaxValue length and *POOF* OutOfMemoryException.

[Morxeton]

Originally Posted by arul

Why Int32? Int16 should be enough. I might be an evil user that sends a packet with int.MaxValue length and *POOF* OutOfMemoryException.

The only people that would be connected to this system are staff members managing the spawning system.

No regular users could connect to the system without knowing a valid userid/password to log on with. The way authentication works right now is both sockets are required to be connected before authentication can take place. Once both are connected, the client sends the CryptoObject, which contains the cryptography information to enable encryption, over to the server. The server enables encryption and sends a packet to the client stating that encryption is enabled, proceed with authentication. The client then sends an encrypted AuthObject which contains the user credentials. Once validated, the connection has been deemed authenticated and then traffic can flow freely between the two.

Now, if a corrupt staff member were to mod the client's source code, which I will be releasing with the package, and attempt to send a bad packet over with an int.MaxValue length, the server would keep storing all of the information into a buffer until the length was reached. When the server sends a ping packet over and receives no response (because that packet was stored into the growing buffer instead of properly handled), it will disconnect the rogue client. Although they could try to eat up a bunch of memory until that disconnect occurs... that is a good point and I will look into putting in the appropriate safeguard to prevent that from happening.

Int16 wouldn't necessarily be large enough since the MaxValue goes to 32767, which would equal ~32.7KB. There might be data larger than that to be sent over as a packet, I haven't determined that yet.

Thanks again for bringing that flaw to my attention.

EDIT:

Once I determine what the maximum packet size may be, maybe I'll just create a max packet length constant variable and any packets received with a length greater than the max, it will simply disconnect the client with an invalid packet length exception... If the max size happens to be under 32767, then I'll switch to int16. It'll save 2 bytes of traffic per packet...

EDIT #2:

I may change the way authentication works by requiring a crypto key file to be generated from the MSS app and distributed to the staff members who will use the system so that they must authenticate with the key file. Right now the crypto key is randomly generated and sent. The receiver doesn't verify the key, it just simply uses the key it received to enable encryption.

[arul]

For performance reasons I'd consider a static packet buffer (ie. array, not collection). I wrote a netcode from scratch a few months ago where I have the limit set to 32767, and if there's larger data to be sent it's being encapsulated into special object which handles coalescing of individual data chunks.

Anyway, looks like solid and robust system, can't wait to see the sources .

[Morxeton]

Thanks! I'm still getting familiar with network code. This is my first attempt.

I created a custom object for handling each connection called a SocketObject. This object contains the sockets, buffers, and what not for the connection.

As a part of the connection establishing process, when the client connects to the server, it sends over two different packets, one for determining it's the CommSocket, and the other for determining it's the DataSocket for each socket that connects. There is also information sent over with these packets that contain the GUID for the connection. The GUID is a 128-bit random System.Guid. This GUID along with the SocketObject is stored into a Dictionary with the GUID as the key. This GUID tells the system that these connections are from the same client, rather than basing it on RemoteEndPoint, which would be useless if there were multiple users connecting in from behind a firewall for example.

The buffer size is set to 4096. The length that is sent in the packet header simply tells the system that there is X amount more data to be received from the stream for this packet. The system is buffering all of this into a MemoryStream which is easily dumped into a byte array using .ToArray(). When the length is reached, it takes the MemoryStream and dumps it into a byte array and sends it off to the packet handler. It assumes the next byte is the Id for the next packet, which it will be unless an authorized user of the system creates a rogue client. Before a packet is sent, it calls to a method called GetBytes() which simply takes the Id, Length, and Buffer and converts them into a byte array. The Length variable on the packet is automatically set when the Buffer variable is set.

Any way you look at it, I need to put a size restriction on the packet's buffer to prevent some idiot, whom actually has authentication to log into the system, from creating a rogue client and eating up all of the system memory and crashing the server.

I made the code as dynamic as possible where I have a bunch of basic core objects and methods etc. that run the heart of the networking engine and then on top of that are the customized stuff for handling the specifics. This makes it easier for me to make tweaks to the custom stuff and not worry about the core networking engine, because it's already laid out and doesn't need to change. This also allows me to run a seperate Controller (inherits BaseControl), which is used for running the specifics on top of the core, and let's say for some reason I wanted to host a chat server in the same app, I could easily code that and still utilize the core network engine with the way it is dynamically handled.

I'm still refining the whole underlying code as I go along... I would be interested in taking a peek at the network code you have written, if you don't mind The entire source of this system will be released with the system itself. If you are interested in looking at what I have now, I could send it to you. I could always use another set of eyes to discover things I've overlooked or not doing efficiently.

[arul]

More robust and scalable than I thought. I like it

And yes, I'm interested in looking at the code. I'd send the sources I've written if they weren't a part of commercial product :\

[arul]

oops. double post :[

[Morxeton]

Originally Posted by arul

I'd send the sources I've written if they weren't a part of commercial product :\

Ah, gotcha.

Originally Posted by arul

More robust and scalable than I thought. I like it

And yes, I'm interested in looking at the code.

Thanks Please send me a PM with your e-mail address and I'll shoot the source on over.

[Morxeton]

I've created a network engine library, dubbed "Variassa", that handles the entire network engine for the MSS application and for the MSS scripts on the RunUO server. This engine is robust and allows you to add different controllers to handle different connections.

For example, Variassa only uses one Listener, and when the connecting app connects to Variassa, it tells Variassa which controller it wishes to connect in to. Variassa looks up the controller name (controllers are tagged with a custom attribute) and verifies it exists, and if it does, it allows the connecting app to send over the credentials for authentication. Each controller Variassa hosts can have different credentials for authenticating. This allows me to create the new "MegaSpawner System" and have it host a controller in Variassa, then when I revamp my "Time System", it too will host a controller in Variassa. Only one port is listening and one networking engine is handling two different applications.

There are static packets that the system uses for the initial connection, then each controller gets the core packets assigned to it locally. Any custom packets you create in your app will be registered with that controller. Once the Listener creates the SocketObject and establishes the connection, it passes it off to the controller, and everything is handled inside that controller, authentication, all the packets, everything...

I will be releasing the source to Variassa in addition to the source of the MegaSpawner System app. Variassa is useful as a network engine even outside of RunUO. You can do what you want with it, just need to know how to integrate your app into it (docs will be included). This allows you to create client/server applications that utilize Variassa and you do not need to worry about the core socket handling at all.

I've been tweaking and optimizing Variassa. It is far from complete, but I have to give thanks to arul for his suggestions. He has been very helpful!

Development for MSS is still in progress. It has slowed down because I am concentrating mainly on Variassa, but it will proceed once Variassa is complete.