Background Saving (For those that dislike Worldsave pauses)

[Bmzx007]

Background Saving (For those that dislike Worldsave pauses)

Hi Again, heh.

A new release of the original Background Worldsave core modification.

Please note...THIS IS STILL EXPERIMENTAL AND IT IS A CORE MODIFICATION.

For those that missed the original post, This modification saves the world in the background so your players are not interupted at all while playing (Even with the deep copy, it still does not pause).

Now thats out of the way. This one works slightly differently than the old one after battling with it all day.

When the save rolls over, it uses the Hashtable/ArrayList CopyTo method, and makes a copy of the keys and values into separate single dimensional arrays. Then it merges the two new arrays backtogether into a new Hashtable/ArrayList..

* New Findings *
CopyTo() is not a deep copy, after finally finding a reliable source out of many sources that were incorrect. So please be aware this is still a shallow copy. A deep copy requires quite a bit more work to get going (Reflection, IClonable, or Serialization), and I wont be attempting this at least for another 5 weeks. (Or I might wait until the next RunUO).
* End Findings *

The RAM jumps about 30-40MB when a save occurs, then the GarbageCollect plummets it back down when it finishes.

However, as a side effect of the deep copy, the world now takes around 0.3 - 0.5 of a second more to save. But being a background save, you dont notice it.

This was tested on a World of 150,000 Mobiles, 28,000 Items, and 1 Client.

* Further Testing *
Further testing with 30 clients has shown the RAM usage increases by about 10MB normally and on a save. And the save time does not increase from the typical 2.3 - 3.0 save times with the above world.
* End Further Testing *

The Vendor Restock, and Item Decay has been moved outside of the thread, into when the save actually finishes, and before the GarbageCollect occurs.

To install, Replace your World.cs file with mine, compile, and run.

Also do not forget to update your Autosave.cs file in your scripts, so that when a save occurs, if it is already saving, it will not try to back the files up.

Or find and replace the code that is shown below.

public static void Save()
{
      // Modified Piece
      if ( AutoRestart.Restarting || World.Saving )
            return;
      // End Modified Piece

      try
      {
            Backup(); 
      }

      catch{}

      World.Save();
}

If you do not add the above code to your scripts, you are looking at your save and one of your backups being wiped. You have been warned.

 

[Ray]

Thanks for sharing, the basic save method published by runuo is very optimized, indeed, but a background save would be really fantastic.

But i've got some apprehensions that this mod would be thread save and redundant.
What am i talking about:
It starts new threads to save all data, but doesnt stop the game. Well thats the meaning of Background Save of course. But it allows players to change the data while saving. As long as the save time is fast enough to win the race between a saving and changing, it doesnt matter. But if not, your save could be corrupt.

The other point, the mechanism can even crash while saving. If one thread modifies a value... lets say, by deleting an item while saving, it would try to carry on saving this item. With a little luck, it saves the deleted one, otherwise, it raises a thread or nullreference exception.

The only way to do a background save with this method, is by creating a deep copy just for saving. A shallow one wouldnt help much. But a deep copy also means double RAM size.

 

[Bmzx007]

If you took a look at it, it clones all the data before it saves, thereby removing the chance of crashing while modifying.

The MS clone method, locks the array from modifications while it is being used. You CANNOT allow modification while it is being itterated through this is a .NET thing, I spent ages looking for solutions, and this is the best one to come.

Basicly this creates 4 threads and starts them at the same time, to keep the 4 arrays as close as possible (sync). Then it takes copies of each array (Which does take quite a bit of memory) and saves the copies. The copies are safe when they are taken, then released as soon as its done.

When everything is finished, it performs a garbage collect to cleanup any excess usage (intentional).

 

[Arahil]

your approach is basically good, but still contains some stumbling blocks, i think.
first of all, i would freeze the world while copying the collections holding the world data, to be completely sure there are no diffs through the multithreaded character of your worldsave.
second, and more important, there are some methods i recommend to do in the mainthread, like deleting (decaying) items or restocking the vendors. finding the items and mobiles that need special care can be done in the saving-threads, but they should just be passed to the main thread, for example via timer.delaycall.

how good you saving works on a live shard i cannot say - i just think that the huge amount of ram needed for that is something that will scare off many admins and/or devs (including me), and will slow down runuo because of the use of the virtual memory.

 

[tophyr]

I'm not sure if you're aware of this or not, but RunUO already has a threaded save built in. Find the enum SaveType - in World.cs, you can set the SaveType to SaveType.Threaded, and then in World.Save() it will create diskwriter threads and buffers. Very effective.

 

[Bmzx007]

Ok, perhaps some more stats first.

On a world with 150,000 items, and 28,000 mobiles, saves are around 2.0 seconds on the machine stated above. And the ram increase while saving is only about 10-20MB.

Freezing the world is something I wanted to get away from, that was the whole idea of this project. Before I had a single save thread, which did them one after another, but doing them in 4 threads, gets the best synchronisation. Characters are still able to move, but there is about a fraction of a second (just noticable) where SOME tasks will take that fraction longer to start (Opening a gump as one) as I said though, its barely noticeable.

As for decays, vendors, etc. None of that has been modified from the original save methods, and it still works as per normal. Check the code out, its all there.

Dont get me wrong, this was thoroughly tested and thought through when I did it. It wasnt just written, tested for 10 minutes, and thrown out. I've been coding and testing this for 3 weeks, and havent had any sync problems, any exceptions (after I fixed the issues I was having), and have even done some heavy stress-testing on the CPU to make sure it would still perform.

The only thing that hasnt done, is testing it with a decent amount of people.

 

[Bmzx007]

I'm not sure if you're aware of this or not, but RunUO already has a threaded save built in. Find the enum SaveType - in World.cs, you can set the SaveType to SaveType.Threaded, and then in World.Save() it will create diskwriter threads and buffers. Very effective.

Yes I am aware of this, BUT that Threaded save still stops the world. This code does not, again, that was the purpose of the project. Run a test shard, and try it out, you'll see what I mean.

This is no less different in performance or effectiveness than a normal save. Its about 0.1-0.2 of a second in difference.

Please guys, Download the code and either look and/or test it before responding, Alot of the questions so far seem to be uneducated guesses as to what It actually does.

 

[Ray]

If you took a look at it, it clones all the data before it saves, thereby removing the chance of crashing while modifying.

The MS clone method, locks the array from modifications while it is being used. You CANNOT allow modification while it is being itterated through this is a .NET thing, I spent ages looking for solutions, and this is the best one to come.

I did take a look. But the ArrayList.Clone() Method doesnt perform a deep copy which is needed. It just clones the ArrayList.
Take a look at MSDN

Return Value
A shallow copy of the ArrayList.
A shallow copy of a collection copies only the elements of the collection, whether they are reference types or value types, but it does not copy the objects that the references refer to. The references in the new collection point to the same objects that the references in the original collection point to.

short version: it just creates a new reference. The data will be shared.

It doesnt lock nor create a deep copy of the collection.
You can test it, add a million items to your world and start the save. The memory wouldnt go up as dramatically as it would with a deep copy.

 

[Bmzx007]

Ok, But then take this example.

When using a simple copy method such as...

Hashtable x = World.Mobiles;

When a save came around, when the table was modified, I ended up with an exception that the table was modified during itteration.

With the .Clone() method, I no longer get that exception. And this was tested for 3 days solid running (Usually it'd crash in an hour). I ran various different tests over it that modified the array, again no crash.

So the .Clone() method is perfectly safe, and that is why it is used, as opposed to anything else. And though you say it doesnt lock, when the save occurs trying to perform some operations, it takes a fraction of a second longer, for the first 0.2-0.3 second of the save, indicating its locked the part its trying to use.

I did say this was thoroughly tested, and that was one of the issues I ran into, and solved. Test it yourself if you need proof.

 

[Ray]

Ok, But then take this example.

When using a simple copy method such as...

Hashtable x = World.Mobiles;

When a save came around, when the table was modified, I ended up with an exception that the table was modified during itteration.

With the .Clone() method, I no longer get that exception. And this was tested for 3 days solid running (Usually it'd crash in an hour). I ran various different tests over it that modified the array, again no crash.

True.
The collection has been cloned. Nothing will change this clone, thus you'll get no modified-exception.
But in fact, the data can have changed. Since the collection is cloned, it wont be affected by this change. It's blinded out. Unfortunately, that does apply to the 'cloned' reference types (Item, Mobile, etc), cause it's a shallow copy. They still refer to the original memory/data.
Now this data can be changed without raising an exception. Its not thread save anymore, and the result can be random - either correct, obsolete, deleted or null, depending on changes and garbage collection state.

So the .Clone() method is perfectly safe, and that is why it is used, as opposed to anything else. And though you say it doesnt lock, when the save occurs trying to perform some operations, it takes a fraction of a second longer, for the first 0.2-0.3 second of the save, indicating its locked the part its trying to use.
I did say this was thoroughly tested, and that was one of the issues I ran into, and solved. Test it yourself if you need proof.

Clone is a safe method, yes, but the data attached isn't. It just copies the reference *pointer*, not the data attached to this reference.

 

[Bmzx007]

Dont get me wrong, If I sound like Im attacking you Im not, im just responding.

The way I see it with that information, the closer the data is when it is saved to when the save actually started, the better it is. (Less problems with Sync issues).

 

[Ray]

I don't feel offended. At some curious reason, most people read my posts as harsher than i meant them.
I'm just trying to help with an upcoming problem you'll encounter with this script.

i'll make a sample

ArrayList A
Containing 5 Reference Type objects. In memory management view it'll only stores a links to the data. Like a hyperlink in HTML.

looks like: 22 = |0|1|2|3|4|

where 22 is the position of your arraylist wherever the kernel allocated it to.
each element has it's place in memory elsewhere random. But we know where through this stored link.

at memory position 0, theres the Element 'Foo'
at memory position 1, theres the Element 'Bar'
etc...

if you do an assign ArrayList B = A, it'll mean that B now hat the value 22, thus refering to position 22. So B is just a synonym für A.

If you do a shallow copy, ArrayList C = A.Clone()
it'll mean that C isnt a synonym anymore, its a NEW Arraylist, in example placed at memory position 23, and every reference is copied also.

But C will still look like 23 = |0|1|2|3|4|
and so it still refers to the old position, a shallow copy.
The collection has been cloned, but none of the elements.
Thats faster, more memory effecient, but not the thing we want to do.
Each element inside the collection is still a synonym, referring the same element.

I really don't have a clue how to deep copy such an array, cause none of these types have got a copy constructor that would be needed for it.




On a second thought, deleting and adding would not cause much problems, cause Garbage-Collector versioning surpresses the deletion of a referenced object and adding is no subject by creating a shallow copy.

But interacting/changing could be very critical.
lets think on a player that takes an item from his chest just in the moment, the chest saves and increments through its items stored.
This is a 1 to 1000 chance at all modify requests. But i can't imaging other side effects, they can be very random, even with a good structured TryCatchFinally thingy - but this would be a must, in case that happens

 

[Bmzx007]

I actually did plenty of Item drops, pickups, chest dropping during my testing and didnt come up with anything once I used the clone method.

I'll keep trying to reproduce what you describe, and it was something I thought of, but I think it'd have to be a long shot to get it to occur.

And I didnt mean your posts were harsh, I just didnt want anyone to take my responses the wrong way.

 

[Serp]

This method will get have weird and unpredictable effects on a shard. Most of the time it'll work fine of course, but occationally it'll screw things up. But still, great try and you probably learned a lot!

 

[Bmzx007]

How will it cause unpredictable effects (Just curious). Out of the 3 weeks Ive been running it (Loading the saved data as well), it hasnt caused anything out of the ordinary?

 

[Serp]

How will it cause unpredictable effects (Just curious). Out of the 3 weeks Ive been running it (Loading the saved data as well), it hasnt caused anything out of the ordinary?

The world is still live, things happen to the mobiles and items. They change parent (are moved, or traded), items are added to containers and mobiles, properties are changed etc. I see two major problems:
1. what happens if it began to save container A, and while it's iterating through the items to save references to them, an item is added to that container? The collection will be modified and an exception is thrown.
2. what happens when events in the world change properties of 2 or more items and the changes are dependant on eachother? If I'm selling a bunch of armor to you for 5k gold and we accept the trade during the save, 5 of the armor pieces may have been saved (and will bounce back to my character on a revert) while the rest of the pieces will be saved on your character. There are lots of more examples of this.

 

[Bmzx007]

Working on a solution to solve the issue of shallow cloning, expect an update either tomorrow, or at least by the end of the week. I understand the issue now.

 

[Arahil]

first of all, i did read your code before i responded.
the reason for the amount of memory growing just 20 mb is exactly the shallo copy thing ray explained very well before.
to the side effects: what i meant in my post with passing item.delete to the mainthread was exactly that. imagine the following situation:
an item is lying around in the world, decay-timout already reached and just waits for the next worldsave to be deleted.
then a player comes around and picks up the item. there might be a check if the item is deleted etc and in that very moment, the player is still able to pick up the item. but, in the time between the checks and the very action of picking the item up, your worldsave-thread runs over this item, and since the data is still set to make the item decay, the worldsave-thread deletes the item.
then, the main-loop is again on the processor and tries to continue to make the player picking up the item. but now, the item is already deleted (or at least marked to be deleted) - and so there is definitely a bad side effect.

this is also the reason why the runuo devs desided to make runuo rather single-threaded (all the other threads an asynchronous calls are queued and processed by the main-thread), synchronising threads is a really difficult thing to do, especially if you modify an existing program...

also, just making a deep copy in another thread is not thread save - except you freeze the world for the few moments where the deep copy is made.


hope you got my point...

 

[Atomic]

Pretty good, even changing to a deep copy it should take a much smaller freeze time as what takes most of the time isn't copying the data, is writing it to the files

 

[Bmzx007]

Re-released, Please see first post in this topic for details.