Save core modification!!It's possible working on it..

[Alis]

Firstly this my first post to the modification column on the board;

About a 2-4 days ago i was working on the save methods how could we change the present model of the save method i actually dont know what method does the "Runuo Team" use for this project.But i have some some questions and some suggestion and some request..

We know that not all the objects found on the world is movable some partial of the world changes and grows such as "items,acounts,mobiles,logs,etc..".What firstly as i said that i dont know the method of the save's done in the world.

My first question is that how the procedure of the save acomplished = what i mean is this ;

When a save is done it save all the statics again and saves the voletile(changed) "items,acounts,mobiles,logs,etc.." .If we are using this method then i have suggestion;

Why dont we use the save method that only save the voletile(changed) "items,acounts,mobiles,logs,etc..".By this uniqe system the save time will fall from 100 -> 1

This system is possible ;how ? like this =

Evrey object must have an attribute marking it is changed or not ; and every time whenever the object is subject to change like a move of a object in the world from a -> b ; it must be marked as changed and evrey new created object must be marked as changed whenever created.When ever archive time is achieved like as 30 min default save time ; does objects having the changed attribute must only be archived(saved) and does attribute will be altered to static(unchanged) status.

Therefore the amount of data needed to be archived(saved) tremendusly decreased ; and it will be a payback of fast archived time(save).Beside this one ; we have to review the archive mechanisim to allow us to write-back only given object in proper place.If we make a statisc between old and new verison of archive(save) we will find the diffrence of two archive(save) is less then the size of the both of the archives(saves).

This was for zippy he wrote a while ago that people are not writing here anything this made him sad so i got this idea and working on it..

[Phantom]

You lost me at about...

About a 2-4 days ago i was working on the save methods

[Victor]

Originally Posted by Alis

Firstly this my first post to the modification column on the board;

About a 2-4 days ago i was working on the save methods how could we change the present model of the save method i actually dont know what method does the "Runuo Team" use for this project.But i have some some questions and some suggestion and some request..

We know that not all the objects found on the world is movable some partial of the world changes and grows such as "items,acounts,mobiles,logs,etc..".What firstly as i said that i dont know the method of the save's done in the world.

My first question is that how the procedure of the save acomplished = what i mean is this ;

When a save is done it save all the statics again and saves the voletile(changed) "items,acounts,mobiles,logs,etc.." .If we are using this method then i have suggestion;

Why dont we use the save method that only save the voletile(changed) "items,acounts,mobiles,logs,etc..".By this uniqe system the save time will fall from 100 -> 1

This system is possible ;how ? like this =

Evrey object must have an attribute marking it is changed or not ; and every time whenever the object is subject to change like a move of a object in the world from a -> b ; it must be marked as changed and evrey new created object must be marked as changed whenever created.When ever archive time is achieved like as 30 min default save time ; does objects having the changed attribute must only be archived(saved) and does attribute will be altered to static(unchanged) status.

Therefore the amount of data needed to be archived(saved) tremendusly decreased ; and it will be a payback of fast archived time(save).Beside this one ; we have to review the archive mechanisim to allow us to write-back only given object in proper place.If we make a statisc between old and new verison of archive(save) we will find the diffrence of two archive(save) is less then the size of the both of the archives(saves).

This was for zippy he wrote a while ago that people are not writing here anything this made him sad so i got this idea and working on it..

You should make the modifications and see if it makes any improvments in save time.

[Ravatar]

The seralization methods are already highly optimized, but good luck with em =)

[PerfectWing]

That's a logical enough idea you've got. I too would like to see the results.

[Alari]

I know of other systems that save changed objects instead of doing a full world save (they call them 'delta' saves) but then you have to write the save system to load the full save and apply any partial saves to the full save to arrive at the 'recent' saved world.

The systems that use delta saves are intended to be run on old hardware (I mean OOOLD like a 486 or something) where hard drive space, memory, etc are at a premium. RunUO by it's very nature (as a .NET) application needs newer hardware.

I'm sure you could decrease the save time by doing partial saves but you have to balance that with what could go wrong. If one of the delta saves is corrupt, (due to disk scratch or whatever) then you have to roll back to your last full save and associated deltas (hopefully those will all be good)

As well, the systems that I know of that use delta saves don't have quite the item churn that RunUO does. The difference between the last full save and the delta 15 minutes later may be a whole TON of deleted objects in RunUO. (dead mobiles, decayed objects, etc) RunUO would have to keep a record of those to write to the delta save, using both memory and disk space to track things which are no longer even in the game world. (Even just the object serial number and a 'deleted' flag takes memory and disk space.)


Could you save time, disk, and memory with partial saves? I guess the answer would be a firm 'maybe' :> Sorry I can't be more helpful than that, your best bet would be to compare two different running system with RunUO, same specs and world, except one doing full saves every time and the other implementing partial saves in between.

[Ohms_Law]

a better tack to take then partial saves would probably be to do constant saves. add indexer's to every object that is serializable, and have the serialization routines run through each one slowly and save when other things are idle. the only problem with that, that I see, is not being sure what's been saved when. ie.: shard crashes and you restore a backup, how do you know what's been lost?

[noobie]

btw, i just looked at runuo core, it saves all the data iteratively.
but it has some issues..it is possible to make it more efficient..

[Quantos]

Remember that there is no support for custom/modified cores.

[Alis]

Ok now i got these result's my the old archive sys. used to backup about 12 sec. with a small thing i test'ed the new system it gave this result

data size : 1mb

Old archive sys. 12.3 seconds this is a small test file i created in the same platform
New archive sys. 2.2 seconds the same test file..

During this operation i didn't use runuo but the platform is the same so it should work as simillar.

Well quantos yeah i know that but =) can't do all the scripts .. adding each a flag to work the archive sys.

But it proved that it work's faster.

Guy's by this way we dont need any superconductor harddisks or high ram.. My fellow friend said why are you working on this if people want good effiecency then they would pay allot of money on superconductor hdd's.

Well my answer was simple not all the runuo users could afford this money i dreamed a thing that is possible and fast equalivent to to fast conductor disks.. well i think i opened the door of pandora's box..

Further day's i would post some more data and verify here so you can see the diffrence between

[Victor]

Originally Posted by Alis

Ok now i got these result's my the old archive sys. used to backup about 12 sec. with a small thing i test'ed the new system it gave this result

data size : 1mb

Old archive sys. 12.3 seconds this is a small test file i created in the same platform
New archive sys. 2.2 seconds the same test file..

During this operation i didn't use runuo but the platform is the same so it should work as simillar.

Well quantos yeah i know that but =) can't do all the scripts .. adding each a flag to work the archive sys.

But it proved that it work's faster.

Guy's by this way we dont need any superconductor harddisks or high ram.. My fellow friend said why are you working on this if people want good effiecency then they would pay allot of money on superconductor hdd's.

Well my answer was simple not all the runuo users could afford this money i dreamed a thing that is possible and fast equalivent to to fast conductor disks.. well i think i opened the door of pandora's box..

Further day's i would post some more data and verify here so you can see the diffrence between

If you modify the base of which all items inherit from you can get all the scripts to use what you need to get your save system to work, right?

[Nochte]

yes--you will shorten save time, but in turn you will lose power elsewhere. every time a script modifies an item, the item's changed flag will have to be updated. this increases the number of operations in normal use dramatically. if you've got the processor to heave the extra load, then it's a definite improvement...otherwise it's going to mean lag.

[Victor]

Originally Posted by Nochte

yes--you will shorten save time, but in turn you will lose power elsewhere. every time a script modifies an item, the item's changed flag will have to be updated. this increases the number of operations in normal use dramatically. if you've got the processor to heave the extra load, then it's a definite improvement...otherwise it's going to mean lag.

Why would it produce lag if you are already changing an item? it does not take much more to update a flag.

[Ohms_Law]

Originally Posted by Victor

Why would it produce lag if you are already changing an item? it does not take much more to update a flag.

but it takes more time than not updating a bool value, or any other value for that matter.

[swtrse]

Originally Posted by Ohms_Law

but it takes more time than not updating a bool value, or any other value for that matter.

Come on the 10-20 CPU Cycles that are needed, to do that extra work is not even worth of been called.

[Alis]

Still on this project ; working on the vacation..

I know of other systems that save changed objects instead of doing a full world save (they call them 'delta' saves) but then you have to write the save system to load the full save and apply any partial saves to the full save to arrive at the 'recent' saved world.

The systems that use delta saves are intended to be run on old hardware (I mean OOOLD like a 486 or something) where hard drive space, memory, etc are at a premium. RunUO by it's very nature (as a .NET) application needs newer hardware.

I'm sure you could decrease the save time by doing partial saves but you have to balance that with what could go wrong. If one of the delta saves is corrupt, (due to disk scratch or whatever) then you have to roll back to your last full save and associated deltas (hopefully those will all be good)

As well, the systems that I know of that use delta saves don't have quite the item churn that RunUO does. The difference between the last full save and the delta 15 minutes later may be a whole TON of deleted objects in RunUO. (dead mobiles, decayed objects, etc) RunUO would have to keep a record of those to write to the delta save, using both memory and disk space to track things which are no longer even in the game world. (Even just the object serial number and a 'deleted' flag takes memory and disk space.)


Could you save time, disk, and memory with partial saves? I guess the answer would be a firm 'maybe' :> Sorry I can't be more helpful than that, your best bet would be to compare two different running system with RunUO, same specs and world, except one doing full saves every time and the other implementing partial saves in between.

Well ; I didn't mean to make and incremental backup(they call them delta saves).

As; Ohms Law mentioned putting some index or using some diffrent method like as direct acces thru index's or some advance hashing algorithims maybe implemented; therefor i wrote archive mechanisim need to be reviewed.

If we review your concern like as that the lost update; in incremental backup will not cause a catostrophic case since the lost change will effect only one or two user therefor even such a case is happend.

This case is an acceptable risk .

The same risk (disk sracht or whatever) is also is valid for full backup;Therefor the method as you cald delta save's has no extra risk then full archive..Which is your concern..

We may further disscus the new archive mechanizim on a updated object's is going to be incremental backup or direct acces the world image; like as indexing or hashing to locate object position..My intentions is to create universe container image on a disk and making civilization thru hased index algorithim and serilize objects thru this mechanizim.This maybe objectionabal by someone else since the disk corruption will be influence stability of the system ;therefor what I am thinking the universe image will be furtherly backuped as well (but in slower priority).

Also ; we can consider an intermediate archive redirector that only holds the object refrence and disk location.When ever an object is changed we can acces this table and change object location on the archive and point new location to serilize the object.By this method we will not overwrite old object information but create a new serilization data.This may help us to rebuild universe again.(Also we can put a time stamp on serilization data).

---------------------------New disscusion same topic-------------------------------

For sympthom relief of current archive system I am also wondering why archive and living system is not run concurently.If concern is making identical copy of living system with archive is froze running system and make backup is not a good approach to me .. If we start to archive an object and this object is furtherly changed during backup will not harm our goal since this object will be archived with next cycle.Therfore we can archive universe without froze(with lower priority).This will be appear as unfrozed system(like osi).

[Alari]

Originally Posted by Alis

If we review your concern like as that the lost update; in incremental backup will not cause a catostrophic case since the lost change will effect only one or two user therefor even such a case is happend.

Well, I was mainly thinking that if, say, full saves were scheduled at once every 15 minutes on a shard not using the volatile save thing, and on a shard that IS using it, full saves happen once an hour and partial saves happen once every 15 minutes, then if a file gets corrupt on both of them the one has only lost 15 minutes while the other has lost up to an hour.

But I don't think I fully understand what you intend so that may not apply. :>

If you mean continuous saves, those have up sides and down sides. =) The main way to do continuous saves seems to be to use a database. However, I like RunUO in that it doesn't require the admin to know anything about databases or have a database installed. That, and I've heard some horrible things about DB corruption, especially in server/emu software that uses databases because backups of that database tend to be rare.

[Alis]

Well, I was mainly thinking that if, say, full saves were scheduled at once every 15 minutes on a shard not using the volatile save thing, and on a shard that IS using it, full saves happen once an hour and partial saves happen once every 15 minutes, then if a file gets corrupt on both of them the one has only lost 15 minutes while the other has lost up to an hour.

But I don't think I fully understand what you intend so that may not apply. :>

If you mean continuous saves, those have up sides and down sides. =) The main way to do continuous saves seems to be to use a database. However, I like RunUO in that it doesn't require the admin to know anything about databases or have a database installed. That, and I've heard some horrible things about DB corruption, especially in server/emu software that uses databases because backups of that database tend to be rare.

If the same scenario will be applied to standart backup which is taken every 30min; if both of the backup's were corrupted then we will have 1.30 hour previews state it will not change the risk that you concern.
I am also not sympatetic with incremental backup.I am mainly dreaming direct update the universe on the disk; and frequently make copy of this disk file for rescuing.

[Alis]

Well; the situation that i see the lack of people doing D&R on some things make's me sad; makes me think that nearly 90% of the users is only using this forum for copy paste..

[Alari]

Originally Posted by Alis

Well; the situation that i see the lack of people doing D&R on some things make's me sad; makes me think that nearly 90% of the users is only using this forum for copy paste..

Unfortunately that's probably the case. I'm sure most of the people who feel they have created something worth sharing have posted it. The rest? Either they feel what they've created isn't worth sharing, or they want to keep their custom scripts exclusive to their shard. (Or are scripting for someone running a shard who wants to keep the scripts exclusive.)

Personally I haven't had a need to modify the core, and if I had it would be something not really worth posting. About the only thing I can think of that might need to modify the core and would be worth posting is poisonous swamps (making swamp tiles poison the player if they walk over them) but I haven't even begun work on that. :>

[Alari]

Originally Posted by Greystar

that wouldnt even require a core mod anyway, but I've been toying here and their with this concept and think it can be done with playermobile mods.

I thought it might need a core mod, but wasn't sure. Basically I want even static swamp tiles to be poisonous, to save the trouble of having to make invisible poison traps or whatever all over the map. Like I said I haven't started tho. :>

[Alis]

Well it be good that what the high council think about what I am working on?

[Alis]

-Test Phase 2 -

Ok today we tested more optional's of the new sys. We were amazed but also shocked.. Firstly the new system work's better then the old archive method.. As at the first phase we told that the compile of the archive method was about 10times better then the old one ;

Today we relized that the new archive system need's alot of editing on all core scripts..Nearly 99% of the scripts must be added with flag's.. to aprove the stability of the total script.So this need's a group work and i hope that not all the user's are copy paster's and also some of them is actually doing something to bring thing's better.

[Victor]

Originally Posted by Alis

-Test Phase 2 -

Ok today we tested more optional's of the new sys. We were amazed but also shocked.. Firstly the new system work's better then the old archive method.. As at the first phase we told that the compile of the archive method was about 10times better then the old one ;

Today we relized that the new archive system need's alot of editing on all core scripts..Nearly 99% of the scripts must be added with flag's.. to aprove the stability of the total script.So this need's a group work and i hope that not all the user's are copy paster's and also some of them is actually doing something to bring thing's better.

I am willing to help out. What needs to be done. Can you post what you have changed in the core so I can test this stuff out?