Save core modification!!It's possible working on it..

[A_Li_N]

Originally Posted by Quantos

What I mean is, how would this newer method of saving deal with a delta save when over 2000 inactive accounts, all with characters and houses, and items, banks, etc are cleaned off of the shard.

Wouldn't the deleting of those things mark them as 'changed' and therefore be saved accordingly at the next save? Might take a bit longer, but the concept is still there (I believe)

[Alis]

Yeah exactly like what A_Li_N said is correct the account how matter much it is it will be checked as changed().The time will not extend on the save ..

Noobie

What youa talking about is correct but ; there is another solution as far as I know but it would be good to see you helping me on this.In other means what a thing done more than 2 people is better then 1 people.Contact me plz

[Alis]

ok then with some search and some asist i diceded to explain step by step what we are going to do.As far as you all know this project is a project open for all people on extracting and asisting the method module we told.

Now firstly we are loonking at a way to stop the freeze done on the world during the save condition.

In the core you must firstly find the World.cs in the c sharp script we are searching the world save class so lets open the script fitsly ;

/

***************************************************************************
 *                                 World.cs
 *                            -------------------
 *   begin                : May 1, 2002
 *   copyright            : (C) The RunUO Software Team
 *   email                : info@runuo.com
 *
 *   $Id: World.cs,v 1.4 2005/01/22 04:25:04 krrios Exp $
 *   $Author: krrios $
 *   $Date: 2005/01/22 04:25:04 $
 *
 *
 ***************************************************************************/

/***************************************************************************
 *
 *   This program is free software; you can redistribute it and/or modify
 *   it under the terms of the GNU General Public License as published by
 *   the Free Software Foundation; either version 2 of the License, or
 *   (at your option) any later version.
 *
 ***************************************************************************/

using System;
using System.IO;
using System.Collections;
using System.Reflection;
using System.Threading;
using Server;
using Server.Mobiles;
using Server.Accounting;
using Server.Network;
using Server.Guilds;

namespace Server
{
	public class World
	{
		public enum SaveOption
		{
			Normal,
			Threaded
		}

		public static SaveOption SaveType = SaveOption.Normal;

		private static Hashtable m_Mobiles;
		private static Hashtable m_Items;

		private static bool m_Loading;
		private static bool m_Loaded;
		
		private static ArrayList m_DeleteList;

		public static bool Saving{ get { return m_Saving; } }
		public static bool Loaded{ get { return m_Loaded; } }
		public static bool Loading{ get { return m_Loading; } }

		private static string mobIdxPath = Path.Combine( "Saves/Mobiles/", "Mobiles.idx" );
		private static string mobTdbPath = Path.Combine( "Saves/Mobiles/", "Mobiles.tdb" );
		private static string mobBinPath = Path.Combine( "Saves/Mobiles/", "Mobiles.bin" );

		private static string itemIdxPath = Path.Combine( "Saves/Items/", "Items.idx" );
		private static string itemTdbPath = Path.Combine( "Saves/Items/", "Items.tdb" );
		private static string itemBinPath = Path.Combine( "Saves/Items/", "Items.bin" );

		private static string regionIdxPath = Path.Combine( "Saves/Regions/", "Regions.idx" );
		private static string regionBinPath = Path.Combine( "Saves/Regions/", "Regions.bin" );

		private static string guildIdxPath = Path.Combine( "Saves/Guilds/", "Guilds.idx" );
		private static string guildBinPath = Path.Combine( "Saves/Guilds/", "Guilds.bin" );

		//static World()
		//{
		//	Load();
		//}

		public static Hashtable Mobiles
		{
			get
			{
				return m_Mobiles;
			}
		}

		public static Hashtable Items
		{
			get
			{
				return m_Items;
			}
		}

		public static bool OnDelete( object o )
		{
			if ( !m_Loading )
				return true;

			m_DeleteList.Add( o );

			return false;
		}

		public static void Broadcast( int hue, bool ascii, string text )
		{
			Packet p;

			if ( ascii )
				p = new AsciiMessage( Serial.MinusOne, -1, MessageType.Regular, hue, 3, "System", text );
			else
				p = new UnicodeMessage( Serial.MinusOne, -1, MessageType.Regular, hue, 3, "ENU", "System", text );

			ArrayList list = NetState.Instances;

			for ( int i = 0; i < list.Count; ++i )
			{
				if ( ((NetState)list[i]).Mobile != null )
					((NetState)list[i]).Send( p );
			}

			NetState.FlushAll();
		}

		public static void Broadcast( int hue, bool ascii, string format, params object[] args )
		{
			Broadcast( hue, ascii, String.Format( format, args ) );
		}

		private interface IEntityEntry
		
			Serial Serial{ get; }
			int TypeID{ get; }
			long Position{ get; }
			int Length{ get; }
			object Object{ get; }
		}

		private sealed class RegionEntry : IEntityEntry
		{
			private Region m_Region;
			private long m_Position;
			private int m_Length;

			public object Object
			{
				get
				{	
					return m_Region;
				}
			}

			public Serial Serial
			{
				get
				{
					return m_Region == null ? 0 : m_Region.UId;
				}
			}

			public int TypeID
			{
				get
				{
					return 0;
				}
			}

			public long Position
			{
				get
				{
					return m_Position;
				}
			}

			public int Length
			{
				get
				{
					return m_Length;
				}
			}

			public RegionEntry( Region r, long pos, int length )
			{
				m_Region = r;
				m_Position = pos;
				m_Length = length;
			}
		}

		private sealed class GuildEntry : IEntityEntry
		{
			private BaseGuild m_Guild;
			private long m_Position;
			private int m_Length;

			public object Object
			{
				get
				{	
					return m_Guild;
				}
			}

			public Serial Serial
			{
				get
				{
					return m_Guild == null ? 0 : m_Guild.Id;
				}
			}

			public int TypeID
			{
				get
				{
					return 0;
				}
			}

			public long Position
			{
				get
				{
					return m_Position;
				}
			}

			public int Length
			{
				get
				{
					return m_Length;
				}
			}

			public GuildEntry( BaseGuild g, long pos, int length )
			{
				m_Guild = g;
				m_Position = pos;
				m_Length = length;
			}
		}

		private sealed class ItemEntry : IEntityEntry
		{
			private Item m_Item;
			private int m_TypeID;
			private string m_TypeName;
			private long m_Position;
			private int m_Length;

			public object Object
			{
				get
				{	
					return m_Item;
				}
			}

			public Serial Serial
			{
				get
				{
					return m_Item == null ? Serial.MinusOne : m_Item.Serial;
				}
			}

			public int TypeID
			{
				get
				{
					return m_TypeID;
				}
			}

			public string TypeName
			{
				get
				{	
					return m_TypeName;
				}
			}

			public long Position
			{
				get
				{
					return m_Position;
				}
			}

			public int Length
			{
				get
				{
					return m_Length;
				}
			}

			public ItemEntry( Item item, int typeID, string typeName, long pos, int length )
			{
				m_Item = item;
				m_TypeID = typeID;
				m_TypeName = typeName;
				m_Position = pos;
				m_Length = length;
			}
		}

		private sealed class MobileEntry : IEntityEntry
		{
			private Mobile m_Mobile;
			private int m_TypeID;
			private string m_TypeName;
			private long m_Position;
			private int m_Length;

			public object Object
			{
				get
				{	
					return m_Mobile;
				}
			}

			public Serial Serial
			{
				get
				{
					return m_Mobile == null ? Serial.MinusOne : m_Mobile.Serial;
				}
			}

			public int TypeID
			{
				get
				{
					return m_TypeID;
				}
			}

			public string TypeName
			{
				get
				{
					return m_TypeName;
				}
			}

			public long Position
			{
				get
				{
					return m_Position;
				}
			}

			public int Length
			{
				get
				{
					return m_Length;
				}
			}

			public MobileEntry( Mobile mobile, int typeID, string typeName, long pos, int length )
			{
				m_Mobile = mobile;
				m_TypeID = typeID;
				m_TypeName = typeName;
				m_Position = pos;
				m_Length = length;
			}
		}

		private static string m_LoadingType;

		public static string LoadingType
		{
			get{ return m_LoadingType; }
		}

		public static void Load()
		{
			if ( m_Loaded )
				return;

			m_Loaded = true;
			m_LoadingType = null;

			Console.Write( "World: Loading..." );

			DateTime start = DateTime.Now;

			m_Loading = true;
			m_DeleteList = new ArrayList();

			int mobileCount = 0, itemCount = 0, guildCount = 0, regionCount = 0;

			object[] ctorArgs = new object[1];
			Type[] ctorTypes = new Type[1]{ typeof( Serial ) };

			ArrayList items = new ArrayList();
			ArrayList mobiles = new ArrayList();
			ArrayList guilds = new ArrayList();
			ArrayList regions = new ArrayList();

			if ( File.Exists( mobIdxPath ) && File.Exists( mobTdbPath ) )
			{
				using ( FileStream idx = new FileStream( mobIdxPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryReader idxReader = new BinaryReader( idx );

					using ( FileStream tdb = new FileStream( mobTdbPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
					{
						BinaryReader tdbReader = new BinaryReader( tdb );

						int count = tdbReader.ReadInt32();

						ArrayList types = new ArrayList( count );

						for ( int i = 0; i < count; ++i )
						{
							string typeName = tdbReader.ReadString();

							Type t = ScriptCompiler.FindTypeByFullName( typeName );

							if ( t == null )
							{
								Console.WriteLine( "failed" );
								Console.WriteLine( "Error: Type '{0}' was not found. Delete all of those types? (y/n)", typeName );

								if ( Console.ReadLine() == "y" )
								{
									types.Add( null );
									Console.Write( "World: Loading..." );
									continue;
								}

								Console.WriteLine( "Types will not be deleted. An exception will be thrown when you press return" );

								throw new Exception( String.Format( "Bad type '{0}'", typeName ) );
							}

							ConstructorInfo ctor = t.GetConstructor( ctorTypes );

							if ( ctor != null )
							{
								types.Add( new object[]{ ctor, null } );
							}
							else
							{
								throw new Exception( String.Format( "Type '{0}' does not have a serialization constructor", t ) );
							}
						}

						mobileCount = idxReader.ReadInt32();

						m_Mobiles = new Hashtable( mobileCount );

						for ( int i = 0; i < mobileCount; ++i )
						{
							int typeID = idxReader.ReadInt32();
							int serial = idxReader.ReadInt32();
							long pos = idxReader.ReadInt64();
							int length = idxReader.ReadInt32();

							object[] objs = (object[])types[typeID];

							if ( objs == null )
								continue;

							Mobile m = null;
							ConstructorInfo ctor = (ConstructorInfo)objs[0];
							string typeName = (string)objs[1];

							try
							{
								ctorArgs[0] = (Serial)serial;
								m = (Mobile)(ctor.Invoke( ctorArgs ));
							}
							catch
							{
							}

							if ( m != null )
							{
								mobiles.Add( new MobileEntry( m, typeID, typeName, pos, length ) );
								AddMobile( m );
							}
						}

						tdbReader.Close();
					}

					idxReader.Close();
				}
			}
			else
			{
				m_Mobiles = new Hashtable();
			}

			if ( File.Exists( itemIdxPath ) && File.Exists( itemTdbPath ) )
			{
				using ( FileStream idx = new FileStream( itemIdxPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryReader idxReader = new BinaryReader( idx );

					using ( FileStream tdb = new FileStream( itemTdbPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
					{
						BinaryReader tdbReader = new BinaryReader( tdb );

						int count = tdbReader.ReadInt32();

						ArrayList types = new ArrayList( count );

						for ( int i = 0; i < count; ++i )
						{
							string typeName = tdbReader.ReadString();

							Type t = ScriptCompiler.FindTypeByFullName( typeName );

							if ( t == null )
							{
								Console.WriteLine( "failed" );
								Console.WriteLine( "Error: Type '{0}' was not found. Delete all of those types? (y/n)", typeName );

								if ( Console.ReadLine() == "y" )
								{
									types.Add( null );
									Console.Write( "World: Loading..." );
									continue;
								}

								Console.WriteLine( "Types will not be deleted. An exception will be thrown when you press return" );

								throw new Exception( String.Format( "Bad type '{0}'", typeName ) );
							}

							ConstructorInfo ctor = t.GetConstructor( ctorTypes );

							if ( ctor != null )
							{
								types.Add( new object[]{ ctor, typeName } );
							}
							else
							{
								throw new Exception( String.Format( "Type '{0}' does not have a serialization constructor", t ) );
							}
						}

						itemCount = idxReader.ReadInt32();

						m_Items = new Hashtable( itemCount );

						for ( int i = 0; i < itemCount; ++i )
						{
							int typeID = idxReader.ReadInt32();
							int serial = idxReader.ReadInt32();
							long pos = idxReader.ReadInt64();
							int length = idxReader.ReadInt32();

							object[] objs = (object[])types[typeID];

							if ( objs == null )
								continue;

							Item item = null;
							ConstructorInfo ctor = (ConstructorInfo)objs[0];
							string typeName = (string)objs[1];

							try
							{
								ctorArgs[0] = (Serial)serial;
								item = (Item)(ctor.Invoke( ctorArgs ));
							}
							catch
							{
							}

							if ( item != null )
							{
								items.Add( new ItemEntry( item, typeID, typeName, pos, length ) );
								AddItem( item );
							}
						}o

						tdbReader.Close();
					}

					idxReader.Close();
				}
			}
			else
			{
				m_Items = new Hashtable();
			}

			if ( File.Exists( guildIdxPath ) )
			{
				using ( FileStream idx = new FileStream( guildIdxPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryReader idxReader = new BinaryReader( idx );

					guildCount = idxReader.ReadInt32();

					CreateGuildEventArgs createEventArgs = new CreateGuildEventArgs( -1 );
					for ( int i = 0; i < guildCount; ++i )
					{
						idxReader.ReadInt32();//no typeid for guilds
						int id = idxReader.ReadInt32();
						long pos = idxReader.ReadInt64();
						int length = idxReader.ReadInt32();

						createEventArgs.Id = id;
						BaseGuild guild = EventSink.InvokeCreateGuild( createEventArgs );//new Guild( id );
						if ( guild != null )
							guilds.Add( new GuildEntry( guild, pos, length ) );
					}

					idxReader.Close();
				}
			}

			if ( File.Exists( regionIdxPath ) )
			{
				using ( FileStream idx = new FileStream( regionIdxPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryReader idxReader = new BinaryReader( idx );

					regionCount = idxReader.ReadInt32();

					for ( int i = 0; i < regionCount; ++i )
					{
						int typeID = idxReader.ReadInt32();
						int serial = idxReader.ReadInt32();
						long pos = idxReader.ReadInt64();
						int length = idxReader.ReadInt32();

						Region r = Region.FindByUId( serial );

						if ( r != null )
						{
							regions.Add( new RegionEntry( r, pos, length ) );
							Region.AddRegion( r );
							regionCount++;
						}
					}

					idxReader.Close();
				}
			}

			bool failedMobiles = false, failedItems = false, failedGuilds = false, failedRegions = false;
			Type failedType = null;
			Serial failedSerial = Serial.Zero;
			Exception failed = null;
			int failedTypeID = 0;

			if ( File.Exists( mobBinPath ) )
			{
				using ( FileStream bin = new FileStream( mobBinPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryFileReader reader = new BinaryFileReader( new BinaryReader( bin ) );

					for ( int i = 0; i < mobiles.Count; ++i )
					{
						MobileEntry entry = (MobileEntry)mobiles[i];
						Mobile m = (Mobile)entry.Object;

						if ( m != null )
						{
							reader.Seek( entry.Position, SeekOrigin.Begin );

							try
							{
								m_LoadingType = entry.TypeName;
								m.Deserialize( reader );

								if ( reader.Position != (entry.Position + entry.Length) )
									throw new Exception( String.Format( "***** Bad serialize on {0} *****", m.GetType() ) );
							}
							catch ( Exception e )
							{
								mobiles.RemoveAt( i );

								failed = e;
								failedMobiles = true;
								failedType = m.GetType();
								failedTypeID = entry.TypeID;
								failedSerial = m.Serial;

								break;
							}
						}
					}

					reader.Close();
				}
			}

			if ( !failedMobiles && File.Exists( itemBinPath ) )
			{
				using ( FileStream bin = new FileStream( itemBinPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryFileReader reader = new BinaryFileReader( new BinaryReader( bin ) );

					for ( int i = 0; i < items.Count; ++i )
					{
						ItemEntry entry = (ItemEntry)items[i];
						Item item = (Item)entry.Object;

						if ( item != null )
						{
							reader.Seek( entry.Position, SeekOrigin.Begin );

							try
							{
								m_LoadingType = entry.TypeName;
								item.Deserialize( reader );

								if ( reader.Position != (entry.Position + entry.Length) )
									throw new Exception( String.Format( "***** Bad serialize on {0} *****", item.GetType() ) );
							}
							catch ( Exception e )
							{
								items.RemoveAt( i );

								failed = e;
								failedItems = true;
								failedType = item.GetType();
								failedTypeID = entry.TypeID;
								failedSerial = item.Serial;

								break;
							}
						}
					}

					reader.Close();
				}
			}

			m_LoadingType = null;

			if ( !failedMobiles && !failedItems && File.Exists( guildBinPath ) )
			{
				using ( FileStream bin = new FileStream( guildBinPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryFileReader reader = new BinaryFileReader( new BinaryReader( bin ) );

					for ( int i = 0; i < guilds.Count; ++i )
					{
						GuildEntry entry = (GuildEntry)guilds[i];
						BaseGuild g = (BaseGuild)entry.Object;

						if ( g != null )
						{
							reader.Seek( entry.Position, SeekOrigin.Begin );

							try
							{
								g.Deserialize( reader );

								if ( reader.Position != (entry.Position + entry.Length) )
									throw new Exception( String.Format( "***** Bad serialize on Guild {0} *****", g.Id ) );
							}
							catch ( Exception e )
							{
								guilds.RemoveAt( i );

								failed = e;
								failedGuilds = true;
								failedType = typeof( BaseGuild );
								failedTypeID = g.Id;
								failedSerial = g.Id;

								break;
							}
						}
					}

					reader.Close();
				}
			}

			if ( !failedMobiles && !failedItems && File.Exists( regionBinPath ) )
			{
				using ( FileStream bin = new FileStream( regionBinPath, FileMode.Open, FileAccess.Read, FileShare.Read ) )
				{
					BinaryFileReader reader = new BinaryFileReader( new BinaryReader( bin ) );

					for ( int i = 0; i <regions.Count; ++i )
					{
						RegionEntry entry = (RegionEntry)regions[i];
						Region r = (Region)entry.Object;

						if ( r != null )
						{
							reader.Seek( entry.Position, SeekOrigin.Begin );

							try
							{
								r.Deserialize( reader );

								if ( reader.Position != (entry.Position + entry.Length) )
									throw new Exception( String.Format( "***** Bad serialize on {0} *****", r.GetType() ) );
							}
							catch ( Exception e )
							{
								regions.RemoveAt( i );

								failed = e;
								failedRegions = true;
								failedType = r.GetType();
								failedTypeID = entry.TypeID;
								failedSerial = r.UId;

								break;
							}
						}
					}

					reader.Close();
				}
			}

			if ( failedItems || failedMobiles || failedGuilds || failedRegions )
			{
				Console.WriteLine( "An error was encountered while loading a saved object" );

				Console.WriteLine( " - Type: {0}", failedType );
				Console.WriteLine( " - Serial: {0}", failedSerial );

				Console.WriteLine( "Delete the object? (y/n)" );

				if ( Console.ReadLine() == "y" )
				{
					if ( failedType != typeof( BaseGuild ) && !failedType.IsSubclassOf( typeof( Region ) ) )
					{
						Console.WriteLine( "Delete all objects of that type? (y/n)" );

						if ( Console.ReadLine() == "y" )
						{
							if ( failedMobiles )
							{
								for ( int i = 0; i < mobiles.Count; )
								{
									if ( ((MobileEntry)mobiles[i]).TypeID == failedTypeID )
										mobiles.RemoveAt( i );
									else
										++i;
								}
							}
							else if ( failedItems )
							{
								for ( int i = 0; i < items.Count; )
								{
									if ( ((ItemEntry)items[i]).TypeID == failedTypeID )
										items.RemoveAt( i );
									else
										++i;
								}
							}
						}
					}

					SaveIndex( mobiles, mobIdxPath );
					SaveIndex( items, itemIdxPath );
					SaveIndex( guilds, guildIdxPath );
					SaveIndex( regions, regionIdxPath );
				}

				Console.WriteLine( "After pressing return an exception will be thrown and the server will terminate" );
				Console.ReadLine();

				throw new Exception( String.Format( "Load failed (items={0}, mobiles={1}, guilds={2}, regions={3}, type={4}, serial={5})", failedItems, failedMobiles, failedGuilds, failedRegions, failedType, failedSerial ), failed );
			}

			EventSink.InvokeWorldLoad();

			m_Loading = false;

			for ( int i = 0; i < m_DeleteList.Count; ++i )
			{
				object o = m_DeleteList[i];

				if ( o is Item )
					((Item)o).Delete();
				else if ( o is Mobile )
					((Mobile)o).Delete();
			}

			m_DeleteList.Clear();

			foreach ( Item item in m_Items.Values )
			{
				if ( item.Parent == null )
					item.UpdateTotals();

				item.ClearProperties();
			}

			ArrayList list = new ArrayList( m_Mobiles.Values );

			foreach ( Mobile m in list )
			{
				m.ForceRegionReEnter( true );
				m.UpdateTotals();

				m.ClearProperties();
			}

			Console.WriteLine( "done ({1} items, {2} mobiles) ({0:F1} seconds)", (DateTime.Now-start).TotalSeconds, m_Items.Count, m_Mobiles.Count );
		}

		public static void SaveIndex( ArrayList list, string path )
		{
			if ( !Directory.Exists( "Saves/Mobiles/" ) )
				Directory.CreateDirectory( "Saves/Mobiles/" );

			if ( !Directory.Exists( "Saves/Items/" ) )
				Directory.CreateDirectory( "Saves/Items/" );

			if ( !Directory.Exists( "Saves/Guilds/" ) )
				Directory.CreateDirectory( "Saves/Guilds/" );

			if ( !Directory.Exists( "Saves/Regions/" ) )
				Directory.CreateDirectory( "Saves/Regions/" );

			using ( FileStream idx = new FileStream( path, FileMode.Create, FileAccess.Write, FileShare.None ) )
			{
				BinaryWriter idxWriter = new BinaryWriter( idx );

				idxWriter.Write( list.Count );

				for ( int i = 0; i < list.Count; ++i )
				{
					IEntityEntry e = (IEntityEntry)list[i];

					idxWriter.Write( e.TypeID );
					idxWriter.Write( e.Serial );
					idxWriter.Write( e.Position );
					idxWriter.Write( e.Length );
				}
				
				idxWriter.Close();
			}
		}

		internal static int m_Saves;

		public static void Save()
		{
			++m_Saves;
			Save( true );
		}

		private static bool m_MultiProcessor = false;

		public static bool MultiProcessor{ get{ return m_MultiProcessor; } set{ m_MultiProcessor = value; } }

		public static void Save( bool message )
		{
			if ( m_Saving || AsyncWriter.ThreadCount > 0 ) 
				return;

			NetState.FlushAll();
			
			m_Saving = true;

			if ( message )
				Broadcast( 0x35, true, "The world is saving, please wait." );

			Console.Write( "World: Saving..." );

			DateTime startTime = DateTime.Now;

			if ( !Directory.Exists( "Saves/Mobiles/" ) )
				Directory.CreateDirectory( "Saves/Mobiles/" );
			if ( !Directory.Exists( "Saves/Items/" ) )
				Directory.CreateDirectory( "Saves/Items/" );
			if ( !Directory.Exists( "Saves/Guilds/" ) )
				Directory.CreateDirectory( "Saves/Guilds/" );
			if ( !Directory.Exists( "Saves/Regions/" ) )
				Directory.CreateDirectory( "Saves/Regions/" );

			if ( m_MultiProcessor )
			{
				Thread saveThread = new Thread( new ThreadStart( SaveItems ) );

				saveThread.Name = "Item Save Subset";
				saveThread.Start();

				SaveMobiles();
				SaveGuilds();
				SaveRegions();

				saveThread.Join();
			}
			else
			{
				SaveMobiles();
				SaveItems();
				SaveGuilds();
				SaveRegions();
			}

			//Accounts.Save();

			try
			{
				EventSink.InvokeWorldSave( new WorldSaveEventArgs( message ) );
			}
			catch ( Exception e )
			{
				throw new Exception( "World Save event threw an exception.  Save failed!", e );
			}

			//System.GC.Collect();

			DateTime endTime = DateTime.Now;
			Console.WriteLine( "done in {0:F1} seconds.", (endTime - startTime).TotalSeconds );

			if ( message )
				Broadcast( 0x35, true, "World save complete. The entire process took {0:F1} seconds.", (endTime - startTime).TotalSeconds );

			m_Saving = false;
		}

		private static void SaveMobiles()
		{
			ArrayList restock = new ArrayList();

			GenericWriter idx;
			GenericWriter tdb;
			GenericWriter bin;

			if ( SaveType == SaveOption.Normal )
			{
				idx = new BinaryFileWriter( mobIdxPath, false );
				tdb = new BinaryFileWriter( mobTdbPath, false );
				bin = new BinaryFileWriter( mobBinPath, true );
			} 
			else
			{
				idx = new AsyncWriter( mobIdxPath, false );
				tdb = new AsyncWriter( mobTdbPath, false );
				bin = new AsyncWriter( mobBinPath, true );
			}

			idx.Write( (int) m_Mobiles.Count );
			foreach ( Mobile m in m_Mobiles.Values )
			{
				Type t = m.GetType();

				long start = bin.Position;

				idx.Write( (int) m.m_TypeRef );
				idx.Write( (int) m.Serial );
				idx.Write( (long) start );

				m.Serialize( bin );

				idx.Write( (int) (bin.Position - start) );

				if ( m is IVendor )
				{
					if ( ((IVendor)m).LastRestock + ((IVendor)m).RestockDelay < DateTime.Now )
						restock.Add( m );
				}

				m.FreeCache();
			}

			tdb.Write( (int) m_MobileTypes.Count );
			for ( int i = 0; i < m_MobileTypes.Count; ++i )
				tdb.Write( ((Type)m_MobileTypes[i]).FullName );

			for (int i=0;i<restock.Count;i++)
			{
				IVendor vend = (IVendor)restock[i];
				vend.Restock();
				vend.LastRestock = DateTime.Now;
			}

			idx.Close();
			tdb.Close();
			bin.Close();
		}

		internal static ArrayList m_ItemTypes = new ArrayList();
		internal static ArrayList m_MobileTypes = new ArrayList();

		private static void SaveItems()
		{
			ArrayList decaying = new ArrayList();

			GenericWriter idx;
			GenericWriter tdb;
			GenericWriter bin;

			if ( SaveType == SaveOption.Normal )
			{
				idx = new BinaryFileWriter( itemIdxPath, false );
				tdb = new BinaryFileWriter( itemTdbPath, false );
				bin = new BinaryFileWriter( itemBinPath, true );
			} 
			else
			{
				idx = new AsyncWriter( itemIdxPath, false );
				tdb = new AsyncWriter( itemTdbPath, false );
				bin = new AsyncWriter( itemBinPath, true );
			}

			idx.Write( (int) m_Items.Count );
			foreach ( Item item in m_Items.Values )
			{
				if ( item.Decays && item.Parent == null && item.Map != Map.Internal && (item.LastMoved + item.DecayTime) <= DateTime.Now )
					decaying.Add( item );

				long start = bin.Position;

				idx.Write( (int) item.m_TypeRef );
				idx.Write( (int) item.Serial );
				idx.Write( (long) start );

				item.Serialize( bin );

				idx.Write( (int) (bin.Position - start) );

				item.FreeCache();
			}

			tdb.Write( (int) m_ItemTypes.Count );
			for ( int i = 0; i < m_ItemTypes.Count; ++i )
				tdb.Write( ((Type)m_ItemTypes[i]).FullName );

			idx.Close();
			tdb.Close();
			bin.Close();

			for ( int i = 0; i < decaying.Count; ++i )
			{
				Item item = (Item)decaying[i];

				if ( item.OnDecay() )
					item.Delete();
			}
		}

		private static void SaveGuilds()
		{
			GenericWriter idx;
			GenericWriter bin;

			if ( SaveType == SaveOption.Normal )
			{
				idx = new BinaryFileWriter( guildIdxPath, false );
				bin = new BinaryFileWriter( guildBinPath, true );
			} 
			else
			{
				idx = new AsyncWriter( guildIdxPath, false );
				bin = new AsyncWriter( guildBinPath, true );
			}

			idx.Write( (int) BaseGuild.List.Count );
			foreach ( BaseGuild guild in BaseGuild.List.Values )
			{
				long start = bin.Position;

				idx.Write( (int)0 );//guilds have no typeid
				idx.Write( (int)guild.Id );
				idx.Write( (long)start );

				guild.Serialize( bin );

				idx.Write( (int) (bin.Position - start) );
			}

			idx.Close();
			bin.Close();
		}

		private static void SaveRegions()
		{
			int count = 0;

			GenericWriter bin;

			if ( SaveType == SaveOption.Normal )
				bin = new BinaryFileWriter( regionBinPath, true );
			else
				bin = new AsyncWriter( regionBinPath, true );

			MemoryStream mem = new MemoryStream( 4 + (20*Region.Regions.Count) );
			BinaryWriter memIdx = new BinaryWriter( mem );

			memIdx.Write( (int)0 );

			for ( int i = 0; i < Region.Regions.Count; ++i )
			{
				Region region = (Region)Region.Regions[i];

				if ( region.Saves )
				{
					++count;
					long start = bin.Position;

					memIdx.Write( (int)0 );//typeid
					memIdx.Write( (int) region.UId );
					memIdx.Write( (long) start );

					region.Serialize( bin );

					memIdx.Write( (int) (bin.Position - start) );
				}
			}

			bin.Close();
			
			memIdx.Seek( 0, SeekOrigin.Begin );
			memIdx.Write( (int)count );

			if ( SaveType == SaveOption.Threaded )
			{
				AsyncWriter asyncIdx = new AsyncWriter( regionIdxPath, false );
				asyncIdx.MemStream = mem;
				asyncIdx.Close();
			}
			else
			{
				FileStream fs = new FileStream( regionIdxPath, FileMode.Create, FileAccess.Write, FileShare.None );
				mem.WriteTo( fs );
				fs.Close();
				mem.Close();
			}
			
			// mem is closed only in non threaded saves, as its reference is copied to asyncIdx.MemStream
			memIdx.Close();
		}

		public static IEntity FindEntity( Serial serial )
		{
			if ( serial.IsItem )
			{
				return (Item)m_Items[serial];
			}
			else if ( serial.IsMobile )
			{
				return (Mobile)m_Mobiles[serial];
			}
			else
			{
				return null;
			}
		}

		public static Mobile FindMobile( Serial serial )
		{
			return (Mobile)m_Mobiles[serial];
		}

		public static void AddMobile( Mobile m )
		{
			m_Mobiles[m.Serial] = m;
		}

		public static Item FindItem( Serial serial )
		{
			return (Item)m_Items[serial];
		}

		public static void AddItem( Item item )
		{
			m_Items[item.Serial] = item;
		}

		public static void RemoveMobile( Mobile m )
		{
			m_Mobiles.Remove( m.Serial );
		}

		public static void RemoveItem( Item item )
		{
			m_Items.Remove( item.Serial );
		}
	}
}

As the colum colored in blue you can see that is the save method of the current runuo save module.

You can see that the save is done in the same thread so in our suggestion we started a new thread with nes hashtable for the save creation.

This is one of the steps we started; the other method which people say it needs allot of modification which i dont beleif is still underconstruction..

The steps will be posted untill it will finnish in few day;s

[Zippy]

I don't see a way you can determine if an object's serialization data has changed without modifying every script that does serialization to call a "MarkObjectChanged();" function or whatever.

And the reason a save which does not "freeze"/stall the entire shard (the game thread) will NEVER work is simple:

1) Save Begins
2) My character is saved, along with it my entire backpack (including a stack of gold in my backpack with amount=1000 and a sword)
3) I sell the sword to some guy (who has not yet been saved) for 500gp.
4) The guy gets saved
5) The shard crashes
6) The shard restarts, loads
7) Where is the sword? Where is the 500gp?

You might be able to get around this specific instance of the problem, but the over all problem remains... parts of the world are out of sync with each other. The whole point of a world save is to recover from unexpected crashes but under a system like this, these crashes would render the save data unusable.

Believe me, I've thought about this a lot. The most effective way to speed up RunUO's current saves would be a "delta" save, but as I've said it would just cause too many problems for RunUO users.

[A_Li_N]

I'm not extremely versed in this matter, but wouldn't saving to memory be much faster than directly to HDD? Then dump the mem stream to the HDD in the background. This would still create a pause, but that pause would be much less. I know that this has been discussed before, but I just wanted to bring it up again.

[Alis]

Originally Posted by Zippy

I don't see a way you can determine if an object's serialization data has changed without modifying every script that does serialization to call a "MarkObjectChanged();" function or whatever.

And the reason a save which does not "freeze"/stall the entire shard (the game thread) will NEVER work is simple:

1) Save Begins
2) My character is saved, along with it my entire backpack (including a stack of gold in my backpack with amount=1000 and a sword)
3) I sell the sword to some guy (who has not yet been saved) for 500gp.
4) The guy gets saved
5) The shard crashes
6) The shard restarts, loads
7) Where is the sword? Where is the 500gp?

You might be able to get around this specific instance of the problem, but the over all problem remains... parts of the world are out of sync with each other. The whole point of a world save is to recover from unexpected crashes but under a system like this, these crashes would render the save data unusable.

Believe me, I've thought about this a lot. The most effective way to speed up RunUO's current saves would be a "delta" save, but as I've said it would just cause too many problems for RunUO users.

Well thanks zippy for your suggestion but i allready told that

Im aware there will be some inconsistent case will be occured for those objects be archived and unarchived, the snapshot of the system at that moment will not reflect the real case.When we restore system from full archive and increments we will not achive the recent state of object that will be archived early and changed again till incremental archive is completed.But according to me this will rarely happend because we are talking about system crash and restoring.Also this can be acceptible because the object early archived has very limited possibility of occur.Even it happens it will loose the very recent change.

If we consider the current archive ; this scenerio cost more dramatically since you are going to loose last 30 min change in crash case(that needs to be restored, current user has a chance to complain about lost modifications on their objects).


I except what you said but only the trade done between during the save which such only could be affected during a crash could be macimum 10 people suffering such a problem but this is acceptable because it only happens during a crash phase .Its a terminolgy of racing for same resource.i also said this bedore but it is acceptable.

So it is highyly exceptable.

The other save method has no problems for the runuo users .

What i mean both methods are highly recommended and usefull.

Both of them are stable and only (1.st) method has a probality of having such problem the other delta save method has no pronblem in consistent.Zippy i wished that you have done this way a long time ago which would really change allot of things better.

[noobie]

Zippy has a good point there

actually the first problem is easy to solve, but out of sync problem is nothing easy to handle


for delta save issue, if it would be possible to write a generic hash function for serialization info, the rest would be peace of cake
actually, it is possible (not specificly for serialization info but all props) but it wouldnt be an efficient hash function anyway

[mrkroket]

Originally Posted by Zippy

I don't see a way you can determine if an object's serialization data has changed without modifying every script that does serialization to call a "MarkObjectChanged();" function or whatever.

And the reason a save which does not "freeze"/stall the entire shard (the game thread) will NEVER work is simple:

1) Save Begins
2) My character is saved, along with it my entire backpack (including a stack of gold in my backpack with amount=1000 and a sword)
3) I sell the sword to some guy (who has not yet been saved) for 500gp.
4) The guy gets saved
5) The shard crashes
6) The shard restarts, loads
7) Where is the sword? Where is the 500gp?

You might be able to get around this specific instance of the problem, but the over all problem remains... parts of the world are out of sync with each other. The whole point of a world save is to recover from unexpected crashes but under a system like this, these crashes would render the save data unusable.

Believe me, I've thought about this a lot. The most effective way to speed up RunUO's current saves would be a "delta" save, but as I've said it would just cause too many problems for RunUO users.

I dont see a problem on save crashes. If the shard crashes meanwhile an unfrozen save is being done, just check if save is finished (we can have some flag on disk to determine a succesful save), otherwise is a failed save and you shouldn't use it.

The problem is that it can duplicate objects on valid saves.

1) A sword is saved
2) the sword is sold to an npc for 500 gps
3) the 500gps are saved
What happens? On a restart, we'll have both items.
or
1) Player A have 30K gold
2) Player A is saved
3) player A gives X gold to player B
4) Player B is saved
What happens? Money duped.

Delta saves can be a good approach. Maybe a simple CRC can do the hard job. Let's say we want to save an item:

-We need a new hidden var, a SavedCRC
-On serialize, the item produce an array of bytes that are all those Write().
-Before really saving it, we obtain a CRC value. This should be an unique number depending on the array to be saved.
-Compare new CRC with SavedCRC.
+If equals the item remains unchanged and we can do a delta save.
+If CRC!=SavedCRC then SavedCRC=CRC and save the item as normally

That idea doesn't really change scripts a lot (you don't need to change every de/serialize, just on Core the save manager and the writer & reader). We also must have a complete save every X delta saves, let's say 1 each 12 hours.

Others ideas that can speed up saving times:

1) Do not touch anything. Create a Virtual HD on memory. Save on this disk, and with another program keep polling if a save is completed. Once you have a save, the program should copy the savegame to the real hard disk. The program at start also must copy the last save from disk to virtual for a proper load.
2) A memory buffer enough to keep all the save state. Once on buffer, we can save to disk with another thread.
3) I dont really know how serialize works internally, but it should have a 20MB buffer to save. Once it's full, it should make a blockwrite to disk. its faster to save 20MB as a block rather than 1000 saves of 20KB.
4) Also do a compress to the data on buffer idea 3). On those 20MB of buffer, we'll have a lot of item with many similarities. That means a low entropy, therefore we can compress a lot. Here may happen that compress time+save to disk>raw save to disk, that should be tested.

All those ideas can also be used with delta saves, they work at a different level.

[Alis]

What your talking about is not a efficient technica way the delta saves which im talking about could also use the system that your talking about lkie using delta and using ram save ... which A_Li_N said before but thats not quite effiecnt

What zippy is telling about the not freeze save method is correct but its acceptable 99% its stable and working 1% percent of chance loosing some files.. but this is acceptable the chance of loosing files is way toooo low ..

Mrkrok

You still need good rams and good superconductoer hdd to that but in my method you dont need to waste money on them with 72krpm hdds it will save a shard big like UOhybird in probably maximum 3-4 seconds. ..

[brodock]

i think what you are trying to do is similar to how Transactios work with databases... i was studying some ways to backup a "living database" and they used a simple trick to do a save without stop working...

let me explain:

they start to write every single database to the files...
while the "saving process" is running, every command and modification are stored in a log file...

hummm i came with something that can improve your idea...
also it's already working and developed...
google for XPrevail

[noobie]

@mrkroket:

it is easy to handle duplicate items with a simple saved flag.
the real problem which can not be handled is this:
a)while some method is executed:
-line xx: give some money to player A from B.
-line xy: give an item to Player B from player A.

lets say after the line xx, both player A and B saved, then line xy is executed..

if you reload the world, you will get a corrupted save in a sense..

about CRC:
it is almost same thing to evaluate that CRC with writing it. they almost take same time.
actually i explained it in my previous reply, what you are saying is actually a kind of hash function for serialization info. (see my previous reply)

@brodockbr: using a db for save is slower than using I/O. on the other hand, using xprevail which is a database persistance layer means you wont use binary data. it could take an hour to save the whole world, trust me on this

So overall, i think the most efficient way to improve current serialization would be:
writing save to memory could be an option, or increasing the buffer size which is only 4KB right now (actually i recommended it in another thread about serialization)

for both these options, another customized buffer could be used. using 20mb simple array buffer might not be a good idea. so you could implement a new buffer which is chunks of array.

[mrkroket]

Originally Posted by noobie

@mrkroket:

it is easy to handle duplicate items with a simple saved flag.
the real problem which can not be handled is this:
a)while some method is executed:
-line xx: give some money to player A from B.
-line xy: give an item to Player B from player A.

lets say after the line xx, both player A and B saved, then line xy is executed..

if you reload the world, you will get a corrupted save in a sense..

about CRC:
it is almost same thing to evaluate that CRC with writing it. they almost take same time.
actually i explained it in my previous reply, what you are saying is actually a kind of hash function for serialization info. (see my previous reply)

I don't think it's easy to handle duplicates and collateral effects. Like money, you don't destroy/move the item just change the amount, so you must track any amount change. The same could happen on other things, you probably end up tracking all changes on all vars.
Yes, it isn't a CRC what i'm refering, it's a cryptographic hash, a digest with low collisions.

Could the digest be useful? The thing is a to test how much the disk I/O affects to a save. The best way is to use two server replicas, one on HD and one on Virtual Disk. Take times of both saves and compare. Probably 85% of the time is used on disk I/O

And the 4KB buffer i think it really affects. Writing nx4KB chunks as a 20MB block is faster than all those chunks one by one.

[Alis]

as i said this project is opened source i gave all the things that must be done to work it what i gave are true and works i would be happy to see others exchanging their info on improving stuff on the save method i sended ..

Anyone who cares have suggestions or ideas they could either post it here or reach me from all the connection portals which i have 24 hour online maybe 2 - 3 hours sleep..

[Alis]

Originally Posted by noobie

@mrkroket:

it is easy to handle duplicate items with a simple saved flag.
the real problem which can not be handled is this:
a)while some method is executed:
-line xx: give some money to player A from B.
-line xy: give an item to Player B from player A.

lets say after the line xx, both player A and B saved, then line xy is executed..

if you reload the world, you will get a corrupted save in a sense..

about CRC:
it is almost same thing to evaluate that CRC with writing it. they almost take same time.
actually i explained it in my previous reply, what you are saying is actually a kind of hash function for serialization info. (see my previous reply)

@brodockbr: using a db for save is slower than using I/O. on the other hand, using xprevail which is a database persistance layer means you wont use binary data. it could take an hour to save the whole world, trust me on this

So overall, i think the most efficient way to improve current serialization would be:
writing save to memory could be an option, or increasing the buffer size which is only 4KB right now (actually i recommended it in another thread about serialization)

for both these options, another customized buffer could be used. using 20mb simple array buffer might not be a good idea. so you could implement a new buffer which is chunks of array.

Well what you talking about a memory thing is not a acceptable thing.. which both the system could be done without doing major madification .. maybe 100xtimes i will say this is a project based on a theory of object theory .. by editing only the main frame of the mobiles and the objects would be needed..

you will see in the further days which i will post the main frames modified done on the project.

[Alis]

you dont need to create a hashatable as beofore i wrote this please read it again.

I asume archive is iterativly getting objects from universe container using an Iterator therefore whenever he get a reffrence of an object and serilize an output to archive file and will ask next object from the universe container.Therefore after obtaining object from universe container even it is changed will not be disturb our mission.Assume that we freeze the system and made archive then unfreeze the system again and let the object be changed it is going to be same scenerio with our case. Asume that the archive will be completed only for this object and start with next object.

Im aware there will be some inconsistent case will be occured for those objects be archived and unarchived, the snapshot of the system at that moment will not reflect the real case.When we restore system from full archive and increments we will not achive the recent state of object that will be archived early and changed again till incremental archive is completed.But according to me this will rarely happend because we are talking about system crash and restoring.Also this can be acceptible because the object early archived has very limited possibility of occur.Even it happens it will loose the very recent change.

If we consider the current archive ; this scenerio cost more dramatically since you are going to loose last 30 min change in crash case(that needs to be restored, current user has a chance to complain about lost modifications on their objects).

For this reason I dont agree to use a secondary hashtable or so.What I am telling the archive can be started in background with in another thread(having lower priority) can run similtanusly and freezing is not necceserry.There is nothing to change but removing freezing process and just start archive in scheduled interval (in other word's nothing is needed to be changed but the operation flow going to be modifed).

[Zippy]

The frequency of world corruption on a "non-freezing" save will be much, much higher than you think Alis, especially for larger worlds which is the whole point of this anyways, right? Who cares if the server freezes for .4 seconds, we're talking about the ones that take 30 seconds to save when frozen

These could take minutes to save with a non-freeze method, and any time you load a world saved with this method you're talking about wide spread corruption. (Not just for crashes, but under any and all circumstances.)

Transactions would work fine/great/etc for a RunUO save, but the end problem is the same... something like that requires a major rewrite of nearly every script.

RunUO is over 200,000 lines, and I highly doubt anyone wants to go through and change every script file for a new system. Not to mention the immeasurable number of custom scripts affected... every shard would be facing huge amounts of rewrite just to allow their world to save.