You can split the data into logical sections with your own in-house method of storage, for example the serialization I'm using saves each object to an individual file with a unique name, I use reflection to detect serializable members in classes which overrides the need for writing Serialize/Deserialize methods. All-in-all, there is more I/O operations performed, but it's multi-threaded so at any given time it can be writing 100 objects to their files, it made it possible to write an auto-recovery system for corrupted data, so you never have to wipe all items on your shard again due to a faulty serialization mistake, or even mismatched versions. Tested with 1,000,000 items on standard RunUO serialization method, 14 seconds, my method shaved off about 5 or 6 seconds, so it's worth it in the long run.
What about read/load speed?
And to be honest I'm surprised by the save times, cause on
World: Loading...done (1045052 items, 10001 mobiles) (9.72 seconds)
To save it takes Core: Using standard save strategy
World: Saving...Closing Save Files...
Save done in 1.90 seconds.
World: Loading...done (2089991 items, 20001 mobiles) (18.36 seconds)
World: Saving...Closing Save Files...
Save done in 3.24 seconds.
Or I just misunderstood you. In any case, after doing these tests items.bin is 90mb to think that each item is in its own file.... To have a million(2 in my case) files in a folder. Surely that's insane... I mean even if you group them you are going to end up with unmanageable file structure.
I'm really curious how fast did it load though.
P.S How do you load them by the way, do you have a list containing all the file names? Or just grab randomly out of the folder?
