World XML Export

[astrocreep2k]

World XML Export

Hello All,

This isn't really a request, but more of a question. I was bored one day and started a little side project, and am kind of wondering if anyone would have use for it once its finished. I do not currently own/have access to a live shard (other than the one I run locally for dev purposes), so I am kind of digging to see if there would be any interest in it. Plus I would like to test it on a real shard with lots of accounts, which I dont have.

Its a couple of scripts that will export accounts, character info, guild info, online stats etc in xml format. Its designed to create a very small footprint, and to slowly stream data out at a configurable rate. The shard I currently play on has 35,000 accounts, so I wanted to design it to be able to handle this kind of thing without affecting gameplay. Its eventual purpose is to allow a secondary process (possibly on a different server) to publish the shards info/stats to a web site somewhere. But I am sure it could be used for other things.

Anyhow, please let me know if you are interested, or if you can think of a feature you would like to see from it. I hope to have a beta version ready within a few days, of which I will post in the custom scripts section of the forum.

 

[Kamron]

Long story short... if you want stats and info, use MyRunUO (which uses ODBC/MySQL database I believe). It is not advisable to export the world as an XML since this would be very large, however interfacing the server with a website and database, in order to take requests (or simply having RunUO do the database/web work like MyRunUO does), would be a more practical and viable option.
XML is not made to be streamed directly in the way you are describing.

 

[astrocreep2k]

Long story short... if you want stats and info, use MyRunUO (which uses ODBC/MySQL database I believe). It is not advisable to export the world as an XML since this would be very large, however interfacing the server with a website and database, in order to take requests (or simply having RunUO do the database/web work like MyRunUO does), would be a more practical and viable option.
XML is not made to be streamed directly in the way you are describing.

I agree that a RDBMS works well, but I think XML has a couple of advantages.

First, I have read many posts by admins who run the server on a different server than the site itself. They are concerned with directly updating the DB, especially in cases when the two servers are on different physical nets. As an alternative to opening up access to external address, this could be used to upload data changes indirectly and then imported. I will admit that its not as efficient, but possibly more secure.

Second, some admins do not have the time or ability to setup and manage MYSQL. This solution will require nothing more that a folder full of XML. And likely an app that reads it and spits out static html. And on top of the simplicity, a static, pre-generated page is probably going to be more efficient than a dynamic page that generates stats on the fly. This is even more the case when it comes to rendering images (char images for example).

Lastly, xml can be streamed as I mentioned. I am not talking about one giant file. They would be split up in a manner that would allow individual elements to be updated incrementally. Yes, it could get large, but its all text and probably not that much larger than MYSQL's datafiles after you've added a few indexes.

 

[Seanchen.net]

I agree that a RDBMS works well, but I think XML has a couple of advantages.

First, I have read many posts by admins who run the server on a different server than the site itself. They are concerned with directly updating the DB, especially in cases when the two servers are on different physical nets. As an alternative to opening up access to external address, this could be used to upload data changes indirectly and then imported. I will admit that its not as efficient, but possibly more secure.

Second, some admins do not have the time or ability to setup and manage MYSQL. This solution will require nothing more that a folder full of XML. And likely an app that reads it and spits out static html. And on top of the simplicity, a static, pre-generated page is probably going to be more efficient than a dynamic page that generates stats on the fly. This is even more the case when it comes to rendering images (char images for example).

Lastly, xml can be streamed as I mentioned. I am not talking about one giant file. They would be split up in a manner that would allow individual elements to be updated incrementally. Yes, it could get large, but its all text and probably not that much larger than MYSQL's datafiles after you've added a few indexes.

The size would at least be twice as big, if not more, then a database.

 

[astrocreep2k]

The size would at least be twice as big, if not more, then a database.

I guess that's possible, I will not argue. How did you come to that conclusion? I am curious because I would like to see how large these files could become potentially. What type of database are you referring to?

 

[Kamron]

I think he is referring to MySQL. Which btw, is not stored in plaintext. I believe you are referring to its exportation. MySQL does get large, however in comparison to the worldfile you have, and to plain text with tagging, it would not be as nearly as large.

Also, figure a way to tag everything in the world AND making arbitrary stats which may never get used, seems to be ALOT of uneccessary work. I would rather see the docs in XML format.

 

[astrocreep2k]

I think he is referring to MySQL. Which btw, is not stored in plaintext. I believe you are referring to its exportation. MySQL does get large, however in comparison to the worldfile you have, and to plain text with tagging, it would not be as nearly as large.

Also, figure a way to tag everything in the world AND making arbitrary stats which may never get used, seems to be ALOT of uneccessary work. I would rather see the docs in XML format.

The entire world would not be exported. Only the things that are important for publishing. Right now I have it setup for guilds, accounts, chars, and online stats (chars online etc).

Not sure what you meant by "arbitrary stats which may never get used".

 

[Kamron]

arbitrary statistics.... If you figure that you are going to emulate MyRunUO, then there is no need to. You can install a database via IIS and SQLServer on any Windows XP Professional/Windows 2003 machine.

 

[astrocreep2k]

I guess I dont see it as an MyRunUo emulation. Its probably going to do different things, and the functionality that it does mirror will probably be implemented in a different way... a way engineered with my goals in mind. I am not familiar with MyRunUo, (and maybe I should be), so I can't say how close they will be. But I am pretty sure the end result will not be considered an emulation of MyRunUo.

I guess an example of this is file transfer methods... if you think about it we have alot of protocols that do the same thing, just in a different way and with different advantages. Ftp, Windows File Sharing, IM Transfers, Email Attachments. You wouldn't use FTP to send your co worker a word doc... when they sit right next to you. And you wouldnt use email to upload data to a server in an automated process (Please lie to me even if you do...).

... So, in my mind it makes sense to transfer UO data to remote web servers in XML format via a controlled process, rather than exposing a database to IP spoofing and port scanners, all in the name of direct data transfer. Please... at least implement a vpn tunnel (which your run of the mill admin is not going to do). Ill admit the fastest way to a destination is a straight line, buts its not always the correct way.

BTW, thanks for the responses. The reason I started this thread was to get a few more brains involved, and you've raised some good points. I've been known to be wrong on many occasion, so please continue to question what I am doing if you have a concern.

 

[Kamron]

So, in my mind it makes sense to transfer UO data to remote web servers in XML format via a controlled process, rather than exposing a database to IP spoofing and port scanners

Regardless of the process or transfer type, if it involves an IP and a port, its spoofable. I would assume though that a person would secure the transfer between a database and their server. Similarly, I would assume that there would be a simple handshake to verify connections. This applies to any application or process.

Therefore I still do not see your point.. as XML is more bloated, requires parsing to write, and parsing to read, while databases don't.

 

[astrocreep2k]

Regardless of the process or transfer type, if it involves an IP and a port, its spoofable. I would assume though that a person would secure the transfer between a database and their server. Similarly, I would assume that there would be a simple handshake to verify connections. This applies to any application or process.

Therefore I still do not see your point.. as XML is more bloated, requires parsing to write, and parsing to read, while databases don't.

Yes, anything with an IP can be spoofed, and thats my point. Lets look at this from a damage control point of view. I hack your transfer method and gain access. Would you rather I- A: Gain access to your MySQL database and have direct read/write access to your db.. or B: gain access to a folder designed to import data, that contains incremental updates at most. Its not the protocol that is an issue. Its the method of transfer.

Your point about a person securing the transfer is faulted in that you are assuming. Unless MyRunUo includes step by step instructions on how to secure every ODBC supported RDBMS, I doubt that the majority of its users are going to do so.

Lastly, I find it interesting that you do not see the value in XML as a method of transfer. Alot of very large companies are using this type of transfer. I personally have worked on several projects were I work that used this, and they work quite well. I also have several friends who have worked on projects like this, and these applications handle an extremly large amount of data transfer. Not to mention the fact that Web services are almost the very same thing. And I can assure you the big boys use them. And databases dont require parsing... but the require client libraries and all sorts of code to function. So whats the diff?

 

[Greystar]

If I understand what you want to do (something like a more advanced form of Report Generation, that RunUO already does, combined with a generic form of MyRunUO) I can see how usefull XML would be instead of a database. Since RunUO never reads anything from these XML files, we wouldnt need to worry about LoadLag, however when it creates the files what kind of lag would it incurr to a running active shard. I know that some large shards that have MyRunUO running and dont have the Process power/memory do get some lag when posting data to the database, would what you attempting have more or less impact on an active shard. What exactly would you want to display in this information. I for one wouldnt mind seeing some generic mobile stats posted like MyRunUO, maybe with Monster name and some simple stats like General Str/Int and hits and weather or not it casts spells... Which im guessing what I just typed could be done via an XML output then read into a webpage via some kind of scripting, but im not that familair with that aspect of XML or how to read it from the web.

 

[astrocreep2k]

If I understand what you want to do (something like a more advanced form of Report Generation, that RunUO already does, combined with a generic form of MyRunUO) I can see how usefull XML would be instead of a database. Since RunUO never reads anything from these XML files, we wouldnt need to worry about LoadLag, however when it creates the files what kind of lag would it incurr to a running active shard.

I am assuming you are speaking of the startup time of RunUo when you say LoadLag. I am kind of new to this app so forgive me if I am misunderstanding. I guess that I cannot promise that this would be reduced. Something in the process may still need to load something when the exporter is initialized. Worst case, it should be about the same.

I know that some large shards that have MyRunUO running and dont have the Process power/memory do get some lag when posting data to the database, would what you attempting have more or less impact on an active shard.

I am shooting for less. I looked at MyRunUO and it seems that they do something similar to what I was planning. Everything is throttled, so even if 100 people add an account at the same time, they wont all necessarily get added right away. I am making the scripts configurable, so that you can adjust how often things are exported and how many at a time. Someone who wants faster updates can do so, and someone with limited resources can configure a slower output. In addition to this, all elements are going to be broken out into separate scripts. So if someone has no need for guild data, they just don't include the guildexporter script. I’m hoping that this allows people to tweak the exporter to work well with there shard.

What exactly would you want to display in this information. I for one wouldnt mind seeing some generic mobile stats posted like MyRunUO, maybe with Monster name and some simple stats like General Str/Int and hits and weather or not it casts spells... Which im guessing what I just typed could be done via an XML output then read into a webpage via some kind of scripting, but im not that familair with that aspect of XML or how to read it from the web.

Right now its accounts,characters, and guilds. I like the idea of publishing misc mobile stats though. It would allow the shard to allow potential players to view how they have things configured. And the web page stuff works well with xml. It can be parsed via php,asp/.net,java, all sorts of stuff. I think I mentioned this already, but once this is working I want to create something that can generate html and images from the data.

 

[Greystar]

I am assuming you are speaking of the startup time of RunUo when you say LoadLag. I am kind of new to this app so forgive me if I am misunderstanding. I guess that I cannot promise that this would be reduced. Something in the process may still need to load something when the exporter is initialized. Worst case, it should be about the same.




I am shooting for less. I looked at MyRunUO and it seems that they do something similar to what I was planning. Everything is throttled, so even if 100 people add an account at the same time, they wont all necessarily get added right away. I am making the scripts configurable, so that you can adjust how often things are exported and how many at a time. Someone who wants faster updates can do so, and someone with limited resources can configure a slower output. In addition to this, all elements are going to be broken out into separate scripts. So if someone has no need for guild data, they just don't include the guildexporter script. I’m hoping that this allows people to tweak the exporter to work well with there shard.



Right now its accounts,characters, and guilds. I like the idea of publishingmisc mobile stats though. It would allow the shard to allow potential players to view how they have things configured. And the web page stuff works well with xml. It can be parsed via php,asp/.net,java, all sorts of stuff. I think I mentioned this already, but once this is working I want to create something that can generate html and images from the data.

I would be willing to look at it once it's somewhat working... I never liked having to run additional apps like MySQL and what not and if what you are saying works it would be perfect for Shard Admins like me who like to keep additional APPS off their computer... I've already installed Python and a couple of other things so i could mess around with scripting laguanges, and I can make and understand SQL and related Databases, but I dont like them "stealing" my memory and Processor cycles, but... well, we use what is available .

 

[astrocreep2k]

Does your shard currently have a web site? And if so, does the web site reside on the same sever as RunUO?

 

[Greystar]

Does your shard currently have a web site? And if so, does the web site reside on the same sever as RunUO?

it had a website (its in transition right now (the shard that is)) and yes it was on the same machine.

 

[astrocreep2k]

just released a beta, so check it out at: http://www.runuo.com/forums/showthread.php?t=65960