Building Scaleable Middleware for MMORPGs

CowboyRobot writes "ACM Queue has an article exploring the challenges of developing a reliable platform for an MMORPG, specifically looking at Wish by Mutable Realms. From the article: 'A common scalability problem for distributed multiplayer games relates to managing distributed sets of objects... A player may not be a member of more than one guild, or a guild may have at most one level-5 mage (magician). In computing terms, implementing such behavior boils down to performing membership tests on sets of distributed objects.'"

Scalability and joining guilds

Surely this is a classic example of the Manager pattern. You have a bunch of objects [Avatar] (all alike, at least programmatically :-) who want to perform operations on other objects. If the system has a [GuildManager] class, then access to this distributed network of avatars can be forced through the choke-point of 'can this avatar join this guild'.

The trade-off in terms of scalability is in frequency versus computation. If the operation is commonplace (such as moving around), then a distributed system has a problem. If the operation is not commonplace (such as joining a guild!) then it's painless to use the 'choke' of a manager class to resolve any issues.

Even in the commonplace situation, I would have thought it useful to use overseer-objects whose job it is to remove the extra (unnecessary) information from the problem before trying to solve it... There's no need to care about the avatar in sector (-1000,-1000) if we're currently in sector (0,0)...

It's a cliche, but the rule is 'divide and conquer'. Screaming and leaping is a satisfactory, but usually fatal approach to problem solving, unless you're Kzin.

Simon

Re:Scalability and joining guilds

Avatar doesn't mean any particular "agent of god" powers in this message. He's using Avatar as the generic word for "the persona that a player plays as".

Re:Scalability and joining guilds

Surely this is a classic example of the Manager pattern.

I dunno, I've seen groups where there are 5 managers and one peon. Managers seem to accumulate.

Looks like some people out there need to review what design patterns [google.ca] are.

from first result:

Patterns and Pattern Languages are ways to describe best practices, good designs, and capture experience in a way that it is possible for others to reuse this experience. The Hillside Group takes pleasure in sponsoring many different PLoP conferences that are provided for the betterment of the pattern community.

Re:Scalability and joining guilds

This problem is in no way limited to MMORPG, the problem of authenicating and managing objects across multiple servers/clients is central to all online games. As a hobby games developer with a pretty good understanding of this I suggest you read Policing Online Games and then compare the conceptual pitch to issues in digital cash and online money transfers etc.
These ideas also overlap with the much hated and draconian 'trusted computing' models.

Enforcing a set of rules across a network of untrusted hosts is a fascinating problem. For example Gnunet and Freenet forgo a centralised trust agent and allow trust to emerge from the interaction, and recorded past behaviour, of individual nodes.

Digital 'trust' is sure to remain a huge area of interest. However it will also continue to be an area dominated by soothsayers, witchdoctors and charlatans because it contains a numer of fundamental logical problems which are not solved in the traditional human way of appeal to authourity.

Re:Scalability and joining guilds

Indeed, digital 'trust' is something that cannot be easily handed to, let alone established. Not counting real life relationships, anybody who meets another person online immediately (should) distrust that person. In a sense, this is destroying the foundation of the internet (the sharing of information openly and freely) but it not only works, it is also necessary (sadly). Why? Simple.

When you play a game online, would you rather trust "HarryGoatDeezNtz" who has an absolutely offensive name but you've seenen play online and play well for over 3 months, "UnnamedNewbie(6)" who you've never seenen before and is asking how to play in the chat, or "KingofSpades" who is an absolute asshole with no skills but has been playing the game longer than anyone other than the developers?

The same thing is true with businesses trying to estabish themselves online.

Would you trust Microsoft's Windows which is virus/bug/hack/etc prone, Linux which would require you to hire a full time IT staff just to keep your servers and computers working, or Macs and have your staff constantly ask 'wheres the right mouse button?'

So the question remains? Do you trust the 'veteran' of the three? The 'pro' or take a chance and give the 'newbie' a try?

Re:Scalability and joining guilds

Even in the commonplace situation, I would have thought it useful to use overseer-objects whose job it is to remove the extra (unnecessary) information from the problem before trying to solve it... There's no need to care about the avatar in sector (-1000,-1000) if we're currently in sector (0,0)...

Been there, seen that. Alternate Reality [metalmouth...ctions.com] had a creature in place for doing just the thing you were describing above. Taken from the AR Faq.

The problem with this was that in the AR system each object was unique (except commodities like food, money, gems) and had a structure with attributes (like "spells" which were small intrepreted programs embedded in objects). Those data structures took up memory (16 bytes to about 64 bytes) and on an 8-bit 64K (or 48K) computer we had to limit the amount of items somehow. The way we came up with that was least artificial was to introduce a creature that would eat up objects when the object queue was getting full.

Re:Scalability and joining guilds

Even in the commonplace situation, I would have thought it useful to use overseer-objects

I know this is done to some degree in Everquest. There are NPCs in each zone that exist to augment existing zone-related, PC and NPC situations.

For example, in each zone in EQ, there's an invisible NPC called, "pain and suffering" which appears to inflict damage on a player in certain situations (falling or bleeding to death). I would imagine that similar objects exist to control the weather, which in many cases might signal the client to narrow a player's depth of view and receive less information on objects in the vicinity.

MMORPGs need better real-time characteristics

I've tried a few MMORPGs and have found them all to be lacking in the same key area: one's control over one's character is not real-time. This is a generic description of a problem which surfaces in many ways in MMORPGs, most notably in the combat system. I haven't found one yet that allows real-time combat; it's always "click on the guy you want to fight and press the 'attack' button", then sit back and watch. Typically can do things like cast a spell or use a buff or otherwise make strategic changes to the way that your character is fighting, but you can't aim, run around, swing at the monster, etc, as you can with first person games.

The game that comes closest to the combat system I would want is Jedi Academy, in which the multiplayer mode works just like the first-person real-time perspective of the single player game. You do have to aim, you do have to run around and avoid shots, you do have to swing your light saber yourself. I find this to be infinitely more enjoyable than the MUD-like "you hit the spider for 10 points, it hit you for 5 points" back-and-forth that is common on all of the MMORPGs that I have played.

One gets the feeling in playing these MMORPGs that your client view of the world only loosely approximates what is happening on the server. You can make your character run from here to there and find that other people are "sliding" by or popping in and out as you get only sporadic notification from the server of what's really happening. It all gives a very disconnected feel that I really find unappealing about MMORPGs.

There must be some kind of scaleability limitation though because Jedi Academy only supports about 30 players or so at a time in an area that is far smaller than a play area in an MMORPG. I think that if someone could design an MMORPG that played like an FPS, but had all of the depth and breadth of one of these not-so-real-time MMORPGs, it would be ideal.

As an aside, has anyone beta tested Worlds of Warcraft? It like an excellent execution of the MMORPG genre, but I have yet to read any comments from beta testers on whether or not the fighting is real-time or "faked" like other MMORPGs is ...

Re:MMORPGs need better real-time characteristics

This burns down to the scalability problems the article is mentioning. Real-time characteristics always mean more frequent transfer of potentially larger data packages, and the more frequent processing of those packages. While you may be able to run Jedi Academy with 30 players on a cable connection, the same is not necessarily true for 300 players.
There are ways to at least make bandwidth and processing requirements scale less than linearly with the numbers of players, but the actual problem persists. The more players, the more data. The more data, the more bandwidth requirements and the more latency. The more latency and bandwidth requirements, the more the realtime characteristics suffer. Needing halfways reliable security (read: hack protection) methods doesn't make it any easier.

It is not only the reason why MMOs aren't realtime like an FPS, but also why FPSs aren't MMP like MMOs ;)

Re:MMORPGs need better real-time characteristics

Exactly. A 20 person UT match requires a surprising amount of bandwidth alone to make it as "realtime" as it can get, and a lot of CPU usage. If they were to try and accomplish the same thing on a scale of 5,000 players per server cluster at any given time, it would require too much bandwidth and resources to be profitable.

I see this becoming better in the future as CPU power and bandwidth get more and more available, and the prices of these games get higher and higher.

UT Design Is Not MMOG

A 20 person UT match requires a surprising amount of bandwidth alone to make it as "realtime" as it can get

The large quantity of bandwidth exchanged in a UT (or similar peer-based FPS game) is an artifact of a design as single object view game with no distributed Server-side processing. Instead of waiting for bandwidth and CPU nirvana, there are smarter ways to maximize Server-side entity state updates [slashdot.org] while optimizing Server-Client bandwidth and delivering only environmentally-relevent data. Also, using multiple, distributed Servers enables you to multiplex Server-Client entity state updates using multiple pipes so you don't get a blocks or racing on a single message broker.

Re:MMORPGs need better real-time characteristics

World War 2 Online [wwiionline.com] has sort-of beat that. It's a MMOG FPS, in essence.

Basically, there is a 64 unit "visibility limit". You're only ever told about 64 units max (and sometimes, due to oddities, less) player entities around you, prioritized among several criteria (distance, threat, minimum friend/foe allocations), etc.

It works fairly well, and the structure of the game is such that you have dozens of 30-60 player battles going on at all times, and can move anywhere around the map as you choose, realtime, either by slogging it on foot, driving, flying, or steering your ship. You can also jump from place to place but leave your equipment behind.

Best estimates put the peak server load at about 3-4000 players, with 500-1000 during the low tide, but the game runs 24/7 on a single arena.

The developers aren't swimming in money, but they're in the black and have recently turned up the data update rates to make it more smooth, so there's evidently some room in the budget for bits.

Disclaimer: I'm a day one player, from June 6 2001 on, aka Krenn, of the 1/16 Panzerdivision "Windhund".

Re:MMORPGs need better real-time characteristics

The issue here is you are looking at MMORPGS. RPGs are traditionally turn based combat. I haven't seen many successful RPGs that required one to actually swing or fire their weapon manually.

There are two massively multiplayer games that feature realtime combat

* Planetside, which is an MMOFPS with RPG characteristics (levelling, improving your character by gaining extra implant slots and additional simultaneous skill sets)

* Neocron, which is an MMORPG/FPS (I may be wrong on this one, it was a while ago and I only played the offline trainer, which was supposed to simulate online play)

One of the biggest issues is lag; to reduce lag, which would get horrendous when there are many people in close promixity doing things, the client-side visual representation and simulation, and the server side simulation are never in sync with each other. The server is the final arbiter, but the client tries to the best of it's ability and available information to provide a visual representation of what is going on.

Planetside (and I assume Neocron) solves the lag issue by moving combat resolution for attacks to the attacker's client, and trusting the client's integrity. As a result, you can easily die 3-4 seconds after running behind cover; likewise you can run through an intersection, be in the clear on your end, yet be shot and killed 5 seconds later as someone sees you go through the gap 3 seconds earlier and shoots you.

- MaineCoon

Its not fake, its a dice game

The RP in MMPORG means Role Playing ala Dungeons and Dragons. Thus, the focus is on character development, not first person shooter style twitch. RPGs are based on a dice game, and is really about mathematics. The people with powerful characters are the ones who can do math, not the once with a cable modem in the same town as the server.

Re:MMORPGs need better real-time characteristics

There must be some kind of scaleability limitation though because Jedi Academy only supports about 30 players or so at a time in an area that is far smaller than a play area in an MMORPG.

You hit the nail on the head with the scalability issue. Unless you're playing a game like Planetside where there's no significant penalty for dying, (other than just having to respawn and grab more gear) you're going to have a lot of unhappy players who get 0wned by the LPB twitch freaks.

I think that if someone could design an MMORPG that played like an FPS, but had all of the depth and breadth of one of these not-so-real-time MMORPGs, it would be ideal.

I agree, it would be spectacular. But as it is, there're a ton of people playing SWG who'll just spam damage on players as they load into new zones. Unless everyone in the world is on the same footing connection-wise and the ganeworlds are seamless; a real-time implementation of a combat system would only compound this kind of grief play.

Re:MMORPGs need better real-time characteristics

Dunno, but I've been playing this Eve Online [eve-online.com] stuff and it seems pretty realtime to me. And it supports 7500 people at the same time with no lag... And I must say it's pretty fun to play too, if you like space rpg's ;)

Re:MMORPGs need better real-time characteristics

I've played damn near every MMO out there. if one came out that made me aim and run around dodging, I wouldn't play it. Its not what I want from the genre. I've done twitch combat. I've done it for 20 years. It bores me. So you can hit the left arrow button faster than me. I don't give a damn. Currently MMOs test if you can use your skills and tactics to out think me. That is fun. And luckily, it seem the genre will stay like this for some time to come.

Re:MMORPGs need better real-time characteristics

Well you are lucky that there are DOZENS of MMORPGs that satisfy your gameplay requirements. I am just looking for ONE which satisfies mine.

For what it's worth, the MMORPGs that I have played are pretty weak in the strategy area anyway. Really there is no reason for a fight to last more than 1 second anyway. It might as well work like this: you click on the spider, the server pre-calculates how much damage it would do to you and you would do to it, and the server does the damage and it's done. There is no reason to have to sit and wait while your avatar hack smindlessly at the spider at a pre-determined rate and the spider does the same to your avatar. If there is no skill involved in the actual fight, then just skip it and go to the results!

Yes, it is true that you can cast spells and such, or switch to a different weapon normally. But I've found that it just leads to a formula which you use over and over again when fighting. You click on the spider, you say attack, when it hits you you heal, you watch the attack while you want for your mana to recharge so you can heal again, etc. You might as well just code all that up into a script that you run whenever you want to attack a spider.

I like real time fighting because it brings a fun arcade-y aspect to the game. It also makes it feel like you're more "in" the world and actually controlling your avatar, instead of just sitting back and watching what could have been a MUD anyway if it weren't for the 3d graphics.

Terazona

one's control over one's character is not real-time.

If you've attended GDC then you may have played ZonaBattle, a real-time mechanized battlecar demo game for the Terazona MMOG system. Disclaimer: I work for Shanda Zona [zona.net], the developer of this MMOG architecture, and my views may not represent those of the company, etcetera.

The purpose of the ZonaBattle technology demo is to illustrate that MMOGs do not have rely on sluggish, pseudo-turn-based gameplay. Using the right architecture produces excellent results.

ZonaBattle is not as fluid as some FPS games, but it is peppy and, unlike peer-based FPS games with~64-138 players, Terazona's client-server design enables you to scale the playfield to several tens of thousands of players and those players will experience no increased lag or message bottleneck.

Of course, you can also use Terazona to build "classic" seamless MMOGs. Terazona games do not have to have zones or "shards" and feature a heuristic, autoconfiguring grid system for game servers with dynamic region ownership, environmental simulation, and load balancing. You want more performance to support more players or more complex environment? Just slap in a few more commodity servers, or racks. The game will integrate them automatically and immediately begin dispersing Players and Entities among them.

Players can also exchange state with other local or non-local Entities using various bandwidth- and set-based configurable channels. This is not as easy as it might first appear.

Finally, the entire Server-side system is Java-based, for maximum flexibility and cross-platform support.

Re:Terazona

how does the game world map get distributed amongst the grid servers

Dynamic ownership, distributed object-view model. Very similar to the system described in Queue. You would never maintain a complete unitary in-memory representation of a world - that sucks up too server juice.

I come from a CORBA background as well - what you see with all this kind of MMOG Middleware (Butterfly, Quazal, There.com, BigWorld) is a classic example of evolutionary convergent adaptation.

I forgot to add a standard /. angle - we support MySQL for persistence, among others. In fact, any RDBMS with JDBC should work. We also support Entity creation, modification, bandwidth optimization using an XML-based schema editor (written in Java, of course). So game programmers don't have to fiddle forever making lots of structs and trying to optimize dirty bitmasks for message delta optimization. This lets you get past the tedious stuff quickly and get to the game logic.

Region Ownership

I can't give you specifics because you have not signed an NDA. I can talk in generalities, but I can tell you that TZ's load balancing is not random. Much of the information about how to load-balance such systems can be obtained from reading "Distributing Object State" which ran in GamaSutra a while back [gamasutra.com]. The key is dynamic ownership, not static ownership. As you rightly fear, static ownership leads to slowdown and player hangs.

The best way to maintain ownership would be dynamically, using some sticky heuristics to predictively anticipate where a player will "be" following a move, and alert Servers within some defined "neighborhood" or "ZOC" to update their state. This is non-trivial, because you may be dealing with non-Euclidean geometries, distance metrics, or set/guild membership. Therefore, each distributed Server can update its affected Clients on-demand, without those annoying lags you get with some systems when you can "feel" the Client loading the data from a new Server.

Alerting Servers that currently "own" those possible Regions to prepare to update relevent Entities with info is also required. If no Server owns that Region, then you should have a whole other set of heuristics to determine which game server should own that Region. It may, or it may not, be the Server that "owns" the Entity that is moving into that Region. You probably need to do cost-benefit calculations for assigning/re-assigning Region ownership. You can run Monte Carlo simulations to see how best to describe possible Entity "walks" within the topology.

Similarly, because of the expense of instantiation, you need some pretty tricky finagling to figure out when to relinquish ownership and purge any "ghost" copies of the Entity State that have been following the main Entity "around" within the topology. Of course, the nice thing is that Server-Server entity state exchanges will take place along a fat pipe backbone.

Interestingly, such systems end up looking a little like a Kohonen n-tier feedforward neural network.

Java Server-Side, Clients... whatever

You lost me at the "java-based" though

And oh yeah, only the Servers run Java only. The Client-side API is language-agnostic and platform-agnostic. So you can write Clients in C++ or Java and compile them to Win32, XBox, PS2, GameCube. The Servers don't care which Client belongs to which platform.

The analogy I like to use is NTSC. In the early days of TV without NTSC you had no guarantee that your GE TVs would be able to pick up Motorola format broadcasts. TVs competed within closed markets and featured lock-in. Creating a common broadcast standard enabled all TVs to pick up all broadcasts. TVs could compete on quality anf fucntionality, and broadcasters could compete using content. Using a platform-agnostic MMOG Middleware lets you enjoy economies of scale because your Servers communicate with all kinds of Clients. Client experiences vary, of course, according to display resolution and frame rate ability.

Re:MMORPGs need better real-time characteristics

I've tried a few MMORPGs and have found them all to be lacking in the same key area: one's control over one's character is not real-time....Typically can do things like cast a spell or use a buff or otherwise make strategic changes to the way that your character is fighting, but you can't aim, run around, swing at the monster, etc, as you can with first person games.

That's because of the RPG in MMORPG. In an FPS game, it is supposed to be a contest of your skill and reflexes vs. mine. In an RPG, on the other hand, if I'm a 20th level Fighter and you are a 10th level Fighter, I should be able to always beat you on physical skill. The only way you should be able to win is if I do something strategically wrong. Hence, the lack of detailed control over the physical aspects of combat.

Re:MMORPGs need better real-time characteristics

I haven't found one yet that allows real-time combat; it's always "click on the guy you want to fight and press the 'attack' button", then sit back and watch.

Yeah, that's always bothered me too. Why do you have to click to attack? Surely the computer can take care of the attacking all by itself [progressquest.com]. I'd rather just sit and watch, or maybe go grab a pizza and sit in front of the TV while my Half-Halfling Organist decapitates some Hair Elementals in the Killing Fields. Good times.

Article linked without all the crap

That page is annoying. For anyone on low-bandwidth or Lynx connections, here is the printer/human-friendly page [acmqueue.com].

Re:Article linked without all the crap

You can find a PDF version of the full article (exactly as printed in the magazine) here [zeroc.com].

Cheers,

Michi.

It's the Economy, Stipud

As my Econ professor used to say, "Economics is the study of scarcity." So they need a scalable economy engine, right? Compatibility of guilds is another matter.

"Authors thought of everything".

Lots and lots of modern cRPG, online or not, have way less features than games like Nethack. Just think what you can do about a towel - wrap it around your head, dip it in a fountain, wipe your face. Lots and lots of options. Incredible richness. Seems it can be done after all...

Are there any online rogue/nethack clones out there?

Who needs 10,000 people in a zone?

They say:

Wish is the first Ultra Massive Multiplayer Online Role Playing Game (UMMORPG(TM)). "Ultra" means that Wish supports more than 10,000 simultanous players in a single, seamless world, without any zones or "shards".

In EQ you can have an effect on other characters in your zone (say a hundred people) but you can talk with all the other people on your server (thousands, maybe tens of thousands of people). This is a limitation, but in practical terms it works OK. I don't actually need to interact with more than a few people at a time.

Scalability problem?

I don't see a scaling problem in the example.

Why not just keep a variable on each character associated with each guilds he's in. And then, each guild keeps a list of members.

Thus, a query on the data is always fast and local.

They should focus on a game that has shipped.

Everybody has their own MMO now, but unless they've actually shipped one they really probably don't understand the true challenges in building one.

Wish sacrifices

You make major one major, major sacrifice for so many simultaenously players in Wish. Movement is point and click. It feels like you're playing a strategy game rather than living in a real world. Those of you complaining that you can't joust and dodge in today's MMOs will hate the stilted movement mechanism of Wish, where the path you take is left to a pathing routine.

Lacking confidence here...

I don't really get a warm fuzzy feeling of confidence after seeing things like this:

"At ZeroC we used Java because some of our development staff had little prior C++ experience..."

"...however, a few of us had previously built a commercial object request broker..."

"...designing and implementing middleware is difficult and costly, even with many years of experience. If you are looking for middleware, chances are that you will be better off buying it than building it."

Frankly, I'd feel rather uncomfortable using ICE 1.0 as middleware for my new MMORPG. Yes they could succeed and do a nice job, but that rarely happens especially in the world of MMOs where nearly all games are released way too early in beta form.

Nothing To See - Move Along

Wake me up when you have something written by someone who actually has written and shipped a commercial MMORPG.

Do some of the work client-side...

Given their monstrous system requirements:
* P4 2.0GHz; P4 2.8GHz recommended (or Athlon equivalent).
* 512MB RAM; 1GB recommended.
* 64MB DX 9.0 Video Card (GeForce 3/4 Ti; ATI 8500+); 128 MB GeForce FX or Radeon 9600+ recommended.
* 16bit Sound Card; 24bit recommended.
* 8 GB free disk space; 7200+ RPM recommended.
* Connection to the Internet; 33 Kbps modem minimum; broadband recommended.

...maybe they should find a way to send datasets to the client machines and let them do their own manipulations.. Needing 8 Gig of disk on a 2+ GHz machine has to imply that the server doesn't handle all the real-time work... They are prime candidates for middleware that does some distributed computing and let all the customers' beast machines do the grunt work...

Truly a Dilbert moment.

I came across this quote:
At ZeroC we used Java because some of our development staff had little prior C++ experience.

and immediately though of that Dilbert strip (sorry, no link) mocking the "if all you have is a hammer, every problem looks like a nail" saying. That strip was particularly memmorable to me because the last panel featured a porcupine saying "we must stick them with quills! it's the only way!"

it's all about design & bandwidth

I really think computing power is less significant than the overall game development design when it comes to MMORPGs. After design, bandwidth becomes a factor, and only then is computing power a factor. The only exception I can think of would be requiring power for encryption/decryption.

The notion of parsing datasets for something like guild membership is really trivial. If you want to design a solid MMORPG, it's going to come down to how the world, objects and players are represented.

I continue to be in awe of the capabilities of games like Everquest and SWG. SOE has really created a very robust MMORPG technology -- it's hard for any other game developer to really say they have anything comparable when they can't demonstrate superior performance under the same conditions due to no other MMORPG having anywhere near the quantity of simultaneous players (as Everquest).

IMO, the client side of EQ is pretty straightforward. What makes the game special is the server side and how they manage to manipulate so many players and objects in real time. People complain that too many objects/players per "zone" can lag things down, and that is true, but I have yet to see a better implementation than Everquest. SWG has done away with the concept of "zones" to some degree, but basically, they seem to have implemented some client-side intelligence to indicate at which point additional graphics and information on objects in the distance should be loaded or reported. There are still "zones" in all these games. Some of them implement noticeable loading lags, and others don't.

My outside impression of the technical layout of Everquest is something like this... and I'd love anyone with more info/insight to correct me or elaborate further. I ASSume their system is made up of racks of servers, running Solaris I think. The have some low-level, propietary engine that manages the objects in the world, probably to a back-end database like Oracle. The reason for zones in EQ is that when you enter a new zone, you may actually be switching from one physical server to another. Not only do they have different servers for different shards/worlds, but different servers for different zones. When I see a system message such as, "North Karana, Velketors and Plane of Mischief going down for a brief update", I think that perhaps that's one server they're rebooting, which runs those particular zones. I suspect they stagger high-traffic zones with low-traffic zones on servers, and occasionally when the X number of zones managed by a single server have an unusually high amount of traffic/visitors, you get lag.

What's interesting about MMORPG game design is the balance between handling as much client-side as possible without creating security issues. If the server keeps track of players, NPCs and objects, it's much more difficult for someone to hack, or at least, logs are available to identify issues. The more client-side processing done, the more likely the game can be inappropriately manipulated.

When you take into account the amount of real-time data that goes back and forth, EQ (and SWG) are quite impressive. I don't think database/dataset issues are really the problem as being able to efficiently encapsulate, protect and send/receive the large amounts of data in the real-time world.

use a REAL RDBMS from the beginning

Whatever you do in this model, for christ's sake start with a real RDBMS. I know a couple people who work at an Austin-area MMORPG studio (not EA's Ultima-- that Austin office was shut down last week) and their developers have decided to go with flat files for storing all the data. This is just out of half-assed design specs from day one. Now, the game is several years released and it'd be too much work to re-write code to pull all their flat-file rewrites and have it do proper DB transfers.

Can you possibly imagine the race conditions that exist in this model? Scalability? Forget it.

Off the top of my head, I'd say go Oracle if it's a real commercial product. If not, PostgreSQL. mySQL isn't advanced enough yet to be trusted with storing a seventh-level mage's cloak of invisibility.

Has anyone used JavaSpaces for this?

JavaSpaces [sun.com] is more or less Sun's Jini take on distributed processing.

From a programmer point of view, you start up a "space", and then you can write objects in, take them out, and read them. And that's all. So there are a very few simple operations, and you structure your app around those.

Anyhow, it seems like a couple of JavaSpaces on a rack of servers might serve as a good way to distribute processing/notification/etc. Of course, you're limited to Java and to moving around Serializable objects....

Other types of middleware

I find their choice of RPC as the middleware layer surprising. I would imagine that a vast majority of events in a MMP game need to be passed on to a fairly large subset of players, for example, anytime someone moves, attacks, casts a spell, etc. the people visual or audio range of that player need to be informed. RPC is better suited to one-to-one interactions, not the one-to-many interactions we get in MMP games. It seems to me that a distributed publish/subscribe system like Elvin [dstc.edu.au], SIENA [colorado.edu] or even Mercury [cmu.edu] (a pub/sub system specifically designed for MMP games!), or a LINDA-style shared memory system would be much more appropriate.

This project is going to fail

This project is going to fail, or at least be severely delayed. Why? Well most of the article talks about how they rewrote COM/COMBRA/KOM (etc..), creating their own IDL, their own protocol etc.. However the article is not about rolling your own ORB, it's about designing a MMORPG middleware, which can have little to nothing to do with an object broker.

They even start of the article with some nice pie in the sky requirements. Like a truly scalable world capable of "tens of thousands" of people in the same world realtime load balancing, dynamic world installation. And a bunch of other stuff that probably sounded nice in some brainstorming meeting, though I suspect has little to do with any actual game design.

In my experience this usually happens when the people involved don't really understand the problem they are trying to solve, so they end up writing general purpose architecture to solve whatever problem they eventually do understand, typically dooming the project in the meantime.

-Jon

Easy

god, a real problem finally. I've been doing so much html and asp i miss something more substantial like this. i could solve this, i know it. The real issue here is data -- how do you access it, and how do you store it. The fact is that you could get an in memory object oriented database to keep track of it all. 4gb memory would be enough to support a ton of players. What you do is keep a "DNA strand" for each player with a bunch of binary toggles to control state. Forget "manager" and all that crap that overcomplicates the solution.

cool, a new career!

I do or have done distributed, persistent middleware with tons of threads, caching, dynamic persistence, multiple-databases, multiple-languages and so on middleware for enterprises or those who wish to sell to them many times in the last couple of decades. It didn't occur to me that most of the same talents are essential to games and various virtual spaces.

Thanks! :-)

We are missing the issues.. again

What you think the game developers are stupid?

Based upon the previous posts I would think that we are all mostly clueless with regard to what the technical problems that surround multi-player games really are.

But I wont let that stop me from ranting.

DOAC recently extended the laps time in order to account for satellite users. Please understand that these users typically have no lack of bandwidth, its the latency that kills them being some 600ms round trip time due to physics and stuff. Its hard to trump physics yes?
It only makes sense to level the playing field by making it turn based, just like every other MMPORG out there. After all, there is a really good reason they did it in the first place.
I am assuming that by turn based they mean that your have something akin to time slices within your area of infuence.
If it is not turn based (within time slices) then it becomes freely timing based. I got OC48... what you got? DRAW! BANG! I win.

Just kidding, I dont really have OC48....

ZODB

Seems like a perfect application for the Zope Object Database [zope.org]. It does all these things for you out of the box. And you get to write it all in lovely lovely python. aaaah.

Reinventing the Jini and JavaSpaces wheel

This would have been much easier if they had used Jini [jini.org] and JavaSpaces [artima.com]technology. There is even a commercial implementation [intamission.com] that supports incremental evolution of the distributed model.

Jini and JavaSpaces are being used in a variety of organizations to build large, distributed, reliable, scalable systems that integrate a wide variety of existing systems, including those written in languages other than Java. The technology seems a good match for this problem.

Patrick

Erlang

It seems a possible application for Erlang. From the FAQ (http://www.erlang.org/faq/t1.html): "Erlang is a general-purpose programming language and runtime environment. Erlang has built-in support for concurrency, distribution and fault tolerance. Erlang is used in several large telecommunication systems from Ericsson. The most popular implementation of Erlang is available as open source from the open source erlang site." Included in the libraries, there is also a soft real-time database called Mnesia (like JavaSpaces but extended with relational operations). There is some articles about using it in games here: http://www.erlang-projects.org/Public/projects/gam es/

Re:Here are some initial beta screenshots

Doesn't look any better than EQ :-)

Re:Here are some initial beta screenshots

Cute guy with his shirt off [mutablerealms.com]

The cute one has his skin off, too.

Re:I *SO* want this game...

and here's the solution:

- Don't buy the game (it's cheaper that way)
- Don't play the game (you can't get spammed/cheated if you're not a player)
- Don't install the game (you'll save lots of resources this way)
- Dont' wait for it to come out (you can spend the time with your loved ones or go on a hike)

Shalom

Re:That sounds complicated

Lets all go play Savage on Linux instead.

Is it actually available for Linux? The site seems to imply that it is currently Windows Only.

Re:That sounds complicated

You missed BattleZone 1 and 2, then? Good games.

Re:Here are some initial beta screenshots

Oy, what have we come to when Mutable Realms counts as pr0n for this poor guy?

Baby, you need to get out more. After a few drinks, they all look like the avatars of your dreams. :)