(We know that Gametap will return Uru: Ages Beyond Myst to their rental service. That's the original single-player Uru game -- without even the expansion packs -- and you can't hack Gametap downloads to add new content. The news is therefore irrelevant to me. I'm interested in Uru as a shared, extendable universe.)
My last post worked around to the idea of a fan-created Uru game. Not an extension of the existing Uru, not managed by Cyan; a completely new system, open-source and run by the players. This post contains my sketch of how to do it.
In other words, this post will be too technical for the game nerds, and too handywavy for the programmers. I have no code to back it up. I have experience running a multiplayer gaming service... which has never had more than a dozen people online at a time. So -- dismiss away, if you want.
On the up side, the technical issues here are not specific to Uru or to the Myst universe. Want to run a graphical MMO adventure? Here's a plan. Go to town with it.
This post does not rise to the level of a proposal. It's a sketch; it's my answers to a bunch of hypothetical questions. I'll argue the merits of each point separately; you don't have to accept my ideas on one point just because you buy another.
This is a conservative proposal; it's a system I think I could build. However, I am not volunteering to build it. Planning is the fun part. I have too many unfinished projects already to pick up a new one. But I can gab about ideas all day. Lucky you, eh?
I will rely entirely on open-source software. Cyan's Plasma engine will not be used.
I do not want to infringe on Cyan's ownership of the Myst series, or their ability to bring back Uru Live someday. (It will always be true that Cyan might bring back Uru Live.)
The plan will be implemented by a small group of part-time programmers. (Not crazy genius programmers; just programmers.) (Maybe I am a crazy genius programmer, but I'm not volunteering to build this, right?)
The result does not have to be as impressive as Uru Live. (See previous point!) We can tolerate shakier graphics, less scalability, and so on, because we have no budget and nobody's doing this full-time.
The game world consists of a set of small areas -- Ages, in Myst parlance. Each Age is replicated in many instances. Any player or group can get an instance of their own; instances are cheap to create on the fly.
The goal is for players to hang out in the environments and chat.
The goal (also!) is for players to contribute new Ages, and explore each others' Ages.
Socializing happens in medium-sized groups -- perhaps twenty or thirty. That's the size of a conversation. Getting a hundred people together in one place would be neat, but it's not a design goal.
Exploring is done alone, or in small groups. Interactive environments stop being fun when a bunch of strangers are jumping in front of you and messing with your stuff. Adventure-style game logic -- with specific, discrete actions and unique results -- tends to have a small number of actor roles. When you have a hundred people cooperating on a task, that's CRPG design, not adventure design.
In other words: while I want a game world that encompasses hundreds or thousands of people, I'm not worried about rendering them on the a single screen or managing their simultaneous access to a single real-time puzzle.
I envision my game system being much more heterogenous. I don't like the idea of enforcing one server for everybody, or one player database. I don't even want a single group of people operating the game. This should be an open system. That means anybody can run a shard. Download the software and set it up; you're on.
Shards should be completely independent. We don't want one badly-maintained shard to corrupt others. We also don't want the whole system to die because one person went on vacation. By keeping shards separate, we can keep problems contained. We can also address scaling issues -- crudely but effectively -- by setting up more shards.
Anyhow, there won't be any notion of global avatar progress, because there is no global set of goals or achievements imposed. So there's no real reason to share avatar information between shards.
Naturally, I expect one shard to be the common place where people hang out. There could also be a development shard for the Writers, a testing shard for the Maintainers (or maybe several), and private shards for small groups or for the hell of it.
When you decide to run a shard, you decide what Ages it will contain. This will be a plugin system; you download an Age plugin (Python code) from whoever wrote it, stick it into your server directory, and presto. Some Ages may be used in all shards, others might be featured by a single shard.
There can be a published list of public shards -- I'll accept that much centralization. (It could be as simple as a wiki page.)
The Called, exiled from the Cavern, gather on the surface to discuss their future. Their Relto books no longer work; Yeesha has apparently abandoned them. But the Guild of Writers announces a breakthrough: intensive computer analysis has found some of the underlying structure of the D'ni linking symbols. Surface dwellers are beginning to create effective Linking Books.
While the Guild cannot replicate all of Yeesha's techniques, they have discovered some useful tricks. Instancing is one. More important is the ability to replicate linking pages -- literally print them out on an inkjet printer. (This only applies to modern-made Books, not to Relto or D'ni Ages.)
This means you can carry around a sheaf of link-home pages on your belt. When you use one, it remains behind (and disintegrates -- biodegradable!) but you still have the rest of the sheaf. Similarly, if you come across a Book that you want permanent access to, you can lift out a linking page and take it home to your library. As long as the Book's creator included a whole sheaf of pages, the Book won't be damaged, and there will be plenty for future explorers.
Your "home" in this scenario is not a beautiful island Age. It's just a small room in your house or apartment -- a closet that you've converted to an office for your Uru work. Very plain. (You might get to customize the color of the walls.) You have a desk, a bookshelf and a stack of linking sheets. This is your entry point to the shard, but its game function is more like the Nexus than like Relto.
You have a book for a community Age (Ahra Pahts or something like it). You have, or will gather, more books and pages as the shard provides them.
Since all these Ages are written by surface dwellers, we have no access to any D'ni Age we are familiar with. Nor will you find any links to the Cavern, or any other place on Earth besides your home. There may, however, be traces of the D'ni out there in the Ages we explore -- perhaps even other civilizations with linking technology. The Tree has many leaves, and no one knows what the next one might hold.
(This scenario takes a deliberately conservative approach to Cyan's intellectual property. I assume that the people running this game would want Cyan's permission, but would not want Cyan to take an interest in controlling the game world. To get that, I take a long, ostentatious, good-faith step away from their financial interests. Cyan has always hinted that their outlook on player-created Uru content would be "No D'ni history, no D'ni Ages." I can live with that.)
(Of course, if Cyan is cool with us using their toys, that's great. I'd want to at least use the familiar Linking sound effect!)
In a fan-run game, I don't think one group should be in control of all the story. That model doesn't even start to work without a lot of community trust of the game-masters -- a different kind of trust than mere technical adequacy. Cyan held that trust tentatively, and (for a lot of players) squandered it. I do not propose to re-vest that trust in another cabal.
Rather, I'll let everyone do their thing. (The Myst logic of many separate Ages encourages this.) If something good emerges, it'll have to emerge on its own. So this section of the plan is up to everybody. Yes, including you.
(Security is a weak spot. I will address this later.)
Using Jabber as the transport layer for a game is a compromise. I'm using it because I'm used to it. I've implemented a board-game system (Volity) using Jabber, and I am writing this plan to work very much the same way.
Basically, the shard server is a Jabber bot. Each Age instance is a Jabber chat room, managed by an instance server, which is also a Jabber bot. Your client is a Jabber client with a fancy graphical display; when you enter an instance, you join that chat room.
More importantly, Jabber is not optimized for many tiny messages. When you walk around in Uru Live, you are sending a constant stream of avatar position updates. My Jabber-based system can't do that. (The XML snippets aren't huge but they're bulkier than a typical MMO walk packet.) So it would send those updates much less frequently. Maybe once every ten seconds, or less often than that. In a crowded instance, the server might have to filter that down even further. Chat, emotes, and action messages would have priority.
Basically, movement of other avatars would look like occasional jumps. You wouldn't get the walking-around that you're used to. You might fail to see someone who just arrived, or see somebody present who really walked away thirty seconds ago. A static conversation -- several people standing in one place talking -- would be fine, though.
As I said, it's a compromise. I think of chat and puzzle-solving as being more important to Uru than smooth animation.
(By the way, when I say movement is jumpy, I'm talking not talking about your movement. Your viewpoint will move smoothly, like in any 3D game.)
(An alternative: you could treat movement updates as a very special case. Don't send them via Jabber; have a separate stream, binary-encoded and UDP. You'd need to treat the UDP data as unreliable, superseded by any Jabber data about that avatar. But I was planning to do something like that anyway. See the "Synchronization" section, below.)
Other possibilities: Irrlicht; NeoEngine; Crystal Space.
Note that I have listed 3D graphics engines. I am not considering game engines, MMO engines, or online virtual world systems. I don't want a software package that comes with communication or server code; I'd just have to rip it out.
It may sound stupid to plan to write MMO communications code "from scratch". But it's not really from scratch. Jabber is a complete, working, scalable message system. And the game server code needed for adventure gaming is really not complicated. Jabber bots that talk to a database, and write a limited amount of instance state, should be easier to maintain than a third-party virtual-world system.
(Of course it's perfectly reasonable to plan an Uru game using an existing MMO client-server solution. It's just not this plan. For the sake of completeness, here are some open-source virtual-world projects: Uni-Verse; Croquet; Project Darkstar.)
By the way, the Age is identified by a URI (a string), rather than by a sequence number. (URIs are great. Anybody can invent a URI without worrying about collision -- just pick one based on your home domain or email address.)
The server module is downloaded once by the shard admin and installed into his shard config. The module contains the URL of its Age package. (Or the shard admin can specify a mirror URL, I guess, if he doesn't want to rely on the original author's web server.) The URL can be any web address.
There's also a general UI package associated with the shard. This contains clothing, menu images and scripts -- the stuff that the player can see in any Age and instance of the shard. (But note that different shards can offer different UIs. So you might have a standard Uru KI interface on one shard, and an enhanced KI on another.)
When a client logs into a shard, it downloads the UI package; when it links to an Age, it downloads the Age package. These are simple HTTP downloads, using the URLs that the server provides. We use an HTTP HEAD request (or If-Modified-Since) to skip the download if the client already has the package cached.
(Since downloads will be slow, we might offer the player a "download in background" button on the download screen. That effectively cancels the link, and lets him wander around the original Age or go somewhere else while his client finishes the download. We don't really want a "download all Ages" option, because there's no upper limit to the number of Ages on a shard.)
The client caches packages. So it will have to offer a way to clear the cache. Probably just a list, showing the last time each one was loaded, and a button to delete the older ones.
Note that this system is very modular: an Age package must contain every resource used in that Age. This is simple, but not efficient. (If a texture appears in three Ages, you'll download it three times.) I think simplicity is more important to begin with. If resource sharing becomes important, I'd do it by wrapping the shared resources up as a separate package; then the server module requests a list of packages (instead of just one).
There are several upgrade scenarios:
When you start your client and select an avatar, the client contacts the shard server. The server notes that your avatar has no active game session, so it tells you to link to your "home" instance.
(By the way, Jabber lets you log in with multiple clients at the same time. My game architecture is fine with that. You can drive two avatars around on different shards, or even on the same shard, as long as they're different avatar records.)
When you enter a new instance, the shard server starts up an instance server, which is also a Jabber bot. The instance server creates a Jabber chat room and sends the address back to your client. The client joins the chat room, the instance server sends you the local state (including other avatars), and then you are officially in the instance.
When your client disconnects, your instance server is notified (Jabber handles this automatically). It can then notify other players (in that instance) that you've vanished. (The shard server doesn't have to care that you've logged out.)
(Emotes are regular Jabber chat messages with a special tag naming the emote. Jabber lets you add special tags to any message. Other clients, like iChat or Google Talk, will just ignore the tags.)
Jabber gives you a friends-list. Whenever you log on, Jabber automatically notifies your friends; again, the instance server (and the shard server) don't have to be involved. Your client will add special tags to your presence notification to indicate which Age you're in.
(This won't work identically to Uru Live. For example, when you friend someone, they have to accept it -- that's a Jabber policy. And we can have a "recent" list, but it won't include the player's Age location, because you only get presence notifications from friends and people in your chat room.)
Certain polygons will be marked in the Age file as "floor". Your avatar can move around a floor polygon (strictly two-dimensional movement, no jumping). When you reach the edge of a floor polygon, if there's an adjacent floor polygon in the mesh, you can continue on to it. If not, you stop.
Ladders are "floor" polygons with a vertical orientation. The client doesn't have to treat them specially, except to change your movement animation. Same goes for water. You can't fall through the world because there's no falling. You can't get flung through the ceiling by a misaligned ladder. Everybody is relieved.
(Yes, that ladder flinging thing really does happen with Uru's Plasma engine. Take a look at the Guild of Writers' instructions for ladder placement. Not pleasant.)
Some "floor" polygons trigger scripts. We will get more into this later, but this is how you do a falling or jumping action if you want one in your Age. You mark a specific polygon with a script saying "When the player walks here, move him down to that floor polygon there." (Or panic-link him with a falling animation.)
I'm assuming that simple animations -- a door opening, a globe rotating, an avatar falling straight down -- will be easy. However, Uru Live currently has a huge palette of fluid avatar animations, for emotes and story actions. I don't think we're going to replicate those, not right off the bat. We're certainly not going to have the perfect finger-on-button, hand-on-rung precision that Uru Live offers.
But I've already conceded that avatar position updates will be slow and jumpy. So weak animations don't really make things worse. You're just going to have to get used to avatars standing around and being stiff.
If someone then comes along and offers better animations, that's great.
This scheme is optimized for Myst-style adventure gaming. That means small Ages, with a relatively small amount of persistent data. (Kadish Tolesa has maybe fifty stored values for all its puzzles.) And not too many people interacting with one instance's puzzles at a time.
(If you want gigantic Ages with lots of players interacting with lots of objects, I can't help you. That's a hard problem. Go talk to the people who run Second Life or OpenCroquet.)
The general rule is: the outcome of game actions will be decided by the server. (This is in contrast to Uru Live, which left a lot of decisions to client scripting.) Jabber messages are reliable and in-order, so as long as the server processes commands sequentially, consistency is assured.
This costs. Reliable, in-order messages can be slow; they can bottleneck. So we have to be careful to make server-side decisions quickly.
Therefore, our other general rule is: the instance server keeps all the instance state in memory. It shouldn't hit the vault database every time a door opens or closes. For transient events (like the Delin/Tsogal door opening), it won't hit the database at all. Permanent state changes (like the Bevin book room door opening) will have to be written to the vault, but that can happen as a periodic checkpoint. As much as possible, we want to avoid a player action blocking on server database work.
Nonetheless, when you push a button, there will be a brief pause before you see a result. If the server is stressed, this may be a long brief pause. In general, you will be "stuck" to the button until that server reply comes back. (Although in some cases this won't be necessary.)
This brings up the flip side of game actions: moving around. I've said that this system is pretty sloppy about player location. The client knows where you are, but the server (and other players) aren't updated that often.
"Great," you say, "leave location up to the client." But in many cases, game actions are triggered by location. (For example, the Cleft bridge that breaks when you step on it. Or the Gahreesen auto-doors. Or the Teledahn prison.)
It's easy to mark some "floor" polygons as having (client) scripts, so that stepping on them has an effect. But I just said that game actions suffer a brief pause, while the server replies with the result. Should you be beset by these pauses while you walk around an Age? I think not. Free movement is a big part of the immersive experience, and it's worth making an exception to keep it smooth.
The problem is, if you don't freeze when you enter a script region, you run the risk of "outrunning the script". (You start to run across the Cleft bridge. The server is running slow, and by the time it reacts, you've already reached the other side. Whoops! Now you're somewhere impossible. Not a big deal in the Cleft, but it could be a plot bug in another Age.)
To prevent this, we give these regions a "sticky" attribute. When you enter or leave a sticky region, the client sends a location update to the server. (The client does this regularly, of course, but it is careful to send one when it crosses a sticky boundary.)
Until the server acknowledges this update, you are logically stuck to the region. As far as the client is concerned, you're frozen. You can still move your avatar around, but you can't push buttons or trigger any other scripts.
Hopefully, this hidden freeze will be brief (a quarter second, just like being stuck on a button). Then the server reply comes back, and everything continues on; you never ever notice it's happened.
But say the server replies "Hey! The bridge just broke!" As far as the server is concerned, you were on the bridge when it broke. Game logic decrees that you must fall. So the client runs the "break and fall" event -- regardless of where you were. You teleport back to where you're "supposed" to be (the break-point of the bridge).
Again, in the normal case, it's only been a quarter second since you hit the bridge. Falling makes sense. But if the server is slow? Well, you might be teleported back several steps. Maybe even from the far side of the bridge. Too bad. Adventure game design needs reliable plot logic.
(Mind you, not every script region requires this kind of rigid control. If a room's lights brighten as you enter, that's just cosmetic; it's no big deal if it's out of sync with your "real" entering time. So that region wouldn't be marked sticky.)
There are more details to this plan. (What if you enter another sticky region while logically frozen? What if you fall off a cliff? What if you log out?) I won't go into them, because this section is already way too long.
Really large Ages: We'd like to support areas which are very large, but are divided into sections. The client doesn't render distant sections, thus saving polygons. (UL does this in Aegura. If you haven't noticed, it's because it does it well.) The 3D engine should support this out of the box; it's an old trick. Hopefully, it'll be easier than the Plasma engine's "multiple page" trick -- I understand that's kind of a hack.
Too many avatars: Even in a simple Age, if enough avatars show up, the 3D engine will eventually choke. The crude solution is avatar pop-in: the client only renders the fifty closest avatars to you. (Once again, we shovel all the graphical problems onto avatars.) A less crude solution (which UL employed) is levels-of-detail on the avatar models.
Client threading: Uru Live had a lot of problems where the client's Python scripting would hold up the Plasma rendering. (That is, some network traffic would arrive and cause a frame-rate glitch.) I haven't looked at the Python-Ogre interface, but I really hope we can avoid that. It will require careful implementation of the interface. I expect this to be the hardest part of the whole system, really.
Instance server overload: The biggest traffic load on the instance server is avatar movement updates. (Recall that chat, emotes, and Age-location are handled by Jabber directly.) In a crowded Age, the server will have to start ignoring movement updates. (Not the important scripted ones, but the normal step-by-step updates.) Jumpiness will get worse. We might even need a signal to tell the client to send fewer movement updates.
Vault database overload: All persistent updates in a shard are going to a single SQL database. (I am not smart enough to design a distributed database cluster. I can install MySQL.) My only plan is: don't create Ages which require lots of data written frequently. Keep everything in memory. Checkpoint occasionally; if possible, checkpoint only the state which has changed. (But you have to be careful to keep the in-DB state of an instance consistent.) If problem remains, bring up another shard somewhere.
Well, small UDP packets, anyway.
As I've said, this is a compromise strategy. Jabber buys you a lot: good authentication and encryption is not easy! And it costs a lot. But I'm not just making a quid-pro-quo argument. I'm trying to pick out what's actually vital to an MMO adventure game, and what people are simply used to.
Free-ranging, real-time movement is a recent addition to the adventure world. Myst 1 through 4 used discrete movement -- click on a location to jump there. Most adventures still work that way, although a handful (including Myst 5) have followed the Uru model.
Adventure interactions require you to be in particular locations at particular times -- the times when you interact. (With puzzles or with other players, it's the same constraint. Although other players are more flexible, since you can generally interact with them anywhere.) In between interactions is essentially gravy. Important, immersive gravy; it's when you absorb the game world. But in terms of what you do, the game doesn't care whether you're looking around at fixed angles, or turning in place, or moving freely. (Not until you enter one of those scripted regions!)
Uru players are used to the MMO model, where every avatar in view seems to move freely. (With occasional glitches; but then, Uru has much better movement animation than most MMOs.) But I'll note that not all adventure gamers are convinced. One of my friends tried Uru and then told me that he wanted a "click to jump" interface. The 3D movement repelled him; he hated maneuvering towards the button or lever or whatever he had his eye on. He just wanted to decide to be there.
That's not how I feel; I like free navigation, and so I've written this proposal to have it. But we should remember that there is no unassailable requirement for this stuff. You could take my plan and drop the free movement entirely; build the game engine to render a series of still views, with other avatars clustered in nice natural-looking groups around their respective location points. It would still be an MMO adventure game. (And you'd be able to drop the section about the sticky regions. I mean, eww. And I wrote it.)
(Old versions of Python contained a "rexec" module which was supposed to offer sandboxing. This is now deprecated, because it doesn't work -- Python actually got too powerful for it. There is also a secure-Python project I've seen, but I don't know if it's been verified or tested at all. I'm keeping an eye on it for possible future use.)
We can't ignore this issue. Anyone who downloads a free game and runs it is trusting the programmer; but this system is an extendable game, and it's intentionally as open as possible. We want a flood of new Ages -- we can't ask the user to trust everybody who contributes to that flood.
We're going to have to start by manually inspecting Age scripts as they are contributed. Ideally, we'd have one "safe" shard -- the common one, to which new players are directed first -- whose admin guarantees:
This is additional headache and bureaucracy, which sucks. Development shards, testing shards, and private shards will certainly take a more freewheeling approach, which means we'll have to educate users about the dangers. That sucks too.
The up side is that inspecting Age packages should not be slow or difficult. Normal Age script code will be simple; it will call almost nothing except standard game APIs. If a script calls open(), socket(), or most general Python library APIs, something is wrong. If a script is obfuscated, something is wrong.
More of the Ongoing Uru Review
Other Uru Stuff