EgoWiki > Documentation > CharacterPersistence	EgoWiki webs: Main | TWiki | Know | Sandbox
	Documentation . { Changes | Index | Search | Go }

The Problem with Persistence

We want to build a rich game world, where characters can come and go, and where the player can make changes that have lasting effect. However, we want to make sure that we can do this without causing problems for scriptwriters. Consider the following example:

The player has a pet, named EvilPet, and the pair of them visit a shop. EvilPet steals something, and he's chased out of town with his owner. Some time later, EvilPet is polymorphed into a suit of armor. The player decides to take the armor with him in remembrance of his friend, and comes eventually back to the shop. The shopkeeper remembers EvilPet -- but he probably shouldn't be able to recognize him as a suit of armor.

Another example:

As before, EvilPet and his owner steal from a shop and are chased out of town. But this time, EvilPet is killed and left to rot as a corpse in the dungeon. Whether the player returns to the shop or not, the shopkeeper still remembers EvilPet -- how we remove EvilPet from the game without causing errors for the shopkeeper script?

Decisions

We need to make a few decisions in light of these examples about what happens to characters when they are no longer part of the current module. Here are the questions we need to answer:

• Do characters continue to exist after the the player leaves the module they are in?
• Do they continue to exist forever?
• Do all characters continue to exist for the same amount of time? If not, how do we decide/indicate which characters persist and for how long?
• If some characters don't exist forever, how do we handle scripts like the shopkeeper that try to remember them?
• If all characters exist forever, how do we keep from using too much memory?
• If characters change in form, how do we reconcile their current state with the memory of characters like the shopkeeper?

Discuss.

-- ElminI - 23 Oct 2004

The second comes back to active Referencing. An example would be a pet attempting to go to it's owner, or an item changing an attribute on it's owner.

I think that it's important that these two things remain separate issues. The first one can be solved easilly by using a value that is not a reference. "shopkeep.knowThieves[ target.id ] = 1" would be a very simple way of solving this problem. Value Lookup is what is needed, which is really just a by-value comparison. It does not need to be able to ever directly address the actual character through the remembered list. This really shouldn't be stored by reference in my opinion since that would complicate matters (saving and restoring).

The other problem, Referencing, is much more complicated, and deserves all of the above questions. It would be best if the solution to this problem was strictly enforceable, because failure to enforce it strictly will result in some very bad script errors (referencing nils, etc).

Some other applications avoid this problem entirely by not allowing, or atleast discouraging, storing references to characters like that. They are only allowed to store by-value, and must look it up each time with a function: getcharacter(id), so that it can be tested each time. Honestly, I think this may be the best policy. It explicitly promotes the scripter to check the return. It probably won't work well here though due to speed requirements.

We need a test case. Since the shopkeep and EvilPet example is more of a value comparing issue it does not quite work properly.

Case study: Soul Bonded Pet

In this situation the PC has a pet that was made to share some hitpoints with through a spell. The animal was accidentally left in a cave when the player went to town. When the player takes damage, how should it behave, considering the pet is missing?

Case study: Get my book!

Imagine a typical gopher quest. An NPC wants a book from the other side of the world. The NPC may want to hold a reference to the NPC friend that is giving the book, and the actual book itself, so that it knows the status. That way if the book gets destroyed or NPC gets destroyed, it doesn't hang there forever waiting. It can give a generic failed message.

Should this be allowed? How would the item be loaded and referenced? If it is allowed to be loaded out of it's module-context, the scripts should remain inactive for the most part, since it has no module to interact with. Loading the entire module seems like a bad idea as well - things may happen that assume the PC is present, not to mention the resources that will require.

I don't have a good answer to any of these. Picking an option and going with it sounds to be the best idea. Possibly just using nil if a character reference is missing, and have the scripts process defensively. This doesn't allow for verification that it will alway work properly though - there may be bugs lurking that cannot be spotted. Some of this could be avoided by having a special "nil ghost character" that would have some kind of default stats. This could lead to bad behavior, but would allow for error reporting, and reduce the chance of a fatal error.

I'll add more to this as I think on it.

-- ArakoN - 23 Oct 2004

Solutions to Case Study 2: 1 - Flexibility, or rather making allowance for the errors of the PC by allowing more flexibility in game. Using this solution, if the PC failed at a quest or somesuch he could attack the NPC that gave the quest and fight it in order to attempt to gain the reward by force. And if the NPC did not carry this award on himself than the PC would be able to follow clues that the NPC dropped to eventually find it himself. While I cannot think of a game that uses this at the moment, A.D.O.M. did have a great system where you could kill the people who gave the quests just to find out what would happen, though the reward was usually far less (if any) than you would have gained by completing the quest without fail.

2 - Rigid, or rather knowing that the PC will fail but not really giving the PC much recourse in case that happens. This is more or less like the solution that you proposed, but the NPC would probably then either send the PC somewhere else to get the object or come up with an alternate quest. But either way, the PC would have to do something else to make up for his failure.

I really do not think that the above two scenarios are mutually exclusive, and both could be used with few problem. However, I, myself, prefer the first one, but that is because I love flexibility and it makes more sense to atleast havethe option of killing NPCs that you don't like. Besides, it could always be an option for the player. I'll leave it to either someone else, or me at a later time to talk about this and player persistence.... I think that I opened more questions that I provided an answer to. One of them being: "If such a system is implemented, would other NPCs remember a players deeds?". I can't really comment on the difficulty or not of doing that in coding at the moment (though I may propose ideas later), sooo....

Solutions to Case Study 1: For case study one, I propose a simple little solution. Limit soul-bonded pets affects by range. Now of course, this doesn't make much sense on the outset due to the nature of "soul bonded", but then it would be really annoying to leave a dungeon, forget one has a soul-bonded pet, and then come back to check on it and find that it's dead. :P

This requires more discussion though, on all points, to get the thoughts flowing (and in the correct pattern).

-- SpyroVII - 23 Oct 2004

It sounds to me like the best solution then is to not allow loading of characters that are not in the module, because they arn't "live", but rather stored and packed. I think this is a good idea in a lot of ways. It is a lot simpler. All scripts would have to defensively check for "nil" references when looking at external/linked characters.

I agree with the gopher-quests - it should be open ended so that the character wants just the item, and doesn't care how you get it (beg, barrow, or steal). If the item somehow gets lost, there should always be a way out of it. If a book can get burned and destroyed, there should be a way to atleast end the quest in a failed manner. Some of this requires some communication between objects in spearate modules, but it can probably be faked by having some status variables attached to the PC, or maybe some global store. That may be a bit offtopic though.

Is there any way we can make this method more "hardened" against problems?

-- ArakoN - 23 Oct 2004

Status variables had been what I was thinking in my original writing, but now that you've mentioned it, a global store of some sort would probably also be good, if only because I get the feeling that it would perhaps be an alltogether simpler solution (atleast at the outset).

And by problems, I assume that you're referring to the inherent difficulties of keeping things open-ended, yet still allowing the "conclusion" of the game to be reached. The only way to keep open ended (and even semi-open ended) systems from becoming mucked up is by having quality modules and bug-free module/object coding. Which really won't be the case at all in the beginning, but eventually things will straighten out. If I misunderstood your question then feel free to let me know.

-- SpyroVII - 23 Oct 2004

Just to play devil's advocate, what's wrong with letting the garbage collector decide when to release characters? It's easy to disable all functional parts of a character (movements, etc.) without impacting scripts, and there's lots of convenient information that's stored with the character. There's also no question then of loading/saving particular characters; if the game state is saved, all referenced characters (whether active or not) will be saved with the rest of the game state. Possible problems:

• Memory: Characters themselves don't take up much memory, but if have characters that swap models or skins regularly this could become a problem. One way to easily take care of this is to dump models and save only filenames, then reload the files on respawn (if there's ever a respawn for that character). Another way is to wrap characters in some other transparent structure that becomes invalid (but protects against script errors) when the character is freed, and make scripts be expicit about when they need persistence from a particular character -- by default the character could be freed when it is no longer part of an active module.
• Increased complexity for scriptors: scripts would need to check that a character is part of the active module before doing things like initiating actions, but we can protect against this at the engine level, and it's not actually any different from what scripts would have to do for other schemes. Actually, this will be a somewhat less complex than working with status variables, because remembering a character is just a matter of saving a reference to it somewhere, and you can have whatever information you want from it. Also, status variables require cooperation from the referenced object's script, where saving a reference does not. The shopkeeper example is relevant here; why should EvilPet? know to set a status variable when it's dead, and how should the shopkeeper know to read it?

-- ElminI - 25 Oct 2004

Hrmm, the propensity for error that's present with this method seems to be greater than that of using a database like area for object storage. It just doesn't seem to offer too many benefits vs. the other method.

On the flip side, I'm not very concerned with memory (since given the average speed on computers nowadays, we can afford to take advantage of that if only just a bit), and it does sound rather convenient. Hrmm, I had intended to reply with other possible scenarios of why it wouldn't work (or would present problems), but the truth is that it actually sounds like a very workable idea, and it completely fixes the problems that would result from scripts trying to check a status field that doesn't exist. Hrmm....

-- SpyroVII - 25 Oct 2004

Here's some things I can think of that could be a problem with having an out-of-module disembodied character loaded and referenced:

• Environmental checks: If the referenced NPC recieves stimulus where it would need a find_target or map-related function, this would be a problem.
• Activities: Stimulus to a referenced NPC could make the NPC try to preform something it shouldn't be able to, like dropping something, creating an item to put in it's inventory, etc.

Since our goal with disembodied references would be to allow scripts to be fully transparent to whether an object is in-module, or disembodied, then the script will assume all of this stuff is available. Dangerous, and complex. This of course won't be a common problem, but that makes it almost more dangerous.

On the other hand, Elmin hinted at a possible solution by having some form of explicit reference that would allow for disembodied loading - that could work, since the scripts interacting with that would be fully aware of the limitations. I support that idea. It would make it easier to make a "quest manager" style interface, that can make sure all assets to a quest are active, and allow for an exit strategy that can inform the player in some manner. There would still be some of the above problems, but I think it would be more managable since it couldn't happen on accident.

Again though, I do not think a shopkeep should be storing a reference to anything they remember. The shopkeep does not ever need to actively reference an NPC that way. I think it adds complexity that's not necessary. I strongly believe that each object should have a unique ID that can be stored as value. It should make saving and loading of these characters a lot easier as well. I see no bad points to allowing this.

-- ArakoN - 25 Oct 2004

What I'm advocating is to allow scripts to be able to determine whether a character is available in the active module for interaction, and to issue warnings and take no other action when a script tries to treat an inactive character as an active one. The set of functions for interacting with active characters from scripts is fairly small, and it makes sense to do these kinds of checks anyway, so I'm not worried about this part. If we're going to make it possible for characters to be referenced at all, there will be some potential for error no matter what we do; we can't make the engine write the scripts so they're safe. The best we can do is recover gracefully.

The reason I'm against using unique IDs is that it doesn't solve the safety problem at all, or any other problem that isn't already solved -- references can be stored whether alternative methods exist or not, and to make the scripting environment safe from errors we really need to protect against inappropriate use of references to inactive characters. And if we're going to do that, we might as well use them to their full advantage and save ourselves from introducing the new complexity of unique IDs (I know the complexity isn't big, but we'll do better to avoid it if we can).

To address the shopkeeper problem specifically, the way to write shopkeeper scripts is to have them compare references to characters that are known to be active, for example on the receipt of some kind of alert or poll -- there's no other legitimate way to use the references in that context. We can't prevent shopkeepers from using their references any other way, but we also can't prevent them from doing arithmetic with IDs -- and we can't warn them if they do arithmetic with IDs, but we can if they try to interact with inactive characters.

But now I think I see the value of having a separate mechanism for long-term recognition of characters. We could do it this way:

self.memory[to_recognize:token()] = 1 -- or whatever we need to remember about this character

-- sometime later:

if self.memory[candidate:token()] then
  -- stuff
end

-- or 

if some_token:equals(candidate_char) then
  -- stuff
end

In this example, the token() method returns a userdata with the same C pointer as the character, but with a different tag. This new type has no member besides equals(), which returns true if the argument is the same token or the character it represents. It represents well in savefiles, since userdata is saved separately from the C structure it refers to. Fundamentally tokens aren't any different from UIDs, but the interface is a little more focused (a token is nothing but a token to scripts, where a UID is both a number and a reference to a character) and the implementation is sparser (the memory allocator takes care of generating the IDs and releasing them, and nothing needs to get saved with the character, since its index in the savefile effectively becomes its UID). We should still protect the character methods against inappropriate use, and characters will still need to retain references (the soul-binding thing is a good example of where this would happen), but this encourages scriptwriters to separate the active referencing of a character for continuing interaction from the long-term recognition problem.

-- ElminI - 25 Oct 2004

OK, I see you updated before I posted my reply. I'll post it below just in case there is anything else of importance in it. I came to the same conclusion you did however (with tokens). I'm happy with that solution. At the end below there is some questions regarding the saving/loading of characters as well. And I also agree that we should still protect character methods from misuse like that as well.

-- Look at it this way: Use of Unique ID's vs a reference is a design-time decision to limit the complexity that can take place within that script.

If a script designer chooses to use references, they must be prepared for all reference touching's to mean an active character, that is fully addressable. They must also be aware that any changes they make to them must be checked to be sure that character is actually in-module, or not.

The choice to use a unique ID, at design-time is a choice to not have to care whether a character is loaded or not. This will reduce the amount of possible problems in the script a lot. There are really only boarderline cases where out-of-module referencing is important. Very specific cases that require it be more thought out - not something everyone should be encouraged to do.

The very nature of the unique ID system is what prevents misuse of a reference. Doing any arithmetic on ID's would be a stupid thing to do, and I've never ever seen a problem like that occur. It would take a gross misunderstanding for someone to attempt that, compared to the much more "natural" attempt to use a function that isn't available due to the character being out-of-module. Also, with Lua, ID arithmetic is completely preventable just by using a "token" object of some sort, instead of an actual real-number.

Letting everything store a reference to other characters, like the shopkeep example (don't forget guards, towns, and misc dungeon creatures that are tracking the PC) is what's going to add the complexity. Suddenly we have to be VERY concerned with figuring out which characters need loading, and which characters don't. Almost every character will be storing references that could be out of module when they never ever need to access any of the character's functions, except to see whether they recognize them.

Also, without having an internally unique ID, how will we be able to link references to the actual character upon load? It's easy if a state is serialized, and deserialized, since the character reference will be trackable. If a character goes from one module to the other though, that reference will change, correct? Or what if a new creature is created with the same reference that a creature in a previous module had (unlikely, but possible). When the PC returns to the original module (town?) then linking the stored references with the character become difficult. Unless I'm missing something?

-- ArakoN - 25 Oct 2004

OK, glad we finally came to the same conclusion. So, the saving troubles:

When I looked at this a while ago, I came pretty quickly to the conclusion that the only good way to deal with characters moving between modules is to keep the lua state open and simply direct scripts to re-orient themselves when their characters are respawned into another module (probably a respawn alert). Part of this is already in CVS; when you start the game, the Lua state is opened and never closed (even though you might select different modules) until you quite the application completely. So the only time references need to be recreated is during saving/loading, and this is pretty much taken care of by the savefile framework (at least, it should be). In that case, the unique ID is the pointer to the C structure -- on saving, it gets translated into an index in the savefile. On loading, it gets translated back to a C pointer, and the framework is responsible for making sure that every reference gets translated properly. This is why I was saying that the saving/loading concerns are basically orthagonal to modules -- we have to be able to maintain a persistent state through module changes, and indeed we want to support (if I remember our discussion earlier) multiple modules being active at the same time (for indoor areas and small sub-modules). This is quite a radical departure from the 2.22 model, which is to dump all the characters after each module is completed, and start from scratch with each new module. This is also why I'm not as concerned about using references as tokens (even though I still agree we should separate the two).

-- ElminI - 25 Oct 2004