Jump to content

How to completely delete an object (reference) from the game.

lee3310

By lee3310
in Creation Kit and Modders

 Share

Recommended Posts

lee3310 Collaborator

lee3310

Posted
Posted (edited)
Hi, i experienced a weird bug recently and there is no way to reproduce it cause my save is almost 4 years old and i don't think someone else preserved an Fo4 save file that long :sweat: .
So after a while, any save i make is unloadable (infinite loading screen) and if i revert back to an older save, i can play and save for a while until the save get corrupted again. The older the save a revert to the longer it takes to trigger the bug (could be more than a month of normal gameplay if i load a very old save). After redownloading all the files and reinstalling the game twice, i discovered that if i disable one mod (that i worked on for a long time and relatively script heavy) i can load the broken (unloadable) saves again, so it's not a file corruption but rather something building up over time until the game fails to load the save.
Since the mod i disabled spawns an infinite number of NPCs and delete them, i am suspecting that the game does not completely delete the refs when using "Delete()" or killessential(), even if DPPI shows nothing.
I'm saying this because when i "prid" a deleted ref, it still shows up in console ("[D][PP][T]"), whereas vanilla deleted NPCs from random encounter for exemple shows "none" after cell detach or unload.
To test my theory i need to know how to completely delete a ref from memory so it can't be "prid" on anymore and does attaching a script to a ref prevent that or not ?
PS
No stack dump in papyrus log and nothing wrong with the save files when i check them using fallrim tool.
Edited by lee3310
Link to comment
Share on other sites
hereami Proficient

hereami

Posted
Posted

Interesting now, that's relevant to what i asked too and worried about in advance. No answer here, just questions.

 

What is method for said spawns and were they all Essential?

Should such phantoms be accessible for Quest fill too? Need to know about their possible presence without console commands.

After disabling the mod and saving does the file size change noticeably? Either way, nice to know it's not fatal and saves, probably?, might be "repaired" by disable mod - save - enable mod back untill it bloats again. Could be worse then.

Link to comment
Share on other sites
LarannKiar Mentor

LarannKiar

Posted
Posted (edited)

If the deleted ([D]) ref is persistent, then even if it's IsCreated(), without making it non-persistent, it can't be deleted normally. ( [PP] basically means persistent (or "Persistent Promoted" it seems but I'm still not sure..) ). ( ForcePersistent causes the console selected ref to become its own "promoter" so it can't remove persistency caused by other Forms either ).

 

 

I usually don't advertise tools or mods I made.. but I added a MarkForForcedDeletion(), IsPersistent() and GetPersistentPromoters() to Garden of Eden (the script extender). The first one ignores all removal preventing mechanism (first of all, persistency) and force marks the reference for deletion ( which takes place on the next native "cleanup refs that no longer needed" event, typically OnUnload() or the next time a save game created after calling it loads ). The third one reports all Forms that keep ("promote") the reference persistent. Unlike DPPI, it looks beyond VM ("Papyrus") persistency. For example, it sees persistency caused by SetLinkedRef and being a Quest Alias, etc..

 

Calling MarkForForcedDeletion() on editor-placed referenes causes the game to treat them as if they were just added to the game the first time ( on the next save load and only if the parent mod the it originates is still active ). Permanent solution for IsCreated() references though.. however, for them, any Papyrus variables which may point to such ref will point to invalid memory addresses, "nullptr" ( interestingly, immediatelly after calling MarkForForcedDeletion() and not after the actual cleanup event which truly makes the ref disappear ). RefCollectionAliases shrank though.. unsure why.

 

 

The alternative is manually deleting the ref in Fallrim Tools. The (possible) problem with it that you may not remove the ref's actual memory address. (GetMemoryAddress() is also available in Garden if you'd like to use other tools to "nullify" the address while it's running, which can hardly be recommended :smile:).

 

Edit: and yes, you're right. The game sometimes don't remove references that should be removed, causing them to exist (persist..) in the save forever.

Edited by LarannKiar
Link to comment
Share on other sites
lee3310 Collaborator

lee3310

Posted
  • Author
Posted
  On 10/15/2023 at 7:36 PM, hereami said:

Interesting now, that's relevant to what i asked too and worried about in advance. No answer here, just questions.

 

What is method for said spawns and were they all Essential?

Should such phantoms be accessible for Quest fill too? Need to know about their possible presence without console commands.

After disabling the mod and saving does the file size change noticeably? Either way, nice to know it's not fatal and saves, probably?, might be "repaired" by disable mod - save - enable mod back untill it bloats again. Could be worse then.

No essential, spawned using PlaceActorAtMe() with no extra argument other than actor base , and yes, i've got the exact number of unloaded persistent refs using "SKK Object Counter" (around 13300), so alias fill is a good option to get to them.

Link to comment
Share on other sites
lee3310 Collaborator

lee3310

Posted
  • Author
Posted (edited)
  On 10/15/2023 at 9:50 PM, LarannKiar said:

If the deleted ([D]) ref is persistent, then even if it's IsCreated(), without making it non-persistent, it can't be deleted normally. ( [PP] basically means persistent (or "Persistent Promoted" it seems but I'm still not sure..) ). ( ForcePersistent causes the console selected ref to become its own "promoter" so it can't remove persistency caused by other Forms either ).

 

 

I usually don't advertise tools or mods I made.. but I added a MarkForForcedDeletion(), IsPersistent() and GetPersistentPromoters() to Garden of Eden (the script extender). The first one ignores all removal preventing mechanism (first of all, persistency) and force marks the reference for deletion ( which takes place on the next native "cleanup refs that no longer needed" event, typically OnUnload() or the next time a save game created after calling it loads ). The third one reports all Forms that keep ("promote") the reference persistent. Unlike DPPI, it looks beyond VM ("Papyrus") persistency. For example, it sees persistency caused by SetLinkedRef and being a Quest Alias, etc..

 

Calling MarkForForcedDeletion() on editor-placed referenes causes the game to treat them as if they were just added to the game the first time ( on the next save load and only if the parent mod the it originates is still active ). Permanent solution for IsCreated() references though.. however, for them, any Papyrus variables which may point to such ref will point to invalid memory addresses, "nullptr" ( interestingly, immediatelly after calling MarkForForcedDeletion() and not after the actual cleanup event which truly makes the ref disappear ). RefCollectionAliases shrank though.. unsure why.

 

 

The alternative is manually deleting the ref in Fallrim Tools. The (possible) problem with it that you may not remove the ref's actual memory address. (GetMemoryAddress() is also available in Garden if you'd like to use other tools to "nullify" the address while it's running, which can hardly be recommended :smile:).

 

Edit: and yes, you're right. The game sometimes don't remove references that should be removed, causing them to exist (persist..) in the save forever.

 

Please do promote tools you made, especially when it's highly relevant to the matter at hand, i was about to request a forceDelete() function before reading you comment (MarkForForcedDeletion() saved the day).

So i did a lot of tests using Garden Of Eden functions, first it was a linkedRef holding the refs into the game even if i was linking them to "none" before delete. Then, GetPersistentPromoters() was just returning the actor's RefID "Forms locking this actor = ["0xff3efda4"]", you said that ForcePersistent console cmd will cause such results, since i'm not flagging the refs as persistent with placeAtme(), do you know of any other way/function that can make a ref "to became its own promoter" ? is there even a papyrus counterpart to ForcePersistent ?

And you where right about fallrim too, i managed to fix one of the broken saves using it but it's not humanly possible to manually delete "10333" ChangeForms (AHCR) and script instances, and the risk of deleting a wrong instance is relatively high if you filter by attached scripts. That's why i'm going to use "MarkForForcedDeletion()" to clean up my last loadable save, fix the mod, and see how it goes.

Edited by lee3310
Link to comment
Share on other sites
LarannKiar Mentor

LarannKiar

Posted
Posted (edited)
  On 10/17/2023 at 4:35 AM, lee3310 said:

 

So i did a lot of tests using Garden Of Eden functions, first it was a linkedRef holding the refs into the game even if i was linking them to "none" before delete.

 

Ref1.SetLinkedRef(Ref2) ( regardless of akKeyword ) should add Ref1 to the persistent promoters of Ref2. ( Adds Console [PP] if the Ref2 wasn't editor-set Persistent (EP] ). SetLinkedRef(None) revokes the promotion. If you still see [PP] then it's either another promoter, or a "strictly Papyrus related" persistence that DPPI can show ( e.g., "anotherRef is registered for whatever event from Ref2" ), or the native ForcePersistent ( the game uses it quite frequently when it thinks the reference "should be" persistent ).

 

Why the console sometimes shows [EP] instead of [PP] and vica versa if persistence is derived from both is currently unknown ( [PP] is more specific, [EP] is more general ). For editor-placed and editor-set persistent refs [EP] is preferred even if they have promoters, for IsCreated() refs [EP] is shown only when [PP] is invoked. If the reference has only one cause of persistence, the console flag changes to the appropriate one.

 

  On 10/17/2023 at 4:35 AM, lee3310 said:

Then, GetPersistentPromoters() was just returning the actor's RefID "Forms locking this actor = ["0xff3efda4"]", you said that ForcePersistent console cmd will cause such results, since i'm not flagging the refs as persistent with placeAtme(), do you know of any other way that can make a ref "to became its own promoter" ?

 

Most of the time, it's due to the game's design ( see "invalid address issue" below ).

 

An attached script can also "self promote" the reference if the script holds the reference as script property ( e.g., PowerArmorFXScript >> SelfRef ).

 

  On 10/17/2023 at 4:35 AM, lee3310 said:

is there even a papyrus counterpart to ForcePersistent ?

 

Soon there will be.. :smile: I finished the testing of ( Bool Function SetEditorPersistent(ObjectReference akReference, bool abSetPersistent = true) native global ), I'll add it to the next update. It sets/clears the [EP] level of persistence.

 

I haven't looked into how editing the array data of the promoters would affect the promoters ( so not the promoted ref ) but the risks of breaking things is very high so I think I'll just make a vanilla-like ForcePersistent(bool abForcePersist) instead of granting full access to the array data. I haven't decided yet..

 

 

The invalid address issue:

 

The console command ForcePersistent exists because "forced persistence" ( or the "self promoted persistence" ) is heavily used by the engine. It's not exclusively Forms and Papyrus variables that can rely on a reference. The game and save games may rely on them as well: for example, to circumvent the "the invalid address issue".

 

I wasn't sure whether to add direct persistence manipulation or not to the script extender.. One of the problems with editing persistence ingame is ( since the data handler performs "the main Form validation" when the game starts and not when a save is loaded ) when one clears the persistence on a reference ( call it "clearedRef" ) while the game is running then decides to reload an earlier save ( "previousSave" ) as a second thought, before doing so, one may need to completely exit the game then relaunch it due to the fact that the "cleared flag" remains in the memory and the game can't handle that if the game data has already been initialized upon loading previousSave that still "thinks" the reference is persistent.. which means, if clearedRef is not in the loaded area where previousSave was created then the game lands on an invalid memory address which undoubtably causes CTD ( as the developers couldn't consider this scenario ). This shouldn't reoccur though. Same applies to MarkForForcedDeletion() on persistent references.

 

So, if you call it on a persistent reference and would like to undo it, reload a save that was created in the area where the reference is ( this shouldn't be a problem in most cases ) or exit the game and relaunch it. Loading any ( pre/post created ) save after relaunch should be okay because the game already had a chance to "react" to the modifed [EP] or [PP]. By the way, the vanilla ForcePersistent comes with this limitation as well ( in case the only promoter was the reference itself and there was no editor-set persistence. In other words, when it completely clears persistence ).

 

  On 10/17/2023 at 4:35 AM, lee3310 said:

That's why i'm going to use "MarkForForcedDeletion()" to clean up my last loadable save, fix the mod, and see how it goes.

 

The safest way to use MarkForForcedDeletion() is probably the following.

 

Call GetPersistentPromoters() and DPPI on the reference to know what Forms and VM variables keep it persistent.

 

Try to nullify and clear them first ( for example by temporarly editing the affected scripts to release the reference ) to prevent the same situations that a mod / Form-record removal can cause.

 

Create a hard save while the reference is loaded, just in case..

 

Call MarkForForcedDeletion(), then create another hard save. Exit the game, open it, then check the Papyrus log for errors, see if there's any "critical". Note that the game may reassign dynamic RefIDs later so the fact you can find it later in Fallrim Tools or in the game doesn't mean the reference wasn't removed.

 

 

Note: the risks of removing a persistent reference should be much lower if it has no promoters and no VM variables point to it ( and that editor-set-only persistence is still vulnerable to the invalid address issue ).

Edited by LarannKiar
Link to comment
Share on other sites
lee3310 Collaborator

lee3310

Posted
  • Author
Posted (edited)

Thanks for those valuable informations, i have a better understanding of how the game handles persistence (delicate subject).

 

  On 10/17/2023 at 4:17 PM, LarannKiar said:

 

The safest way to use MarkForForcedDeletion() is probably the following.

 

Call GetPersistentPromoters() and DPPI on the reference to know what Forms and VM variables keep it persistent.

 

Try to nullify and clear them first ( for example by temporarly editing the affected scripts to release the reference ) to prevent the same situations that a mod / Form-record removal can cause.

 

Create a hard save while the reference is loaded, just in case..

 

Call MarkForForcedDeletion(), then create another hard save. Exit the game, open it, then check the Papyrus log for errors, see if there's any "critical". Note that the game may reassign dynamic RefIDs later so the fact you can find it later in Fallrim Tools or in the game doesn't mean the reference wasn't removed.

 

 

Note: the risks of removing a persistent reference should be much lower if it has no promoters and no VM variables point to it ( and that editor-set-only persistence is still vulnerable to the invalid address issue ).

 

I tried to clear the refs by removing the attached scripts (only works on newly spawned actors) even if there is no vars or properties that point at self in them, and i disabled the linking statements. After that the refs became non-persistent (deleted from memory after kill/disable/delete() like it should be) but if i let other mods interact with them, they end up persistent again. DPPI comes clean (absolutely nothing), and GetPersistentPromoters() returns "self" so i gave up on trying to find the cause since i don't own the other mod(s) manipulating the refs and can't work with self promoters (i don't know where to start).

[10/17/2023 - 01:36:16AM] Customers in list = 4
[10/17/2023 - 01:36:23AM] Form not peristent
[10/17/2023 - 01:36:23AM] Customers in list = 3
[10/17/2023 - 01:36:24AM] Forms locking this actor = ["0xff3efda4"]
[10/17/2023 - 01:36:25AM] Customers in list = 2
[10/17/2023 - 01:36:26AM] Form not peristent
[10/17/2023 - 01:36:27AM] Customers in list = 1
[10/17/2023 - 01:36:28AM] Forms locking this actor = ["0xff3efda6"]
[10/17/2023 - 01:36:29AM] Customers in list = 0
[10/17/2023 - 01:36:30AM] Forms locking this actor = ["0xff3efd6a"]

[10/15/2023 - 11:55:14PM] Persistence information for  (FF1A8579):
[10/15/2023 - 11:55:14PM]     Object Variables:
[10/15/2023 - 11:55:14PM]     Stack Variables:
[10/15/2023 - 11:55:14PM]     Remote Events:
[10/15/2023 - 11:55:14PM]     Animation Events:
[10/15/2023 - 11:55:14PM]     Combat Events:
[10/15/2023 - 11:55:14PM]     Custom Events:
[10/15/2023 - 11:55:14PM]     Distance Events:
[10/15/2023 - 11:55:14PM]     Inventory Filters:
[10/15/2023 - 11:55:14PM]     LOS Events:
[10/15/2023 - 11:55:14PM]     Menu Events:
[10/15/2023 - 11:55:14PM]     Sleep Events:
[10/15/2023 - 11:55:14PM]     Teleport Events:
[10/15/2023 - 11:55:14PM]     Timers:
[10/15/2023 - 11:55:14PM]     Tracked stat events:
[10/15/2023 - 11:55:14PM]     Wait Events:
[10/15/2023 - 11:55:14PM] VM is thawing...

I quickly made a cleaning quest similar to "SKK Object Counter", that only targets dead, unloaded refs with appropriate VMScript attached and delete them using MarkForForcedDeletion():
Screenshot-2023-10-17-183739.png

Screenshot-2023-10-17-183814.png

 

As you can see it worked, no extra error in papyrus log before or after loading the new save but in fallrim however, all attached script instances are still there:
Screenshot-2023-10-17-194823.png ( i don't know why the numbers are different :happy: )

They are attached to "NONEXISTENT CREATED FORMS" now. Is it safe to clean them ? does it matter ? and more importantly, if the refs were normally removed by the game without use of "MarkForForcedDeletion" will the attached instances get removed from the save file or not ?.

Edit:
If i clean the the unattached script instances in fallrim (only the scripts attached to deleted Refs), it will generate a bunch of errors on first load (nothing after saving and loading again).

  Reveal hidden contents

 

Edited by lee3310
Link to comment
Share on other sites
LarannKiar Mentor

LarannKiar

Posted
Posted (edited)

 

  On 10/17/2023 at 7:16 PM, lee3310 said:

 

I quickly made a cleaning quest similar to "SKK Object Counter", that only targets dead, unloaded refs with appropriate VMScript attached and delete them using MarkForForcedDeletion():

Screenshot-2023-10-17-183739.png

Screenshot-2023-10-17-183814.png

 

As you can see it worked, no extra error in papyrus log before or after loading the new save but in fallrim however, all attached script instances are still there:

 

No Papyrus errors and complete script instance removal are unfortunately not really in harmony.. :smile:

 

MarkForForcedDeletion() flags the references "to be removed" on the highest possible level but it doesn't queue up the reference's VM data. It lets the native code to decides what to delete.. this was the least harmful ( and wouldn't like to call it safe ) method I found to make a reference disappear. I understand the issue though.. it's not good for save games. Supposedly, the engine "should" free up the RefID slot and remove the instance data... eventually.

However, the vanilla "cleanup process" rather keeps references persistent for eternity than remove something that "might be" needed later.
Actually, MarkForForcedDeletion() was meant for those certain object references and inventory items ( the ones that have a persistent reference linked to them, these are typically quest objects but not necessarily ) that the engine fails to remove.
For an all-in-one solution, these functions below may prove much more useful. I'll update the script extender tomorrow with them:
Bool Function RemovePersistentPromoter(ObjectReference akReference, int aiFormID) native global
Function DeleteNoCleanup(ObjectReference akReference) native global
Bool Function SetTemporaryReference(ObjectReference akReference, bool abTemporary) native global
Bool Function SetEditorPersistence(ObjectReference akReference, bool abSetPersistent) native global
Bool Function ForcePersistent(ObjectReference akReference, bool abForcePersist) native global
Function DeleteForm(Form akForm) native global
Function UndeleteForm(Form akForm) native global

With these, it will be possible to hook up the ref to those cleanup events that remove script instances ( and everything the game actually supports ) as well.

 

SetTemporaryReference() sets/unsets the "temporary ref flag" ( Console: [T] ) which determines whether to ignore or include the reference in certain Form cleanup. And, if persistence = 0 and temporary = 1 ( Console would show: 'RefName' (ff0123456) [T] ), OnUnload(), the game traces back the necessary VM data and not just purges the reference itself. I tested them with IsCreated() refs that had both [PP] and [EP] persistence and they work.

Edited by LarannKiar
Link to comment
Share on other sites
lee3310 Collaborator

lee3310

Posted
  • Author
Posted
  On 10/18/2023 at 2:47 AM, LarannKiar said:

 

 

  On 10/17/2023 at 7:16 PM, lee3310 said:

 

I quickly made a cleaning quest similar to "SKK Object Counter", that only targets dead, unloaded refs with appropriate VMScript attached and delete them using MarkForForcedDeletion():

Screenshot-2023-10-17-183739.png

Screenshot-2023-10-17-183814.png

 

As you can see it worked, no extra error in papyrus log before or after loading the new save but in fallrim however, all attached script instances are still there:

 

No Papyrus errors and complete script instance removal are unfortunately not really in harmony.. :smile:

 

MarkForForcedDeletion() flags the references "to be removed" on the highest possible level but it doesn't queue up the reference's VM data. It lets the native code to decides what to delete.. this was the least harmful ( and wouldn't like to call it safe ) method I found to make a reference disappear. I understand the issue though.. it's not good for save games. Supposedly, the engine "should" free up the RefID slot and remove the instance data... eventually.

However, the vanilla "cleanup process" rather keeps references persistent for eternity than remove something that "might be" needed later.
Actually, MarkForForcedDeletion() was meant for those certain object references and inventory items ( the ones that have a persistent reference linked to them, these are typically quest objects but not necessarily ) that the engine fails to remove.
For an all-in-one solution, these functions below may prove much more useful. I'll update the script extender tomorrow with them:
Bool Function RemovePersistentPromoter(ObjectReference akReference, int aiFormID) native global
Function DeleteNoCleanup(ObjectReference akReference) native global
Bool Function SetTemporaryReference(ObjectReference akReference, bool abTemporary) native global
Bool Function SetEditorPersistence(ObjectReference akReference, bool abSetPersistent) native global
Bool Function ForcePersistent(ObjectReference akReference, bool abForcePersist) native global
Function DeleteForm(Form akForm) native global
Function UndeleteForm(Form akForm) native global

With these, it will be possible to hook up the ref to those cleanup events that remove script instances ( and everything the game actually supports ) as well.

 

SetTemporaryReference() sets/unsets the "temporary ref flag" ( Console: [T] ) which determines whether to ignore or include the reference in certain Form cleanup. And, if persistence = 0 and temporary = 1 ( Console would show: 'RefName' (ff0123456) [T] ), OnUnload(), the game traces back the necessary VM data and not just purges the reference itself. I tested them with IsCreated() refs that had both [PP] and [EP] persistence and they work.

 

Wonderful, Fo4 persistence manager is a new territory and we are lucky that you chose to explore it. I will wait for the new functions.

Link to comment
Share on other sites
LarannKiar Mentor

LarannKiar

Posted
Posted

These functions didn't work out well.. (in the original conception, akForm was a base form that, upon deletion, all its object references became deleted as well. However, they weren't reliable).

Function DeleteForm(Form akForm) native global
Function UndeleteForm(Form akForm) native global

I added an optional ignore [PP] hierarchy to SetEditorPersistence() (as normally [PP] would need to be cleared first) and RemoveAllPersistentPromoters() for convenience.

Bool Function SetEditorPersistence(ObjectReference akReference, bool abSetPersistent, bool abIgnoreHierarchy = false) native global
Link to comment
Share on other sites
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...