File Loading Research

[Jeoshua]

The following is the culmination of some research I was doing into Fallout 3's poorly understood load ordering process. It will not be very usable for end users. The information below is intended for addon creators and for the purposes of continued research in the peculiar way that the Gamebryo engine goes about everyday business. The creators of FOMM, Wrye, and other similar tools should take special note of the difference between plugins.txt and timestamp ordering.

 

I have saved file access logs to back all of this data up, should they be required, which I will share freely upon request.

 

If you don't understand any of this, skip to the bottom and read the suggestion at the end.

 

File access Logic Flow
----------------------
[init Settings, Sound, Graphics, Disk, Power, etc]

1. Enumerate all esm files in \Fallout 3\Data\
Alphabetical list returned, reguardless of load order
All esp/esm files are enumerated, reguardless of whether they are "on" or "off"
Chunks of files read, offset varies
	Headers?
Files remain open after access

2. Enumerate all esp files in \Fallout 3\Data\
Same comments as above, process seems equivallent

3. All resultant esm/esp files enumerated
Not a file search
	Files are indicated specifically for access, by file date
	Approximates load order shown by Loader/FOMM
All files enumerated reguardless of whether they are "on" or "off"
Files are checked, not opened or closed for access

4. Users\[username]\Appdata\Local\Fallout3\plugins.txt opened
Data read in full, loaded into memory (presumed)
File closed after read complete

5. Files Meshes\Characters\WeaponWobbles\* and Meshes\Characters\CameraShake.nif accessed
Data read in full, loaded into memory (presumed)
Files closed after reads complete

6. ESM/ESP files queried
Files are queried according to plugins.txt
	Files which are "on" are checked in order appearing in plugins.txt
	Files which are "off" are closed at this stage
	Changing plugins.txt order changes order in which these files are loaded (tested/confirmed)

7. ESM/ESP files attempted to be read from Fallout 3\*
Fails gracefully, no files occur in Fallout 3\* of this type
Order is confusing, does not conform to Alphabetics, File Dates, or plugins.txt
	possible override tables (assumed)
This stage overlaps with closing of plugins specified as "off" (rather, not whitelisted in plugins.txt)
System tries MULTIPLE times to access each file in this manner
	Some files more than others (Fallout3.esm)

8. File Sound\fx\ui\loadscreen\initial\ui_loadscreen_initial.wav accessed
Data read in fill, loaded into memory (presumed)
File closed after read

9. Continuation of part 7
Closing of "off" plugins also continues

10. Final closing of all "off" plugins
Happens within microseconds of each other, one after the other
	Closing of these plugins occurs in order of File Dates.

11. ESM/ESP Files allocated into memory
Allocation occurs according to plugin.txt ordering list (confirmed)
Loading more than 2GB of esm/esp data causes overflow (confirmed)
	Overflow can be circumvented by LargeAddressAware flag (confirmed)

12. BSA files accessed
Reads occur in order of plugins.txt ordering list.
All files considered in form Data\[plugin name]*.bsa
	Example: Data\Anchorage*.bsa
		Returns: Data\Anchorage - Main.bsa, Data\Anchorage - Sounds.bsa
	Example: Data\James Invisible Wallbugs Remove V3*.bsa
		Returns: Null
Small portions of BSA files read into memory
	Occurs near beginning of files
	Most likely file tables for BSAs (presumed)

13. ESM/ESP files read in full
Occurs in background
	Load screen and menu file access occur at the same time as files are being loaded
	Fallout.ini options may change this behavior (presumed)
File readings occur in order of plugins.txt ordering list

14. Archive Invalidations applied
BSA files checked for resource files
Files which occur in Data\[Resource Path] accessed (factual)
	File creation dates checked (factual)
	Files are catalogued for future reference (presumed)
Checking of BSA files vs Data files occurs in order of plugins.txt ordering list
Occurs in background
	Further load screen and menu file access occur at the same time as files are being checked

15. Menus loaded
Files for menu graphics loaded
Control hooked to mouse and keyboard
All files in menus\prefabs\* enumerated
Files in menus\prefabs\* read into memory
	Files which are linked in those files are accessed at this stage

[Title Music plays, Slides displayed, Saves enumerated, Controls released to user, Game begins]

?. Game closing
Files closed in order which appears in plugins.txt

Definitions
-----------
File Date refers to time stamp stored in FAT table
Time and Date are considered, and stored as the same variable
FOMM and Wrye both order files with this method
All vanilla BSA files are ordered with the file date x-10-2008 0:0:0 (POSIX timestamp format)
	x is:
		1: Fallout 3 vanilla files
		2: Operation Anchorage files
		3: The Pitt files
		4: Broken Steel files
		5: Point Lookout files
		6: Mothership Zeta files

Conclusions
-----------
Accesses to all data files check root directory, then Fallout 3\[resource path], and finally Fallout 3\Data\[resource path]
For Objects in mods, Data\Meshes\file.nif is therefore functionally equivallent to Meshes\file.nif
For all object informations, a resource path such as Meshes\Resource\ causes fallout to enumerate a list of every file in that directory
Works for sounds, in which the sound object can specify a directory instead of a .wav or .mp3 file
	Resultant sound which is passed back to rendering/calculating stages is chosen at random
Works for music, such as Data\Music\Base\*
	Resultant music passed back to rendering/calculating stages is chosen at random
May work for other types of objects, such as textures or meshes (untested)
	Seems to be transparent to the engine, no extra data specifying a list has been found
There are three "Load Order Sequences" apparent
They are:
	File Name - Occurs in initial enumeration of all files
	File Date - Occurs when loading into memory
	plugins.txt - Used for overrides (tested/partially confirmed)
Plugins.txt is used more often for load ordering purposes than any other type.
Better load ordering can be done by modifying plugins.txt rather than file dates.
FOMM/Wrye only modify file dates.  This is inadequate.
	They also add new plugins to the bottom of plugins.txt instead of ordering said file
	This is presumably the reason that FormIDs do not always match FOMM list, which is based purely on File Date.
Loader modifies plugins.txt without changing file dates.  This is also inadequate.
	Files may be loaded into same memory area if they override or share formids in any way (untested)
Plugins.txt file ordering will effect Archive Invalidation process
Files which occur later in plugins.txt have their data files checked later
Files which are checked later will have their file tables used instead
Plugin files with proper headers and master files will override data no matter where they are in the load order
Files which are later in plugins.txt will take precedence.
	This can lead to problems when multiple files share the same master
	If the plugins.txt file does not match the file date ordering, unexpected results can occur
		This is why one must rebuild their merged patch through Wrye or fo3edit, even if they keep the merged patch up to date manually
	Load ordering tools do not take plugins.txt into account
Files which explicitly specify overrides in their headers take precendence according to the plgins.txt order
	Multiple files overriding the same master will be kept in order according to the plugins.txt file
Tested with and without FOSE
No apparent changes to file accesses or loading orders

Finally:
We are gonna need a bigger boat
FOMM is useless for proper load ordering
Wrye is useless for proper load ordering
Loader is useless for proper load ordering

Best case working scenario I have found:
	Using FOMM to set Load order
	Export Load order
	Edit list with a text editor
		Remove all lines which start with "[ ]"
		Remove "[X] " but keep file name
		Save file as Users\[username]\Appdata\Local\Fallout3\plugins.txt

Edited August 21, 2011 by Jeoshua

[Fonger]

I just use BOSS

[nosisab]

Your research corroborates the advice about avoiding to keep many files under the install folder structure regardless they are marked off (actually is supposed the engine can find the files by it's header too, what makes changing the extension useless or even keeping them outside the /data, this might be something you may want to confirm). It can shed some light about the deadlocks which sometimes occurs when long/complex load order is changed with regard the saved game.

 

Good work, please keep us informed of your progress.

 

@Fonger, you do well, just the OP is about something else, not intended to casual user indeed.

Edited August 22, 2011 by nosisab

[blove]

I see a lot of research there, but are you trying to fix something? If so, what?

[nosisab]

Maybe, at least it may shed lights on the reason why the game hangs on certain savegames load (somewhat common when the load order is changed) and, who knows, help to prevent it.

 

As the OP states, that information is probably more useful to utilities authors; utilities like OBMM, FOMM, Wrye. Depends on how accurate and reliable it proves to be. For one, about the engine enumerating all ESP/ESM it finds in the directory structure, activated or not, is an answer to that issue reported recently about the game malfunctioning under variable number of "active" mods far below the expected limit. Although known from long date the reason was not thoroughly verified until now.

 

To the end user may be enough to understand that knowing how the game "indeed" load the mods helps a lot, since the research seen to point the engine uses the plugin.txt on certain steps and the timestamp in others, which may lead to the dreaded deadlocks and possibly causes of overlapping mods misbehavior when seemingly should have none.

 

The bottom line is all this is meaningless to the end user and using whatever she/he has been using to correct the load order is still, and by far, better than having a totally messed load order.

[Jeoshua]

There is only an issue with the current methods of load ordering when it comes to situations where multiple files overwrite the same file. They will be loaded in the order in which they were turned on, instead of by file date. So basically FOMM and Wrye and BOSS don't give you the full story.

 

In 9 times out of 10, this won't be an issue for most people, as the differences will be minimal. In that 1 time out of 10, however, it could mean the difference between a crash and a smooth game. Rebuilding a merged patch involves deleting it, which deselects it. Then, once you rebuild the merged patch, it is at the bottom of the list. Which will solve many problems.

 

I became curious about why, in the event that I maintained my merged patch by hand meticulously, I would often come across the situation where my game didn't work correctly until I had actually deleted the file and made a new one. This research indicated why.