Benchmarking Skyrim with the ScenicCarriages mod

[kgrins]

Since Skyrim has no built-in benchmarks, reviews on hardware sites always leave me thinking ‘what exactly did they benchmark and why is their results different from X’. Most of the sites seem to do the Whiterun ‘run’: that is run from the gates past the square up to Dragonsreach and look back. Not very scientific.

 

MSI Afterburner (Fraps does too, I believe) has an option to log framerates to a .CSV file. In the past I used that and started a new game to produce a graph of the FPS during the intro. And that works fine, but now there’s another option which isn’t just limited to benching the intro: Sollar’s ScenicCarriages.

 

So armed with Sollar’s mod and my copy of Skyrim (1.5.26.0) with no mods expect for the official high-res TexturePacks and ScenicCarriages, I chose a route (Riften to Whiterun) and logged my framerate to a file. (After selecting the destination from the driver, I do not move the mouse at all.)

 

ScenicCarriages has a horse speed setting and my first runs were at with ‘Horse Speed = Run’:

UPDATE: this is actually Jog (that is 2nd fastest)

http://s18.postimage.org/3p9b6dwl5/At_run_speed.jpg

Min 14.6 12.4 22.8 10.4

Max 60.2 61.1 61.1 62.1

Average 43.3 43.9 43.4 43.4

Journey Time mm:ss 14:28 14:28 14:28 14:28

 

The graphs were occasionally a bit random in that the spikes were not at the same journey time, but the average were very consistent between runs (oh, the max is not that useful since I had vsync turned on). In fact, the averages were so consistent that for the other horse speeds I didn’t do four runs.

 

At ‘Horse Speed = Walk’:

UPDATE: this is actually Run (that is the fastest)

http://s7.postimage.org/8a101pvq3/at_walk_speed.jpg

Run 1 Run 2

Min 18.87 18.85

Max 64.10 53.25

Average 43.34 34.92

Journey Time mm:ss 11:05 11:05

Those two runs were the most different of all my benches. Not sure why run2 had such a slower average. Also notice that the journey time is just over eleven minutes which is faster than ‘run’, that was consistent: seems the horse is faster walking than running.

 

At ‘Horse Speed = Fast Walk’:

UPDATE: this is actually Walk (that is the slowest speed)

http://s16.postimage.org/nv94u4nth/at_fast_walk_speed.jpg

Min 12.47

Max 62.13

Average 46.32

Journey Time mm:ss 37:27

And finally at ‘Horse Speed = Jog’:

UPDATE: this is actually Fast Walk (that is 2nd slowest)

http://s15.postimage.org/8bfj8zxbf/at_Jog_speed.jpg

Min 11.83

Max 60.16

Average 44.82

Journey Time mm:ss 26:50

 

Aside from the second run @ ‘walk’, everything is pretty consistent, therefore ScenicCarriages makes a good and useful benchmarking tool.

 

And comparing ‘run’ and ‘fast walk’ might answer a question Sollar had when I first mentioned that his mod might make a good benchmarking tool: does the ‘run’ impact performance compared to the slower speeds? Well, on my system during my benches ‘fast walk’ gave me 6.5% more FPS than ‘run’ but that’s not a lot. I guess further testing with different CPUs and GPUs could determine where this difference lies: it could be the carriages scripts but equally it could be lots of other things.

 

My hardware specs are: Intel Pentium E5300 overclocked to 3.46GHz, Gigabyte G41M-Combo, 8GB DDR3, Radeon 5770 (default clocks: 850/1200), Samsung SpinPoint F3 HD103SJ 1TB, Win7-64. Settings wise, I am running at 1600x1200 with AA=off, AF=8, texture quality=high, radial blur=high, Shadow detail=Ultra, Decal quantity=Ultra, Water=all checked, and View Distance=all maxed. Basically the ‘Ultra’ preset with less AA and AF. And in MSI Afterburner I have the ‘Hardware polling period’ set to 7000 milliseconds (7 secs) which is the default I think.

 

I know Skyrim is not as hardware intensive as Oblivion was when it came out (I'm using a £70 video card and it runs ok) so getting accurate benchmarks may not be as important but I still find this useful. Oh, and ScenicCarriages is by far the best carriage mod.

 

Another method for benching with ScenicCarriages if you don't want to watch such long journeys is to ask the driver to get off at some stage (on my Riften to Winterhold journey maybe just before the final bridge), save the game and then bench from when you ask the driver to continue. That way you could quickly bench the approaches to several towns etc.

 

UPDATE: Seems the version of ScenicCarriages I tested had used the wrong settings for the speeds so that 'walk' was actually 'run', 'run' was jog, 'jog' was 'fast walk', and 'fast walk' was 'walk'. I've bolded a note about that above.

Edited May 22, 2012 by kgrins

[Sollar]

Thanks Kgrins! Great information, and it does answer my question about the resources... yes, the faster speed does put more strain on the hardware.

 

 

As far as the speed is concerned, I think the confusion is on how the Config Menu works. When you are on Walk Speed, the config screen shows "Set to: Fast Walk". The default setting is Jog, so the Config Menu shows "Set to Run" as button text. (Sorry it's not so intuitive, I didn't like it either, but the way the menu and buttons work in Skyrim/CK is not the most desirable. I'll look at changing it with a sub-menu.)

 

This is the order the speed settings:

Walk - 11:05

Fast Walk - 37:27

Jog - 26:05

Run - 14:28

 

It looks like they are off by one... so that means "Run" is really using Jog, Jog is using Fast Walk, Fast Walk is using Walk and Walk is using Run. With the correct labels, it should be:

 

Walk - 37:27

Fast Walk - 26:05

Jog - 14:28

Run - 11:05

Edited May 22, 2012 by Sollar

[kgrins]

Ah, that makes sense: so the 'walk' was really run. Which might also explain why the variance in average FPS was highest there. Certainly the fastest speed should be the one where the engine has the most problems keeping up. I was under the impression that Skyrim was poorly threaded (the initial benchmarks where they compared dual vs quad core) but with me only having an overclocked dual core it's possible that when moving fast, the engine does start a thread to load the next cell but gets bogged down. Or maybe it just has to load textures and stuff (I only have 1GB).

 

Have to wait for someone who has a faster CPU and more vRAM to bench a journey I guess. No boots of blinding speed but there is player.setav speedmult 500, that would slow down the game a bit...