Jump to content

Photo

NMM (.61.23) is terrible at unpacking highly compressed and sizable archives.

archive unpack

  • Please log in to reply
15 replies to this topic

#1
xrogaan

xrogaan

    Regular

  • Premium Member
  • 55 posts

Hello, I noticed to my dismay that NMM is not relying on third party software to unpack the mod archives. While this may be fine for most of the mods out there, it becomes a nightmare when you try to use it to install mods such as Remodeled Armor for CBBE Bodyslide HDT. I'm using remodeled armors as an example because it present the issue very well.

 

The main archive is quite small, under 200MiB. However, while unpacked the size get up to 800MiB. This leads to a very long period of waiting time (10 to 20 minutes) for a single core process. While if I use a third party tool (like 7z), the unpacking time is about a minute (with an AMD Phenom II X4 965be). This is crazy.

 

I don't exactly know what NMM is doing while unpacking, if it do anything at all. But if it just does that, unpacking, I beg of you, for my sanity and other's, please use 7z if it is installed on the system or implement a multi-core unpacking process.

 

Thanks for your attention.



#2
KunoMochi

KunoMochi

    Unashamed food junky

  • Premium Member
  • 2,031 posts

From what I understand, NMM uses the open source library SevenZipSharp to extract the archives.

https://sevenzipsharp.codeplex.com/

 

However, the devs already know that it takes a while to extract files. I heard that they are hoping to get to overhauling the system some time in the near future, just not a high priority at the moment.



#3
xrogaan

xrogaan

    Regular

  • Premium Member
  • 55 posts

Well, the issue here is mostly the single core lock. If NMM could invoque 7zip to unpack the files, then work on the files, it would be way faster. But of course, you'd need the space to unpack then more space to copy before removing the temporary files.

 

As it is now, I just spent 30 minutes to install a mod just to figure out I installed the wrong version... Which, surprisingly, triggered another hour to uninstall it. - Why does it take more time to uninstall the mod than to install it? - All of that because of this very issue.



#4
DuskDweller

DuskDweller

    Resident poster

  • Premium Member
  • 3,081 posts

There's a known issue with NMM and big compressed archives, it can't be fixed easily (it requires a lot of thinkering with the current source code) so I can't give you any ETA on when we'll be able to address it, but I assure you it's on the todo list.

 

The issue with long uninstall times it's easier to be fixed and we'll take care of it in one of the next patches.



#5
mikeloeven

mikeloeven

    Old hand

  • Supporter
  • PipPipPip
  • 573 posts

I haven't ever experienced something as long as 30 minutes to unpack a mod and some of the mods i use are several gigs in size. Though with my specs it is possible that my system is just brute forcing through the issue.

from experience the process of unpacking archives relies mostly on disk speed and memory which are usually the largest bottlenecks. Could you post your system specs?


Edited by mikeloeven, 21 July 2016 - 12:28 PM.


#6
xrogaan

xrogaan

    Regular

  • Premium Member
  • 55 posts

There's a known issue with NMM and big compressed archives, it can't be fixed easily (it requires a lot of thinkering with the current source code) so I can't give you any ETA on when we'll be able to address it, but I assure you it's on the todo list.

 

The issue with long uninstall times it's easier to be fixed and we'll take care of it in one of the next patches.

I assumed as much, and thus my request to rely on an external, more specialized, software if available. Unless you really for need for NMM to unpack the archives by itself. But hey, what do I know? I'm not a software engineer :wink:

 

7z can be called and made to unpack files silently, but I'm sure you're already aware of that :smile:

 

I haven't ever experienced something as long as 30 minutes to unpack a mod and some of the mods i use are several gigs in size. Though with my specs it is possible that my system is just brute forcing through the issue.

from experience the process of unpacking archives relies mostly on disk speed and memory which are usually the largest bottlenecks. Could you post your system specs?

It's in my profile:

  • Processor
    AMD Phenom II X4 965e @ 3400 MHz
  • Memory
    12288 MBytes DDR3
  • Motherboard
    Asus M4A79XTD EVO
  • Graphics Card:
    Radeon R9 280X
My hard drives are standard mechanical ones (7200rpm). Though your bottleneck theory, with all due respect, smells like *censored*: 7zip do not have to wait that long to process the same archive. It is good to remind you that the archive's size is under 200MiO, and thus isn't too hard to be read. As for the speed of my hard drive, I'll rely on steam telling me that its peak is roughly 12MiB/s (can be wrong, because "steam") which would take about one minute to move 800MiB.
 
I also use 2 hard drives for my mods: one to store the archive and the other where my game is installed and thus contain the virtual thingy. So if a disk is busy reading, the other can totally be writing.
 
The reason 7zip is so efficient is because it support multiple cores which helps with the algorithm (packing and unpacking are just that: math), if I were to limit 7zip to one core, it would be so much slower. And the reason NMM is so slow is because the software isn't taking advantages of all the available cores at its disposal.
 
Thanks for your interest and willingness to help though, I really appreciate it. And yeah, if I had a SSD it would be so much faster but that's not the point.


#7
narphous

narphous

    Old hand

  • Premium Member
  • 803 posts

Add me to the list of finding NMM's install times to be unexpectedly long. I'm running a 4771 @ stock speed, 32 GB ram, and 3 disks, 2 SSDs and 1 spindle drive. The mods live on the spindle drive and skyrim is on one of the SSDs, the OS is on the other SSD along with NMM. I've seen extract times of well over 30 minutes to install a mod of a few hundred meg. I can extract the same archives under 7zip in a minute, or for very large archives, less than 5 minutes.



#8
xrogaan

xrogaan

    Regular

  • Premium Member
  • 55 posts

Yeah, keep in mind that the important part isn't the size of the archive but the size of the uncompressed data. Your archive can be very small but once unpacked can take a lot of space. As I stated, this issue is due to the fact that the algorithms used to unpack the archives can't utilize the full resources of our computer because NMM is bound to one core. It goes faster with 7zip because that software use everything at its disposal.



#9
narphous

narphous

    Old hand

  • Premium Member
  • 803 posts

Yeah, keep in mind that the important part isn't the size of the archive but the size of the uncompressed data. Your archive can be very small but once unpacked can take a lot of space. As I stated, this issue is due to the fact that the algorithms used to unpack the archives can't utilize the full resources of our computer because NMM is bound to one core. It goes faster with 7zip because that software use everything at its disposal.

 

Well, I'm going to blow this theory. I installed a ~7.5 meg mod that unpacked to just over 10 meg last night, and it took just over 15 minutes. NMM just churned on a single core as you say, but not at 100%. Running resource monitor showed the CPU hovering at between 30 and 80 percent for that core, with only a few spikes to 100 percent. Further, NMM spent more time installing the files than extracting them.

If the devs think it would be helpful, I can get a procmon trace against the nmm process and it's children to see if that points them to the place(s) in the code that are likely to be causing this.



#10
xrogaan

xrogaan

    Regular

  • Premium Member
  • 55 posts

Out of curiosity, how many files was in that archive? Could a high amount of files impede on the speed?

 

And yeah, I only have a theory because I don't know how NMM unpack and process the archives. They may process the files on the fly instead of unpacking and then processing.







Also tagged with one or more of these keywords: archive, unpack

Page loaded in: 0.806 seconds