My suggestion for a different ratings system

[SnakeChomp]

Hi

 

Introduction

I have personally never rated a mod here on this site (or any site). I have, however, witnessed the unfortunate situation unfolding with regards to trolls, modders pulling their content and the new commenting policy. There is a bit (okay, a lot) of talk on the bethsoft forums about the rating system: claims that it is broken, complaints about the top 25 picture rating system, and etc, but no proposals (that I have seen, I admit to skipping a page or two in the thread) on a new system.

 

My opinion is that the rating system that is in place does not work. Ratings are being used only to give a modder "kudos" for what they have made. The majority of users will give ratings of 10 to mods they like as their way of saying "thank you" to the author of the mod in question. There are also users who try to actually give a meaningful rating (something not a 10). Unfortunately, there are also some users who come here for the express purpose of giving a modder "the finger" with a rating of 1 (or something else quite poor) and an insulting comment.

 

In a "open forum", such as this site, it is not possible to keep the trolls and flamers away. It is, however, possible to deny them the opportunity to flame. It would be impractical to disable commenting in general (as in, there are no comments on any mod at all), so that is not something which can be done. (I am fine with allowing authors to disable comments on their mods.) What can be done is to deny flamers and trolls the chance to give a bad rating.

 

To support my argument that the current ratings system is broken, I present this quote that I read today on the bethsoft forums.

 

(Kivan on the Bethsoft forums @ Jun 3 2008, 10:50 PM) said:

 

All I can ask: if you like the UOP (and it sounds like a few do, and it keeps getting called "essential") please rate it appropriately especially at Nexus. I put five months of 12-14 hour days on just this update, so this is my only requested payment; it only takes a few seconds and makes the work worthwhile.

 

Right while I was uploading it, someone rated it a "9" and that kicked it down two spots. Rather anticlimatic but typical for me. It was at # 3 for a long time, but no one's rating it even though just the most recent was downloaded 90,000+ times just from Nexus.

 

Thanks!

 

That was my one and only ratings beg ever I hope.

 

I have added emphasis to this quote to support my argument that the ratings system does not work. I did not pick this quote to say anything negative about Kivan, so please do not read "between the lines" to come to such a conclusion. I am quoting this because I found it to be very compelling evidence for why the ratings system does not work. In the emphasized portion of this quote, I personally take "rate appropriately" to mean "give it a 10". I am not saying that this mod does not deserve a 10, because it is an amazing mod. This quote demonstrates what the ratings system is really used for: to say thank you to Kivan (and everyone else) that helped with the creation of the UOP. The system that is in place right was not designed as a way to allow users to say "thanks" to the authors for their mods; that is why I claim the ratings system is broken and does not work.

 

My Suggested Replacement System

My suggestion is to remove the 1-10 rating system entirely, and replace it with a "kudos" system. This system will allow users of the site to give a mod a "thumbs up" type of vote, or "kudos", as a way to thank the author for the mod, demonstrate that they liked the mod, and etc. There is no equivalent "thumbs down" vote, only a "thumbs up" vote. This system is simple, providing the functionality that users want (to be able to thank authors for their mods) without providing an avenue for trolls and flamers to cause trouble. Votes are only that: a vote. There is no comment associated with them, nor is there a comment added. I originally decided against comments as that gave a chance for people to post flames in their kudos vote, but I've thought about it more and believe the ability to comment when making your kudos vote should still be allowed, in order to give a more personalized "thank you", but not a requirement to making a kudos vote.

 

I also believe this will entice more people (like me) to "rate" mods. There is nothing to think about when giving a mod a kudos vote: if you liked the mod and want to let the author know, you simply click the button (or link, etc) to give the mod kudos.

 

What about the top 100 file list, top 25 picture list, and etc?

To allow top 100 file lists and top 25 image lists to continue to be generated, I recommend calculating a popularity rating for each mod. (Note that my vision of a top 100 list is that well established mods will remain on this list and that new mods would take a while to obtain a slot.) This popularity rating should be calculated as a function of: the ratio of the number of views to the number of downloads (such that 1 means every person who viewed the mod downloaded it) and the number of kudos votes it has received. At this time, I do not have a specific formula to recommend, but if this suggestion gains any traction, I would love to discuss possibilities. That formula wouldn't work for images, as they have no download count. For images, I would suggest using a ratio of views to kudos votes.

 

The top 100 and top 25 lists would be generated as the 100 mods and the 25 images with the highest popularity rating. This popularity rating can also be displayed on the mod's page in place of its average rating.

 

How about some newer lists?

Some people may not like having a relatively static top 100 files and top 25 pictures list, so it would be nice to see new lists with different formulas which can be more dynamic. For example, an images list that factors in the number of views per day and kudos votes per day into the popularity rating. That way, newer images which are getting a ton of views and a ton of votes will be at the top of this list, only to be replaced shortly by the next new image that people love.

 

Technicalities

One technicality with rolling out a new system in place of an old system is what to do with existing mods. My opinion here is that every 10 rating should be converted into a kudos vote. No other ratings would have an effect on the kudos vote, and they would just be deleted.

 

Random closing comments

I am truly impressed at the activity of this community such a long time after this game was released. New mods every day, tons of posts in the forums every day, new innovations still being made (I love the possibilities that OBSE brings to the table). I am certainly a nobody in this community, but I am none the less disappointed at the recent friction. I am pleased to see the major players on TESNexus taking initiative to help prevent these problems from occurring again (modders pulling content), but I believe that a better solution to this problem is to to treat the cause of the problems instead of just the symptoms. The ratings system is not serving its intended purpose, and is now providing an avenue for trolls and flamers to hurt this community; I believe it is time it was replaced with something designed specifically for the way users are using this site.

 

Thanks for taking the time to read this.

~Robin

 

(edited once to change my thoughts towards adding a comment along with the kudos vote)

[VilCoyote]

This could work and even be crossed with the top 100 by creating a formula based on the:

k: Kudo received fore the mod

d: number of download of the mod

m: minimum number of downloads needed to enter the top 100 (10k downloads or so)

C: average of the number of kudo per mod on TESNexus

 

for instance, if C = 50, M = 10 000 and each rate = 1 kudo, with the Formula

((d ÷ (d+m)) × k + (m ÷ (d+m)) × C)/100

 

Midas Magic Spells of Aurum: 6.512

Deadly Reflex: 5.686

Unofficial Oblivion Patch: 2.414

New Frostmourne Sword: Not in the top 100 (need less than 320 new downloads) to enter

Lost Paladins of the Divines: 2.692

Adonnays Elven Weaponry: 2.567

The Lost Spire: 14.68 (mostly due because the downloads are also available from the official website)

 

Of course this is just a test formula and represent how well the mod is doing in proportion of kudos/downloads compare to the other mods of the website. The fact that the kudos option is much more simpler than a rating would allows the mods to get more kudos allowing the scores to go higher.

[Lisnpuppy]

I really dont know how all this crazy computer math and stuff works...

but maybe a rating system that only enables if you actually DOWNLOAD the mod?

That doesn't say it will be used...but if you can not rate it a 10 or a 1 unless you click, click on the download...that would seem to make things a bit more fair.

Dont understand someone rating without downloading. I never do.

[LHammonds]

I believe there was a proposal on the ESF where existing votes could still generate decent top lists by simply adding the votes and weighting them based on the vote number as well. So even troll votes would help the popularity of the mod. But even then, it is not exactly fair because older mods have a huge advantage especially if was in a period where the community was at its largest point with the most active members.

 

FYI - Basing it on the number of downloads or number of views is flawed.

 

Why?

• One mod can have multiple files associated with it. All of them help increase the total DL count.
  
• One file can be updated on a normal basis. Which is better...a very good mod that is released with little to no bugs or one that was released and had 20 to 30 updates to fix all the bugs and thus cause the total download count to inflate many more times than the number of people actually download it?
  
• TESNexus has had its fair share of attack and some of the evidence is ether astronomical file/page views which would also skew results.
  

NOTE: By just recording unique IPs is not enough because a targeted trojan/virus can be distributed to cause infected machines to artificially inflate specific / random mods. We had one clever fella use an exploit where every time his page/account/website was viewed, it caused a particular counter to be artificially increased.

 

Dark0ne has already confirmed that the current rating system was abused too much and it is now no longer relevant and needs attention. The problem is that there are many other tasks on his to-do list and this was not designated as one of the highest priority items based on member feedback and his own judgment.

[Ismelda]

Whatever you do it is ok with me :)

 

I hope Dark0ne will fix all the other issues on hos todo list. The popularity contest is not a big deal anyway and thanks to the correct formula the mods with the highest number of good rate are at the top. It feels less artificial than before. A bad rating is much more punishing on low number of rate mods which is good for mod recycling in the top 100 I guess.

[NaaluNaiglemez]

Why?

• One mod can have multiple files associated with it. All of them help increase the total DL count.
  
• One file can be updated on a normal basis. Which is better...a very good mod that is released with little to no bugs or one that was released and had 20 to 30 updates to fix all the bugs and thus cause the total download count to inflate many more times than the number of people actually download it?
  
• TESNexus has had its fair share of attack and some of the evidence is ether astronomical file/page views which would also skew results.
  

NOTE: By just recording unique IPs is not enough because a targeted trojan/virus can be distributed to cause infected machines to artificially inflate specific / random mods. We had one clever fella use an exploit where every time his page/account/website was viewed, it caused a particular counter to be artificially increased.

 

My idea:

• 1. The members declare the mods which they use in " My controls ".
  2. This information would disappear after three months, every member must update it (or prolong it).
  3. On the site, the statistical calculation would be made every 24 hours.
  

The posted statistics would have the quality of a dynamic and serious classification, and updated permanently.

[SnakeChomp]

FYI - Basing it on the number of downloads or number of views is flawed.

 

Why?

• One mod can have multiple files associated with it. All of them help increase the total DL count.
  
• One file can be updated on a normal basis. Which is better...a very good mod that is released with little to no bugs or one that was released and had 20 to 30 updates to fix all the bugs and thus cause the total download count to inflate many more times than the number of people actually download it?
  
• TESNexus has had its fair share of attack and some of the evidence is ether astronomical file/page views which would also skew results.
  

NOTE: By just recording unique IPs is not enough because a targeted trojan/virus can be distributed to cause infected machines to artificially inflate specific / random mods. We had one clever fella use an exploit where every time his page/account/website was viewed, it caused a particular counter to be artificially increased.

 

These are all fair points. At least from my experience on this website, I have found mods with several files each with tons of downloads to be uncommon, and that mods which release several interim bug fixes are a) not very popular in the first place (with some exceptions like midas magic) and b) receive higher download counts only on major updates which increase the version number (1.1 -> 1.2) as opposed to appending a letter (1.1->1.1b) as some mods do for their bug fix releases (again, midas magic seems to do this and has fewer downloads for bug fixes or patches and more for major releases). I do not believe it to be a good idea to ignore the number of downloads in calculating a mods popularity: see the UOP, 150000 downloads in less than a week, and although it has less votes than midas magic, it is clearly the #1 mod on this site (my opinion, of course).

 

It may be necessary to exclude the number of views or a downloads-per-view ratio, given your point #3. Including the ratio would allow exploiters to deflate a mod by inflating its views but not its downloads. Perhaps a kudos-vote-per-download ratio would be better, so long as it is additive with the download count, so that a flood of unregistered users who don't want to register just to add a kudos vote will not detract from a mods popularity.

 

In response to point #1, I have come across mods which list only one visible file and that the file has less downloads than the total download count for the mod. So, some authors choose to remove old versions, while some choose to keep them available. I still do not believe this is reason to exclude download count from mod popularity. Clearly, a mod that is still being updated and still being downloaded is still popular. Any lesser known mods which do this but do not have high popularity will not benefit unfairly from this facility compared to the big mods on this site, which have more downloads than the niche mods ever will anyhow.

 

For my own amusement I'll throw a formula out there and see how it does:

 

k = number of kudos votes

d = number of downloads

p = popularity rating

 

p = sqrt(d) + k

 

Notes: there doesn't need to be a minimum download count or kudos count because of the already established nature of the mods that will end up on the top 100 list with this formula (midas magic, deadly reflex, uop, female body mods (sigh), but not the frostmourne sword ;) ).

 

Lets see how some existing mods on the top 100 list fair. Note that I cannot see how many of the ratings are a 10 easily, so I will instead use the actual rating count instead of the number of 10 ratings like I suggested previously.

 

Midas Magic:

k = 696

d = 169703

p = 411.95 + 696 = 1108

 

UOP 3:

k = 265

d = 142705

p = 377.76 + 265 = 643

 

Deadly reflex:

k = 597

d = 256318

p = 506 + 597 = 1103

 

Frostmourne sword:

k = 225

d = 10236

p = 101 + 225 = 326

 

Book Jackets standard res:

k = 338

d = 77841

p = 279 + 338 = 617

 

Low poly grass:

k = 91

d = 106247

p = 325.95 + 91 = 417

 

Akatosh mount R19:

k = 240

d = 55187

p = 234.9 + 240 = 475

This is one case where not creating new mods for new versions would have helped with popularity (by keeping the ratings all in one place); author's mistake in my opinion. Lets consider R18 and R19 in one mod:

k = 240 + 44

d = 55187 + 14370

p = 234.9 + 119.8 + 240 + 44 = 638

Much better.

 

And lastly I will consider HG EyeCandy Body:

k = 201

d = 101900

p = 319 + 201 = 520

 

I would like to point out that deadly reflex is also one of those mods that frequently releases new versions, "inflating" the download count accordingly. However, with kudos votes being weighted exponentially higher than download count, its still behind midas magic, which wins due to its vote count. I'm sure that UOP will be receiving more ratings over the coming weeks and will eventually reach the #1 slot I feel it deserves using this basic formula.

 

I'm not about to suggest that this formula is the perfect replacement, but only an example of one that leaves the top mods where they are and gets rid of the ones that have no right to compete (sorry frostmourne sword). What this formula does do, is remove the egotistical nature of the top 100 list and replaces it with a historical reference of the mods people have been downloading. The mods that make it to the top will always be there, no matter how many more frostmourne sword variants come to pass. One caveat to this is that, as the game ages and the user base declines, no new mod will ever breach the top 100 list. However, I do not feel it is the job of the top 100 list on this site to be all things to all people; there will always be compilation threads listing all the great mods and any great newcomers as well to serve this purpose.

 

Dark0ne has already confirmed that the current rating system was abused too much and it is now no longer relevant and needs attention. The problem is that there are many other tasks on his to-do list and this was not designated as one of the highest priority items based on member feedback and his own judgment.

 

I must respectfully disagree with the priority assignment, but I'm sure that's not really a surprise. ;) While I would not expect what I am about to offer to be accepted (being a programmer myself, I understand the "pure" feeling of code that is all your own), but I would be willing to volunteer my services if it could help shorten said todo list. I used to be a php tinkerer myself on my own website, and being a recent university graduate on break before going back for masters, I have a lot of time on my hands during the summer months.

 

Lastly, none of the discussion above would apply to the top 25 pictures list, because it makes no sense to exclude the number of views for the top 25 pictures list in my opinion, as all you do with pictures is view them.

 

[edited for minor typographical msitakes]

[Vagrant0]

Seems a bit too complicated. Why not just a simple thumbs up, thumbs down count? And to prevent trolls, make it so that people need to be logged in to vote, and they will have had to have downloaded that file with that account more than 10 minutes ago (to prevent abuse and to encourage people to play the mod before voting). The thumbs up/down count could then be weighted seperately from eachother so that good mods aren't voted bad because someone wants to hurt their standing, and bad mods can't be voted good to make them seem better. Yes, I realize that this essentially removes almost all impact of negative votes. But isn't that the point. It will still count the number of people who voted bad, and make it visible for everyone, but only the good votes would count toward its ranking. Also in this way, if someone uploaded a broken mod which they recieved many bad votes for, it wouldn't disourage them from updating the mod just because they recieved 50 negative votes, since only the positive votes would affect their actual rating.

 

But more importantly, it would be seperate from the kudos system, and be tracked on a mod by mod basis. And since you would be limited to one vote per mod, per account, and be required to download and wait before voting, and since negative votes wouldn't count against mods, but just provide an indication that there were people unhappy with the mod, most of the trolling and abuse would go down. Sure, there may still be people trying to inflate their vote count, but that would probably be easy to track since it is harder to convince others to give you a good rating than it is to convince people to vote poorly on everyone else, and still takes more time than it is probably worth.

 

Essentially, it would be a 0-1 system. People could either choose to say that the mod is good, or to say that the mod is bad, removing the issues which define a 4 from a 6 and the like, and also making more popular mods more likely to rate higher (since there would be more people available to vote), while not giving them different weight because of those downloads.