Rating system overhaul

[Dracomies]

I think what would be a good rating system is if the rating was

 

1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5, 8, 8.5, 9, 9.5, 10

 

That would really give a more accurate count. People wouldn't throw 1's and 10's around anymore.

 

The problem with the "endorse this" and "endorse that"..is that it sort of is like how things are at the movies. You can either give it a thumbs-up or a thumbs-down. But it doesn't really tell you much about the mod...and what will end up happening is that there will be a lot of "B-" and "B" mods that make it in with the crowd of "A's". It's just not specific enough...

 

I think the rating system I recommended would be really specific...because a rating of...let's say..an 8.5/10 tells more than a "I endorse this" mod.

 

Because people won't "bad-mouth" a "good" mod. But sometimes people don't just want "good". Sometimes people want the best.

 

What I don't want happening is a lot of above mediocre mods just getting a thumbs-up. I like getting the absolute best of each category..and I think a rating system is still important for that function.

 

I like the can't rate within 3 hours rule, and the "can't rate unless it is downloaded.

Edited July 9, 2009 by Dracomies

[AncientSpaceAeon]

While I don't really like this rating system before, but I think this is really helpful. While I can't search the unpopular mods easily, but I think the percentage rating system is the best (one unique user download count).

 

Or maybe this type of rating system :

- Perfect

- Almost Perfect

- Great

- Less

 

And @ Dracomies, this rating system is to blocked more vote abuse. (Moderator's job isn't that easy you know.)

[MesseangerOfDeath]

sound verry weired but i dont upload my mods here anyway due to admins banning people so much. i think if a new commenting system is made then we also need to change the baning system so banning is not permanit. we loose so many good modders over being baned forever. thats a wicked thing to do to people.

[lordbelfast]

Because 5 minutes is not enough to test a mod...

[avulsion]

A ratings system serves two purposes (three if you count ego boosting): It allows consumers to differentiate between good and bad products, and it provides the manufacturer with feedback as to the strengths and weaknesses of their product so that they may improve on it. Up until now I have relied on the download stats to browse for files and largely ignored the ratings system due to the horribly skewed distribution of scores. Upon seeing the new system, I went and voted on a few of the things I've downloaded recently. Thumbs up works fine, but when I tried to vote thumbs down for one file I was greeted with this:

 

I am not currently endorsing this file because...

 

I couldn't get the file to work

the file didn't do what the description implied

there were important files missing (animations, textures, etc.) that meant the file didn't work properly

the file caused conflicts with the game or other popular files

the file did not meet my high quality standards

after playing the file it doesn't fit in to my personal tastes or beliefs, etc.

 

The thumbs down option needs a comment box, or a way to connect a negative vote with a forums post (or just flag a forums post as voting positive/negative) for easy assessment. For a small file that changes one or two things, it's fine to say "It doesn't work," but for a large and complicated mod, such a comment is mostly useless. What doesn't work, is it the dialogue, graphics, sound effects, quest triggers, loot drops, spawn timers, all of the above, or does it just CTD immediately? If there is no technical flaw, but the story/art/balance didn't appeal to me or were obviously unfinished, it might be helpful to the author if I could say so. A handful of complaints in the thread might get drowned in a flood of praise, but an author might find it useful to see who isn't happy and why not. Being able to turn on a filter and easily see the pattern of several people saying "Feature X is f**king annoying, please change it and I will gladly reinstall and change my vote" would be invaluable.

 

The presence of a comment box for positive votes is of limited utility, a voter might specify why they liked the mod, and what improvements would make them love it, but for the most part positive comments will consist of short, polite and meaningless messages like thanks, loved it, awesome, great, voted 10, A++++ would buy from this seller again (you get the idea). The positive comments are mostly for ego stroking, Negative comments encourage change. If someone decides to abuse the negative comments box and type something like "this sucks you suck" or "lol u r gay" then ban them and delete their vote. There's nothing to stop them from posting such things in the forum thread, or PMing them to the author anyways. Since neg votes don't actually affect the rating system, people should be encouraged to offer constructive criticism (emphasis on constructive and criticism, if you can't be constructive, don't post, if you can't take criticism, don't upload your work to the internet and ask other people to judge it).

 

 

 

On a slightly more positive note, I think the timers a good idea and will probably help to limit abuse. However, keep in mind that there is no sure fire way to keep people being from being jerks on the internet, so you should take care to make sure that security measures do not overly hinder legitimate users. Allow the authors to determine how long people should wait before being able to vote on their files. If someone uploads a small stat change or retexure, they can set the delay to 0. Someone who spends weeks building a massive new quest chain should be

allowed to require that users spend a few hours testing it out before passing judgment. Additionally, it is worth noting that most of the options currently presented for a negative vote should be instantly obvious, and only one or two should take more than a few hours to notice.

 

A voting reminder is a must have, and so is an update alert system. If I vote up a mod, I'd like to be automatically notified when the new and improved version is released. On the flip side, when I vote down a mod, give me checkbox options like "Technical Issue - Graphics," "Technical Issue - Gameplay," "Opinion - Too Easy." Then when the author updates the file, give them a form with options like "Fixed - Graphics Issues" "Fixed - Dialogue Issues" "Changed - Difficulty" and some server app compares changes to criticisms and makes a binging noise when a certain percentage match. I get a PM telling me to download the new version and see if I want to change my vote, and the author gets a chance to turn red numbers into green numbers.

 

A possible solution to the "old files always have more votes and therefore higher visibility which means people will download them and vote for them more" dilemma would be to have votes expire, either according to a set date (which might cause a sudden drop X months after a mod's release) or some kind of half-life formula, i.e. all files lose half their rating every 28 days, or 2.5% calculated daily. The half-life method would cause mods that get sudden spikes of votes to decline faster than files with a smaller but constant influx of votes, and should discourage most vote rigging shenanigans. At 2.5% a day, a file that gets 200 votes in the first week and 2 votes a week for every week after that would stabilize somewhere between 11-12, the file that gets 20 votes on week one and 4 votes for every week after would stabilize at around 22-23. My math may be somewhat off as it is currently 5AM and this took a lot longer to type out than I had anticipated.

[gsmith2885]

Im not sure that I like the new rating system yet. I realise it hasnt been up long and I plan on giving it more time before I truly deside. I do belive there is one area that can be improved though. In the top files section add an all-time top files as well as the 2-week top files section. I play mostly on my laptop so I dont download some of the more graphics intisive files. Though I also play on my desktop at home when Im there. This addition would make it easier for me to find a file I was looking at on the all-time top files as they most likly wont change alot. Thanks for the time and for the great site. Keep up the great work.

[HugePinball]

Are comments not required, or not added to the thread when giving a thumbs down?

 

I ask because one just popped up on FOOK and when looking for a reason, I just see the canned response for thumbs down feedback.

 

 

OK - I just read the explanation on the page you get when you "non-endorse" a file, and I understand what's going on. The only minor drawback I can see at the moment, is that there is no clear way to directly respond to negative feedback concerns, and for the feedback to really be constructive, I think we'd need some more detail than the simple reasons to choose from (although it is very nice to have them categorized).

 

For example, if "the file caused conflicts with the game or other popular files" reason is chosen, I'd want to know what those conflicts were, and address whether they were actually caused by the mod, already described in readmes, or if they were even real conflicts, and if so, we need specifics if the issues can be corrected. Without a required comment post, I think it's much more likely users will simply never provide details.

 

I realize we could directly contact the user leaving feedback, but that may not be the best solution either (there is often no response to pm's, and in many cases if a user initially describes his problems or concerns with their negative feedback, it is immediately clear without discussion that the fault is not with the mod). Then also, without a comment post added, there is no way to report a malicious or incorrect down-rating if appropriate.

 

I realize you may still be making changes in these and other regards - just pointing this out in case it wasn't being addressed. Otherwise, the new system, new pages (like download/rating history), etc. still look great!

 

 

EDIT: I just fully read through avulsion's previous post. It looks like he already expressed this same concern :)

I also echo all his other points, especially about an update alert system for mods you have previously rated, perhaps on the new download/rating history page so it's not obtrusive. I think many users don't even realize you can re-rate files.

Edited July 9, 2009 by HugePinball

[HugePinball]

I think what would be a good rating system is if the rating was

 

1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5, 8, 8.5, 9, 9.5, 10

 

That would really give a more accurate count. People wouldn't throw 1's and 10's around anymore.

Wow, are you serious? You realize a 1-10 system wasn't working, right? How would a 1-19 system be better? That or the old system would certainly be better if there was a way to ensure that even half the users would use it properly, but I think it's been abundantly demonstrated that's next to impossible. Dark0ne's explained that he'd prefer that type of specific rating system but in practice it's just not used as intended, which is why he implemented changes. I think the new, simpler system, while theoretically less informative, performs much better in reality.

 

Although I'd again like to suggest an additional and optional category rating system available to users after endorsement, even if it's as simple as another thumbs up/down (or no rating), for categories like Concept, Function, Stability, Usefulness, Artwork/Content, and any other major areas, subjective or not. These additional ratings do not need to factor into any rankings anywhere, just viewable on each mod page to provide more information to the user.

[HugePinball]

We still use the thumbs up system but at the download page, but not only it displayed thumbs up and thumbs down. But there's an additional percentage score. Example, if a mod downloaded 100, and got 10 thumbs up, means got 10% score. So the mod with highest thumbs up maybe the most popular, but the one with the highest percentage score is the best.

 

That would be a terribly easy system to abuse; just keep clicking away at the download for a file and watch the percentage score drop. Counting unique downloads (i.e. one download per unique user) was something I contemplated a year back but it isn't feasible; there's over 75 million rows in the download stats table and having to query that table every time someone downloads a file would bring the site to a complete overloaded stand-still.

I was considering suggesting something like that to help address the concerns of some like Aeon and silka, but I hadn't thought about abuse either. What if the ratio was based on # endorsements to file age instead of # of downloads? I realize that would likely be skewed toward more popular mods as well. Maybe some combination of all three of those values could produce a somewhat balanced "rating". And if such a calculated rating value seemed too judgmental to slap right on a mod page, maybe it could just be used for a separate ranking list and/or a search criteria. And call it something neutral like "Weighted Endorsement Ratio" or something. (Do you really think the # of downloads would be abused with such a system? - I have no experience on which to judge.)

 

I think what some are looking for is just a way for quality but less popular mods to be able to stick out alongside wildly popular mods. The ratio value I'm proposing doesn't have to be qualitative, but just provide another criteria for mods to be found other than popularity - then users can read the the comments on the mod page to get a subjective view of quality.

[Alucard1475]

This is a nice change.

 

"...and ratings of 7 or more on files have been changed in to endorsements..."

 

What they're pretty much telling us is to endorse a file if we like it above 70%.