Jump to content

Photo

On the internet, trolls, and finally creating an effective rating syst


  • Please log in to reply
186 replies to this topic

#1
Dark0ne

Dark0ne

    Webmaster

  • Admin
  • 21,419 posts
This is a long one. Kudos to anyone who reads it all.

One of the largest problems with having such a large and ever growing community is moderating it to a standard that you are happy with. This problem is exacerbated further when the standards you set are perhaps more high-brow than the norm within the larger gaming community. Far from being elitist in essence, the standards are set not to shun users away but to simply draw in more mature, relevant and constructive discussion without the ever niggling sway of that high percentage of internet discussion that degenerates in to some poxy flamefest. Whether it works or not, I guess, is debateable.



Trolls


The problem, as I see it, is the anonymity of the internet. A generation ago in order for trolls, half-pints and pipe-weeds to get their rocks off they had to write in to local or national publications with their crap-stirring or discuss it face-to-face, normally in a pub. As a result there were fewer trolls and the ones who did do it tended to be older because of the effort and perhaps finance needed. In comparison much of the flaming you see in the gaming community these days would have left the old troll in a pool of their own blood in a gutter with very few of the their teeth remaining back in the day. As it is now the anonymity of the internet has made these types of crap-stirrers practically drunk on their relative immunity against repercussion; they can get away with acting like utter dolts because they're sat anywhere from a 1 mile to 10,000 miles away from the people they're affecting.

When I was younger, perhaps under 16, I wasn't the brightest spark on the internet either. We didn't have quite the amount of opportunities to be utter tits as we do now but I still managed to make an ass of myself on several occasions. Problem is it's not just the youngsters. As a player of Eve Online for over 2 years I've born witness to some extremely funny stuff at the expense of others but also some extremely embarrassing childishness from a large number of "adults". I say "adults" because while they might be in their mid-to-late twenties everything they do degenerates in to an extremely bad episode of Jerry Springer, with insults, slander and general spack galore. Despite the developer's boastful claims that the average age of the Eve Online player is 27 if you spent two weeks in the community you would not believe it, you'd have thought you'd entered a 4chan/Something Awful forum spin-off but somehow worse because the issue of the day is internet spaceships. A cheap imitation where you lose faith in humanity slightly because the people posting are actually your age or older.

Posted Image
Eve Online - serious business by not so serious people


Don't get me wrong; 4chan, Something Awful, Urban Dictionary and sites of similar ilk fill a great niche on the internet. You'd be dead inside if you went on to one of those sites and didn't find something there that wasn't funny to you (it might take longer for some, and I'm not recommending it to anyone), problem is you'd have to trawl through some of the most disturbing base content you're likely to witness without actively searching for it. It's anarchy online; some good bits, but a whole crap-ton of bad stuff. These sites are great, in their own way, as long as what happens on these sites, stays on these sites. The problem is when impressionable idiots start bleeding out from these sites and thinking that they can act like that and share similar content on any site on the internet. These types intermingle with the gaming community, often taking root in the more devolved side of the gaming community such as the stereotypical whiney, voice hasn't broken yet, underage Call of Duty players and they start spreading across the internet spouting out old 4chan memes like they were written yesterday.



Banhammer


As a result myself and the moderation team here are what you might call "ban heavy" to compensate. We've been doing this a long time now and we like to think we can spot these types from a mile off. Our suspicions are regularly confirmed when I get a nice earful of old memes and insults in to my inbox.

On average the Nexus community is now growing by 2,500 new members every day. That's a new member every 30 seconds or so. With the community already close to 2 million members and a moderation team of just 9 at the moment that's a moderator to every 220,000 members. So why don't I just get more staff?

Posted Image
Nexus registrations per day


Good staff are hard to come by. Over the past 9 years many people have offered to help on the site and I've often taken them onboard only to find they're interested for about a week and then their activity peters off, happy to remain on the staff list without contributing anything substantial to the site. While I'm sure their offer was good intentioned I'm looking for good action, not good intentions. I stopped taking on help from people who offered it around 2006 and instead decided to actively seek out the right staff for the job.

What I look for are staff who are mature, who know the rules of the community well and who have some understanding of the games and a finger on the pulse of the modding community. When I look for new staff I'm looking for people who have shown a dedication to the community; people who have spent their time helping others for the better of the community and have been here long enough, regularly helping people long enough to prove that they aren't just here on a short stay. Most importantly I have to be able to trust them. Indeed none of the staff asked to become moderators, I personally asked them after reviewing their activity within the community, and not just on this site. And it's worked. I honestly believe the staff on this site are the most professional, the most efficient and the most hard-working of any community out there on the internet without being paid.

In light of these tough criteria it becomes hard to find helping hands.



A rating system that's actually useful



A new rating system has been on the cards for the sites for a long time now. Looking back it was debated in length in the private staff forums back in November of 2009. Unfortunately it becomes extremely difficult to create an effective rating and review system for files on the site knowing that there are plenty of trolls out there that are willing to crap on anything without delay.

In the past the rating system on the site was a simple 1 to 10 rating form that anyone could use. It quickly went down-hill as ratings simply became a 1 or 10 and file authors took anything less than a 10 as a personal insult. Most file authors who received anything less than a 10 would instantly message members of the staff; some requesting a review, some outright demanding (very rudely) that the rating be removed or they'll leave the sites never to return again. Needless to say we learnt a lot about some of the members and mod authors from this experience.

As it was the rating system was coupled with a Top 100 system that worked on a Bayesian estimate of the best files that was weighted. A single "bad rating" could mean the difference between 10th and 25th on the Top 100. The formula wasn't perfect but it worked to an extent. It would have worked a whole lot better if the ratings were allowed to remain but the rating system became a way of trolling mod authors.

To avoid more drama than I was willing to deal with at the time the mod authors were often accommodated, some low ratings were deleted and once the more outspoken mod authors were accommodated the less outspoken but still interested mod authors followed suit wanting any of their "bad ratings" removed too. What followed was a year or so of 1 or 10 ratings, with anything in between being called troll bait by most mod authors and requests (or adamant demands) were quickly made to have the rating removed.

Any new system needs to avoid this potential pitfall.

As far as rating systems go on the internet the simple 1 to 5 stars or 1 to 10 or 1 to 100 system is still the most widely used and for good reason; it's simple rather than confusing and it's not time-consuming, which means the user is willing to do it.

I dabbled with the idea of some sort of questionnaire rating system where users would be provided with a set of questions they had to answer with a rating from 1 to 5. The problem being that 5 questions for one file might not be relevant for another file and trying to make a system have pertinent questions for the specific file in question is more work than it's worth.

Instead I've decided to opt for a return to the age old 1-to-10 system with a finer eye for detail. This new system will be exclusive in nature, requiring that users have to pass certain criteria before they are able to review a file. At the moment the criteria I'm working on is a post count of 50 or higher, an activity level of 12 or more, or a Supporter or Premium Member membership. This criteria limits the potential number of reviewers to just under 18,000 as things stand. The hope being that these 18,000 individuals have proven themselves to be mature enough for this community by the very fact they've made 50 posts, uploaded files or images themselves or financially supported the site without getting themselves removed from it.

The overarching aim is to have a review system similar to Amazon's. The user rates the file between 1 and 10 and has to provide a small review of the file, perhaps with a minimum word limit of 100.

Posted Image
Amazon rating breakdown


A rating system is important because it provides potential downloaders with a benchmark on which to base their decision to either download or ignore files while also managing expectation. If you play a mod that's been rated an average of 5 by venerable members of the community then you go in with your expectations at the right level; you're clearly not expecting a Mona Lisa, which might make the user more receptive to a pleasant surprise. The inherent issue is in trying to get mod authors to appreciate that there is always room for improvement and that a 1 is bad, a 3 is below average, a 5 is average, a 7 is good, a 9 is amazing and a 10 is perfect, i.e. unachievable. If I can get this in to everyone's heads; reviewer and file author alike then the system may get used properly, in turn providing more accurate data for potential file downloaders while offering another feedback avenue for file authors.

Similar to Amazon these same 18,000 who will be within the reviewing criteria will also be able to agree or disagree with their peer's reviews, up rating or down rating it respectively. A review with more positive ratings will "float" to the top of the reviews page as it is clearly a respected view, a review that has a negative review from its peers will sink to the bottom of the page, or perhaps be removed completely. Mod authors and reviewers will be able to comment on the reviews themselves as well. To top it off a bar graph will be provided with a rundown of all the ratings the file has received from 1 to 10. The focus will be on quality over quantity.

Gone will be the old appeasement policy of "rate on what it says on the tin, not on anything else". Essentially the old system asked you to rate the file based on what the mod author said it did. If you made an awesome tin-opener object to place in to the game and said as much in the mod description then you'd expect to get a 10 for it. With the new system I want to promote something very different, essentially saying "Great, you've added a tin opener, but where are the tins to use the tin-opener on?". Your file might be good, but it might have a whole lot more potential, it might need tweaking, it might lack an actual purpose. Some mods get added "just because I can". That's great, but why is someone going to want to download a file that replaces all the posters in the game with pictures of you on your holiday? Should such a mod get a 10 rating just because it says what it does? Of course not; the rating and review system should tell you about files you really need to download as well as how good these files are.



Endorsements



Far from becoming obsolete the endorsement system will go hand-in-hand with the new review system. For years now the endorsement system has taken on the role of a rating system and while it has served its purpose of making the life of the staff easier it sucks as an appropriate gauge for how good the files really are.

The endorsement system is really a way of giving kudos to a file, a way of saying "I like it" or "thanks". It tells you nothing about the actual quality of the file only that you liked it enough to endorse it. With this in mind a reviewer can easily endorse a file and say they like it whilst also giving a file an average rating of 5, perhaps lower. Yes, you may like the file, but it's probably not perfect. You can probably find weaknesses, areas to improve on, missing features or niggling bugs that don't make the file a perfect 10.

The endorsement system will have its "Thumbs down" negative endorsement system removed entirely. The system was only really put in place to weed out the trolls (albeit some people did use it properly) and I believe most of the file authors and downloaders know it for the tool it is, well aware that the thumbs down have no affect on anything substantial.

In light of this the new review system will be opt-in, meaning it won't be enabled by default for all the files currently in the database to date. Looking back at the uproar that was made when files received less than a 10 in the past I'm not willing to take another barrage of attacks without the file author accepting the terms of enabling the review system for their files.

And what will the terms be? A willingness to accept that the files you upload might not be perfect, to accept that other users might not see the file in the same light that you do. Yes, we absolutely respect and thank you for your free offering to the community but does this free offering make you impervious to any critical comments? If someone offers me a free yellow snow cone it wouldn't be natural for me to say "well it's free so I can't comment". If you believe it does then please, keep your files here but don't turn on the review system. But most of all file authors must have a willingness to accept that anywhere between a 5 and a 10 is a good rating.

The hardest part of this new system isn't going to be coding it or implementing it, it's going to be drilling in to reviewers and mod authors alike the true value of the rating numbers and that it's in everybody's best interest that their reviews are fair on both sides of the coin.

It's highly likely I will not make a top 100 list for file ratings as a result.

Work on this system will begin soon.

#2
ub3rman123

ub3rman123

    Like a trained monkey. Without the training.

  • Premium Member
  • 5,503 posts
Wouldn't requiring more detail for a rating lower the number of total ratings? Many people aren't willing to put in much time to do things like that.

The main problem with the rating system of 1-10 was that people don't understand what the individual number mean; only that a 10 was good and a 1 was horrible. I like the system Wired Magazine uses for it. THey explain what each number means. 1 is a total failure in anyway (Very very rarely seen), and 10 is metaphysical perfected (Also very rarely seen). Many people gave tens on the old 1-10 system out of fear they'd be insulting the author, and don't know that a 5 means average. If 5 were average for a review, the average rating for all mods should have been 5.

Edit: Okay, I just noticed the last part. And I pretty much restated the ending of it. Sorry, my mind had glazed over by the last few paragraphs.

#3
Dark0ne

Dark0ne

    Webmaster

  • Admin
  • 21,419 posts

The focus will be on quality over quantity.



#4
chronosmagus

chronosmagus

    Enthusiast

  • Premium Member
  • 130 posts
Personally I'd go for the 0 to 5 stars rather than 10, where the average 'decent' mod would likely get around 3.5 stars. Might have to do that back at school we used the 0 to 10 system, where it was actually quite important what you got.

I truly appreciate your effort to keep looking for the best solution on filtering out the most interesting mods, especially since the database is so huge. The small comment included with a rating should give more accurate ratings, as people reading them can see instantly what 'part' most people like about the mod.

One suggestion that just came to mind that is somewhat related. Implement a box containing: "Other users who downloaded this mod, also downloaded:". It's a well proven method that usually improves itself as the database gathers more statistics.. plus it's nearly impossible to cheat in the long run.

#5
WastelandAssassin

WastelandAssassin

    Your Friendly Neighborhood Assassin

  • Premium Member
  • 6,865 posts
well, as long as it was (and it was rather long) it was well worth the effort of reading it in full

well, trolls and such flamers and such are a real problem, and i agree with how they miss-used the commenting and rating systems, so i see the point in making a new system
and i believe you are well on the right track with the idea of the new rating system

however, it does hold a little problem, at least in my eyes
considering the fact that not all files are overhauls, or huge mods, writing a 100 word comment on most mods wouldn't be too easy
also, will these comments be checked, before a persons' review is used?? because if a person will be allowed to write a stupid 100 word comment (which is easier than it sounds, like writing "Blah blah blah" 33 times) then the system will again be open to trollers

so in the core of it, the new system is a brilliant idea, but it might take some time to find the best settings for it, such as the lowest bar for making such a comment (though 50 posts or level 12 does make sense to me)
and just how to write such a comment, with all it's requirements, must be put into some serious consideration

luckily, i agree with you about the efficiency and high quality of the work done by the Admin team, and so i believe you will be able to find the best settings, and implement them into this new system

best of luck, from this humble user
and may people start thinking about what they say, before starting to troll and flame (won't that be a perfect world.......)
Kudos for your hard work :)

Wasteland Assassin

#6
VictoriaG

VictoriaG

    Forever Fallen

  • Supporter
  • PipPipPipPipPip
  • 3,771 posts
Will this rating system be applied to the imageshare to?

#7
Dark0ne

Dark0ne

    Webmaster

  • Admin
  • 21,419 posts

Will this rating system be applied to the imageshare to?


No. While I use "rating" and "review" interchangeable in my post this is going to be a review system at heart. I think the Image Share is better off with a simple "I like this" type system rather than a number rating system considering "art" is so subjective.

#8
VictoriaG

VictoriaG

    Forever Fallen

  • Supporter
  • PipPipPipPipPip
  • 3,771 posts
I agree with that the current imageshare system works well :)

#9
ilDuderoni

ilDuderoni

    I'm so cynical, bananas peel when I look at em

  • Premium Member
  • 1,301 posts
And in the end, all those means were to weed out trolls and keep Mod-creators happy...
Sometimes I think the ones who are in the worst predicament in this situation are the admins who have to make sure it won't go to hell...

Read it top to bottom, and I'm convinced that, if anything, nobody can claim you're not putting enough effort in these websites. I hope you still have enough (Gaming) R&R to keep your spirits up?

#10
Pushkatu

Pushkatu

    Outlander

  • Premium Member
  • 21,527 posts
Good news! I can't wait to see the new sistem in action. It looks more professional and in time those rewiews will come in handy. Keep up the good work!:thumbsup:


Cheers,
Pushkatu!




IPB skins by Skinbox
Page loaded in: 1.046 seconds