The problem with online reputation

When it comes to seeking out knowledgable others about our work, we have a reputation problem. The people who have the highest reputation scores are usually the people who are the most public.

Offline we all know who to turn to ask advice on specific things. Specific people we work with now and people we’ve worked with in the past. Yet when thinking about turning to people online who we may not have worked with before, it’s nowhere near as straightforward. In fact, some of the best people I’ve worked with, the people whose opinions I most highly respect, have really low online reputation scores – as measured by emerging services such as Klout and PeerIndex.

These services are tackling an immensely challenging problem, credit to them. But this is a human problem, not a technology problem, and I’m not sure that measuring what people are saying publicly (or semi-publicly) and how others are responding is the right approach.

In my experience there is little correlation between people who are public, or people who are loud, and people who are knowledgeable. There are some very knowledgeable people who are public and loud. I follow them. I read their stuff. But there are a far greater number of people who are equally (and often more) knowledgeable, but simply prefer not to engage in public discourse. These are the people that matter.

Often, social media tools afford a lack of critical thinking. See something, half read it, tweet and reweet it. It’s easier than crafting a thoughtful response. We’re all guilty. The problem is that it amplifies the connected group, without much analysis of the quality of the content. Through the lens of our online measurement tools the author looks knowledgeable, the content looks good, and the retweeters look influential. But none of those may be true. Being part of a clique community on Twitter does not make someone knowledgeable.

Moving away from the tweets and retweets, “official” approaches like LinkedIn recommendations are also problematic when determining who is knowledgable. Due to social pressure, people only give positive reviews. Everyone is an expert. I’ve seen job adverts where employers ask for at least three reviews on LinkedIn. That’s not a job requirement, that’s a game.

Another approach is to compare people, for example Cubeduel lets you vote between two coworkers. The problem with this approach is that very few people are directly comparable. You can’t compare people who do completely different jobs. More importantly, we’ve built good friendships with many of the people we’ve worked with, and comradery driven obligation takes over. Our friends get the vote over someone who might be better than them.

Honestly.com tries to solve the social pressure problem that exists on LinkedIn by allowing people to leave reviews that are anonymous to others but attributed to the real you behind the scenes. However, the lowest average score I’ve seen is 4 out of 5 stars – so something is clearly not working. I’m guessing that people are afraid of their review being accidentally leaked – and the risk is much greater than simply giving people a good review.

So given this mess, how do we figure out who is knowledgeable by measuring reputation and describing it online?

As with most people problems, I feel the roots of the solution lie offline. From our ongoing face to face interactions, we learn who is knowledgeable, who to turn to for an informed opinion, who is likely to say it like it is, and who has hidden agendas. There is no substitute for that. Reputation is built conversation by conversation at the desks, halls, cafes and meeting rooms of businesses all around the world. Any attempt at measuring reputation needs to map this.

The first step is to map who worked with whom. Not that we both worked at Google at the same time, but that we worked in the same team for an extended period. Then we need to map how much interaction we had. We collaborated almost every day. Or we were in the same meeting every week but not much more. We then need to map areas of experience and expertise. That Paul knows about doing research. Or design. Or he knows about online identity. Or designing communication tools. Finally we need to cross reference those areas with other people who also worked in the same team. Does Paul really know about “online identity”? Do you really trust him to help you create new communication tools? In effect, it’s like turning an internal peer review process inside out.

Mapping this will be challenging, it will involve incentivizing people to rate or describe the people they have worked with, the people they trust. But until we do, we’ll continue to hear about the loudest, most public people, and miss out on the people making the greatest difference inside companies around the world.

Tags: , ,

24 comments

  1. [...] The problem with online reputation – Are you thinking inside out? thinkoutsidein.com/blog/2011/02/problem-online-reputation/ – view page – cached When it comes to seeking out knowledgable others about our work, we have a reputation problem. The people who have the highest reputation scores are usually the people who are the most public. Show influential only (1) $(‘#filter-infonly’).change(function() { var el = $(this); var url = document.location.href; var checked = el.attr(‘checked’); if (checked) { document.location.href = url + ((/?/.test(url)) ? ‘&’ : ‘?’) + ‘infonly=1′; } else { document.location.href = url.replace(/[?&]?infonly=1/,”); } }); [...]

  2. A few people I really respect are convinced that Reputation (with a capital R) is a huge opportunity to inform interactions. I can see how this works out today in the very very basic way in that a Phone + Location that matches a Visa Purchase + Location tells you that “this” person is “here”. I’m not sure that the penchant for “self reporting” is all conquering because people are such bad self-reporters (see behavioural economics reading). Some form of automation is clearly being looked at by Cisco with the pulse product, and they have some history in the space because some of these reputation problems are similar to Skills Based Routing, KM, “answer giving”. On the upside, it seems like a “big problem” and big problems tend to receive funding :)

  3. Love this commentary – and you are right about the information that is hidden.
    I characterise this in two ways:
    a. real-world authority / reputation – the guy who wins a Nobel; the gal who gets the fields
    b. stuff hidden in your head

    Of course we realise that modelling existing social activity can only take you so far. In the same way that before Google indexed maps, it couldn’t give you driving directions, our intention is to invest and acquire, create or derive the missing data to provide that level of insight.

    Your experience that there is ‘no correlation between people who are public’ just seems wrong, or counterintuitive, and certainly contrary to the data that we see through our testing. We see a very definite positive correlation because there tend to be very few people who are knowledgeable who don’t leave some form of digital trail (take for example: http://en.wikipedia.org/wiki/List_of_Nobel_laureates_in_Chemistry#Laureates )

    I realise that when you said ‘no correlation’ you were using it as a turn of phrase. But as you then used it to base the whole of the rest of your argument, I think it’s a bit sloppy.

    The other part of your argument that needs revisiting is this statement: ” Being part of a clique community on Twitter does not make someone knowledgeable”.

    But nor is it the case that ‘ Being part of a clique community on Twitter does not make someone lacking in knowledge’. And in fact we know of several cliques on twitter where the most knowledgeable people in their field hangout- take early stage investing or internet product design.

    And to the conclusion – you are right but you also under-esimate the scope of behaviour change. What happens in a world where more people put themselves online? I remember in mid-90s when not even large retail store were online. Then large ones can online. Now even small ones are online. You characterise this as people who like self-promotion (call them loud) and people who quietly do their jobs. You could easily characterise it as an academic who publishes their work, and an academic who timidly burns their manuscripts but things great thoughts. Or you could characterise it as a view held in a moment in time, when the social tools are just emerging and not everyone participates. I think it’s the latter–everyone will participate, unless they choose not to, so their data will be there for them to benefit.

    best
    a

  4. I agree that in context of the social graph Cubeduel, LinkedIn and Honestly all fail to provide accurate reputations of one’s offline “voice”. I’ll disagree that in context of the interest graph, Klout and PeerIndex do a great job of judging one’s online voice, since no offline voice exists in this setting.

    LinkedIn has a new feature where you can add skills to your profile, and then rate yourself 1-10. Instead of self-critiquing your skillset, what if LinkedIn gave your connections the ability to anonymously rate your skills, and display the mean score. This would provide credible and accurate assessment of each individuals skills.

    Thoughts anyone?

  5. @Paul Agree that it is unlikely that we will see a large amount of self-reported data.

    @Azeem Thanks for the thoughtful reply. A few responses:

    I didn’t actually say that there was “no correlation” between knowledge and public visibility, just that in my experience there was “little correlation”. This is still the basis for what I believe. It’s simply what I’ve observed. This belief is also not universally counter-intuitive. Intuition is based on individual experiences, so perhaps counter-intuitive to you, but not to me. Thinking back over the past 10 years, I have found that the people I respect and trust the most are not the people who are public online. There are exceptions, but in my experience this mostly holds true.

    In my experience, most of these people don’t want to be public online. Public discourse isn’t what motivates them, discourse amongst an internal product development team is what motivates them. there are many very knowledgeable people working inside organizations that have no digital trail that references their expertise. So I have to disagree with your assertion that the most knowledgeable people in the field of internet product design are very visible online.

    I’d love to hear more about the data you guys have that shows this not to be the case. Again, this is simply my view based on my observations.

    From discussions with others, there are many reasons why they prefer not to have a publicly visible presence. For example, some are discouraged by their employer from doing so. Others prefer not to have to deal with the overhead involved. And others still would prefer to invest their energy in internal discussions.

    Finally, I don’t think any of us can predict where this is headed. Maybe people will be forced to deal with a public profile built for them by aggregation engines, or maybe not. Certainly, from research that I and others have done in the past, there are many people today who do not want to have a visible online presence, and who do not want to be found by certain others online. Not sure if you’ve read it, but I highly recommend danah boyd’s talk from SXSW last year: http://www.danah.org/papers/talks/2010/SXSW2010.html

    Would be interested to hear about any data you guys can share that correlates knowledge in a field with public digital trails. This problem suffers from unknown unknowns (how do you find the knowledgeable people who prefer not to engage in public discourse), so I’d love to hear your take. Also looking forward to seeing what you guys are building next.

    @Ryan Hadn’t heard about the 1-10 scale on LinkedIn. Interesting, but as Paul also pointed out, self-reported data is unlikely to be a large part of the solution in this space. It’d be interesting to see them experiment with privately accountable but publicly anonymous ranking (similar to Honestly.com).

  6. [...] The problem with online reputation – Are you thinking inside out? thinkoutsidein.com/blog/2011/02/problem-online-reputation/#comment-9970 – view page – cached When it comes to seeking out knowledgable others about our work, we have a reputation problem. The people who have the highest reputation scores are usually the people who are the most public. [...]

  7. [...] up a post at Facebook. On his personal blog, he’s just posted an excellent commentary called “The Problem with On-Line Reputation”. In it he states:- As with most people problems, I feel the roots of the solution lie offline. From [...]

  8. Let’s not assume that “reputation” is one thing. In fact, it is many things. Is it not made up of character reference, professional reference and performance assessment? I think so and as such then you start to better understand that each has its own identity and each has its own solution. The trick is in incentivizing the collection of data for each. For starters always give the evaluator a choice to offer data anonymously or publicly to various audiences. If the evaluator is skeptical that anonymous data will be “leaked” then the data collector has a perception problem which can be solved in another discussion. In any case, the consumer of “reputation” data understands the inherent biases as they do any commercial advertisement. Therefore, the data provider doesn’t have to be perfect but just perceived to be “good enough” or “more credible” than an alternative by the consumer.

  9. Interesting post. There is a great positivity bias in online feedback on persons whether attributed such as on LinkedIn or anonymous such as on Honestly.com. But not necessarily on products where kicking a brand’s posterior is routine, almost the default. What gives?

    My working hypothesis is, just as we are loathe to tell someone to their face they have BO or they are ill-mannered, on the web too we keep our counsel to avoid mud-slinging. When we open our mouths to say something, we say complimentary stuff. Or else we do not say anything. But we do privately whisper to our friends about that negative stuff.

    Products however are just things, not people. They don’t have feelings. They cannot be nasty to us or our families online. Companies are and behave like non-organic entities. So when we are upset, we let rip.

    That said, I think tools, even those that approximate the real world, will provide an asymptotic fix but will fall short of the complexity of how we determine what we like and don’t like, and more importantly, when we are willing to say it all aloud.

  10. The other approach is for HR recruiters to get out of the “hiring” business and for “managers” to get back into the hiring business.

    An engineering design manager say at BMW can immediately recgonize knowledge and talent. No needs for behavioural test, background test, blah blah blah.

    A HR recruiter at BMW has no idea who is a talented automotive engineer, it is just a guess, and so likely dumps many, many great candidate’s resumes into the trash becuase they are NOT BMW design engineers.

    Somewhere along the line, real managers, line of business mangers gave up responsility to hire and HR recruiters took it over.

  11. [...] he problem with online reputation Cet article a été publié dans Citations. Bookmarker le permalien. Laisser un commentaire ou [...]

  12. Great article and some very valid points. I would argue however that there are many of us, myself included, who are very knowledgeable but can be shy engaging offline. Online publicity gives people like us a voice and recognition that we may otherwise not have received.

  13. Some people don’t want to be found, or can’t be found. What do you expect the sages among you to do, stand up and shout their own praises? The truth is, people can’t really tell the difference. In the end, reputation ends up approximating the curve drawn in any other social game — gregariousness is rewarded, not merit per se.

  14. Well the web worked well for me tonight, something that seems to happen less and less. I bumped into this blog (can’t say stumbled on, cos I did’nt and people would take it I meant another thing)

    Reputation online is a little like being the swot at school and the hooligan on Saturday night. We all know thats a condition that can exist in the real world.

    Is it not reputation but level of education that is actually being measured?

    Your blog has made me think, very cool

    Thank you

    G

  15. I have come to the conclusion that one’s PeerIndex or Klout score should never exceed one’s age. Ideally, it should be about 10 digits less. The closer to a “celebrity” you are, the quicker you age!

    Okay, it’s a satirical comment, but all that to say I couldn’t agree more with your post. Unless we can map the offline elements you illustrated that go into any given person’s reputation among a working group, the metrics these sites come up with are like an oil sheen on top of a body of water: pretty, often dramatic, but no indication how deep the pond is.

  16. Very interesting discussion and some very good commenting on it. I can only say i love that post.

  17. I totally disagree. The premise that offline reputation signals are superior defies my experience. The idea that “Reputation is built conversation by conversation at the desks, halls, cafes and meeting rooms of businesses all around the world. Any attempt at measuring reputation needs to map this” is a selective view. Offline reputation is fraught with signal interference: entrenched decision makers; political dynamics; social and peer pressure.

    But mostly, the problem with offline reputation is (eg) you don’t get the best attorney, you get the one floated by your small network that is a very small random sample vis à vis the whole population.

    I think online reputation has all of the problems of offline reputation (e.g., the outgoing get more noticed) but fewer; and then the overwhelming benefit of reach.

    I don’t see how it has yet created any new, meaningful reputational signal problems that didn’t exist before. It’s all good and getter better

  18. The silver bullet on this problem will be very hard to find, but you did a great step in observing in detail the mechanisms. Great post, I was even thinking retweeting, untill of course keep reading the second half – just joking.
    All your observations fit quite well into something we have discussed recently ( @rossdawson @metaphorage @ithorpe ), the internet (Twitter, FB, Linkedin, etc.) is not flat, but self-organizing “Knowarchies”
    http://geraldmeinert.blogspot.com/2011/04/knowarchy.html

    regards
    gerald

  19. Online Reputation is today too much related to influence. Today, you are an influencer on internet if you have lot of followers on Twitter or a good trafic on your website.

    But what is the problem? If you are good on managing acquisition of new followers or SEO, you can start to be influent juste because you have followers. But do you think that having lot of follower make you someone we should trust? I think, no.

    I think that influencer and expert are people that know their market and what will be the next move. On Beansight, we developed a website where people can share their thoughts and we track who was right. Our algorithm give credibilty points to people that make good statements. Tha’ts everyone have a public profil where you can see what his fields f of expertise.

  20. nice blog post, and even nicer discussion. About online reputation, offline reputation, I think they complement each other. Also, I think we should look for reputation systems that combine different inputs in order to validate the results: As Google uses many different kinds of interactions and behaviors to evaluate websites, we should not look at direct f2f ratings (obviously that won’t work), but look at all the different ways –how– people use the contributions. We need to have an ontology for different uses (what does re-tweeting mean really? does it relate with knowledge/skills?), but also we need to be able to catch the context of use in order to make the right interpretation of the use.

    @Cyril: I had a look at Beansight.com >> Really COOL! Will sign up.

  21. Lyndon Williams

    My only real comment is a concern that everyone is going to have to start living in glass jars, where their every action is going to be put under the microscope for commentary and inevitably for some, trolling. Not everyone in your ordinary everyday interaction with them, even your closest family members, reveal each and every single detail that relates to them. There are mistakes that we all make in life, some of them rather epic in nature and others rather menial. I feel that in an age such as the one that we live in where gossip and vitriol generally moves at such a great speed, that there need to be checks and balances put in to place or else there are going to be problems. And I am talking about personal lives here for the most. I do believe that we all have a right to privacy as individuals and that this should not be violated by technology, no matter what it is. There is such a thing as oversharing. How do we have checks and balances in open networks? If we don’t think about this responsibly now, then legislation will put this in to law later on. We should not have to wait for that now, should we?

  22. From my point of view today’s online marketing strategies and means run both ways. And I say that because it’s extremely probable that you might try to increase your online reputation and at least one if not more of your competitors might try in the same time to diminish your reputation and you find yourself engaged in an activity that is very costly for both you and your competitors but the online situation stays pretty much the same. Yes, it is a good and sure way to improve your online image but it always has to be responsibly taken care of.

  23. [...] -Paul Adams .fb_iframe_widget { vertical-align: top !important; margin-left: 16px !important; } Tweet Tagged: [...]

  24. [...] people who have the highest reputation scores are usually the people who are the most public,” argues Paul Adams. Those who are willing to live their life more freely on the Internet are the reputation [...]