Karma - How is it calculated?

[Stuart Bishop]

Carthik Sharma wrote:

> I just got started using Launchpad to report and triage Ubuntu and
> Launchpad bugs.
> 
> I would love to learn if the process of calculating Karma is
> documented anywhere. I have looked in some Spec files and the wiki and
> found nothing, except some proposals for improvement (including a
> thread in the archives of this mailing list).

When working with Launchpad, certain actions such as commenting on a bug or
translating a string cause a record to be stored in our database of the
action type and the time it occurred. Each action type has a numerical score
associated with it. Each action type is associated with a category (bugs,
translations, specifications at the moment - 'support' category coming soon
I hope).

The score associated with each action type is currently just a best guess.
We have yet to do any serious analysis on the real data to determine which
actions should be rewarded more and which actions rewarded less.

Each day, we summarize this information and calculate everyones current Karma.

For every user with Karma actions, we total the raw scores for every action
they have performed, per category. The raw  score for each action is reduced
depending on its age; this is currently done linearly, so an action that
happened 'today' is worth 100% of its raw score, an action that occurred 182
days ago 50% and an action that occurred more than 365 days ago is worth
nothing.

We feel it important that Karma is reduced by time, as otherwise new users
may feel that they have no hope of catching up with a long time user, even
if that long time user is no longer active. It also means that karma given
incorrectly will slowly become irrelevant (there is at least one outstanding
bug where users are getting translation Karma when they shouldn't, and our
data model does not let us tell this good karma from bad). The way Karma
degrades, and the time frame of the degradation, is open for debate. We can
plug in other algorithms easily enough to test alternatives - the original
events are unaltered so no damage is done when we tweak the algorithm or
alter the per action scores.

We now have the issue that the Karma being given for some categories vastly
outweighs the Karma being given to other categories. At the moment, for
every point being given in the bugs category, 10 points are being given for
translations. And for every point being given in the specifications
category, 200 points are being given for translations. This is a problem
because it doesn't encourage people to do bug triage if they never have a
hope of getting a similar Karma to translations.

In the long term, we hope to reduce this problem by adjusting the scores per
action so that the all of the categories end up with a similar amount of
Karma. However, we now also normalize the categories to ensure that the
weight of the various categories automatically tunes so the total number of
points in each category across the whole system are equal.

To do this normalization, we total the total amount of Karma given to people
in each category (as calculated above - shrunk by time). We then work out
the scaling factor for each category, and then multiple everyone's Karma in
each category by this scaling factor (bugs around 10, specifications around
200, and translations unaltered since that pool has the most points). Note
that because these scaling factors are recalculated daily, it introduces
some 'wobble' into the Karma scores although this shouldn't be noticeable in
the long term.

We then store the adjusted per category scores per user. These are the per
category scores that are visible on your Karma details page
(https://launchpad.net/people/stub/+karma). Your total Karma is the sum of
these category scores.

(above now duplicated at https://wiki.launchpad.canonical.com/KarmaCalculation)

> Also, Using launchpad will be a lot easier for new users like me if
> there was a documentation wiki or someo ther kind of documentation
> that explains how things work, and the right way (With screenshots, or
> a list of actions required) to complete tasks, such as confirming a
> bug, forwarding a bug upstream etc. Maybe the right question at this
> point is, what User documentation exists, if any? :)

(I'll leave it up to people dealing with the UI to comment on this)

-- 
Stuart Bishop <stuart.bishop at canonical.com>   http://www.canonical.com/
Canonical Ltd.                                http://www.ubuntu.com/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/launchpad-users/attachments/20060329/06654352/signature-0001.pgp