This reminds me of my foray into tagging, where I had to dust off my own understanding of distributions. On the one hand, the fact is that the topic can just be plain hard, and at times pretty counterintuitive. On the other, I’m sure that numbers themselves seemed hard until they became part of almost everything we do.

One point that seems like a good one: it’s true that statistics are essential to understanding things like news reports, Google Analytics, sales dashboards, sports stats, etc., and it probably wouldn’t be a bad idea for this fact to be reflected in school curriculums.

It turns out that this depends on whether the power law is an exact inverse (Zipf: a = 1) power law or a higher order power law.

The top user u = 1 has M percent of all taggings, so the curve is t = Mu^-a. Each bar measures the percentage of taggings by that user, so the sum of all bars has to equal 1. So for N users we have

M + M/(2^a) + M/(3^a) + … + M/(N^a) = 1

or

M = 1/(1 + 1/(2^a) + 1/(3^a) + … + 1/(N^a)).

For a Zipf law with a = 1, the denominator is the harmonic series, which diverges; so that means the % of taggings by the top user drops as the number of users N gets larger. We can calculate M by remembering that the harmonic series sums to gamma + ln(N) as N approaches infinity, where gamma is the Euler-Mascheroni constant and ln is the natural log. We can check that this is close enough after N = 100, so calculating N = 10 by hand and using this formula for the rest we have:

Gotta love NumSum. But if a > 1, the series in the denominator converges, so that as the number of users N increases, the % of taggings by the top user M quickly settles to a constant:

This is all in follow-up to the fourth point from this post:

(4) While it is true that “bigger systems benefit from both higher heads *and* longer tails,” in general this usually just makes the histogram fit the curve better; it is rather the shape of the curve that determines whether or not “most activity is from a small group of highly active users.”

A Zipf law is a case where a bigger system actually has a distinct effect: the bigger the system, the lower the percentage resident in the highest ranked bin, resulting in a lower percentage of activity from the most active users. In the case of higher power laws, this percentage quickly settles to a steady constant, so size doesn’t have much of an effect once the system is reasonably big.

As an aside, I was also asked to post the graph presented at TagCamp showing a histogram that fits a “long tail” but not a power law, so here it is:

Although this looks similar to a power law, if we disregard the top two users the histogram actually fits the curve that corresponds to a perfect bell curve PDF. This means that in contrast to a power law, where the average number of taggings per user is essentially meaningless, above this average is maximally meaningful.

I got rid of the “expert” scope column, not because it’s not relevant, but because there weren’t any entries and aren’t likely to be anytime soon — seems like amateurs are more in fashion than experts these days. (UPDATE: Ethan rightly points out that Squidoo is an example of expert filtering, as is really any successful media outlet, including Mike’s TechCrunch itself; I updated the matrix to reflect this and a couple other suggestions — if anyone else has more suggestions for entries, please pass them on!)

Nivi also points out that there are other dimensions that could be added to this matrix, in particular that of metadata location. He mentions that a lot of this valuable data is on your desktop, to which I’d add that a lot is also “locked up” in various applications, e.g. your search history, your tags, your OPML list, etc.

I also thought a great point was that the metadata is about both you and the data it points to:

This metadata is metadata about the data it points to and metadata about your interests and attention. In fact, the utility of a piece of metadata in describing data may be inversely related to its utility in describing your interests. For example, your clicks describe your interests, but they don’t really say anything useful about the data you are clicking on. (Propers to Ethan Stock for this insight).

However, I’m not so sure about the idea of an inverse relationship between the utility of metadata along these two dimensions. Your clicks may not *directly* say anything useful about the data clicked on, but indirectly they’re at minimum a vote (as used by AdWords), and even better a proxy for collaborative filtering (as used by Amazon). Delicious makes this explicit by allowing you to pivot on users, tags (metadata), and URLs (the data pointed to); each one of these dimensions has a different meaning, but I think every one has some utility.

In a post to the del.icio.us mailing list, Clay responds to someone seeking an idea of what the average number of bookmarks is per user. The question is asked in light of another, which is: are most bookmarks made by a small core of heavy users?

Clay responds that:

The deceptive thing about systems like this are that the average is meaningless, as the distribution is not a bell curve. There are, yes, a small core of highly active users, but the decision about who goes in that core is totally random, since the distribution of links per user is roughly a power law.

Both del and spurl get most of their activity from a small group of highly active users, but bigger systems benefit from both higher heads *and* longer tails.

I think that several parts of this exchange are representative of statements that really need some clarification to avoid potential confusion:

(1) The histogram of users ranked by number of bookmarks made (“the distribution of links per user”) is a ranked graph, which by definition is *always* decreasing, and can *never* be a bell curve.

(2) I don’t know about the actual del.icio.us data, but it’s true that many Internet-related ranked histograms seem to fit a power law. However, many others do not, despite exhibiting a “long tail”; they may better fit a negative logarithm, inverse exponential, or more complicated function.

(3) The question of whether the average number of bookmarks per user is meaningful is best decided by considering a completely different, non-ranked distribution, namely the number of users having a given number of bookmarks. This is the PDF corresponding to the ranked graph, and *can* be a bell curve with a meaningful average.

(4) While it is true that “bigger systems benefit from both higher heads *and* longer tails,” in general this usually just makes the histogram fit the curve better; it is rather the shape of the curve that determines whether or not “most activity is from a small group of highly active users.”

Here I want to expand on the second and third points. In a previous post, I showed that if the ranked data fits a power law, then the corresponding PDF is also a power law. Once this is understood, it is clearer why Clay is correct in saying that if the histogram of users ranked by bookmarks follows a power law, then the histogram of number of users per number of bookmarks also does, and therefore it’s true that an average is pretty much meaningless.

However, if the ranked data has a “long tail” but doesn’t really fit a power law, the corresponding PDF *can* in fact have a meaningful average; in fact, it can be an exact bell curve! So while a ranked histogram that fits a power law implies a meaningless average, a ranked histogram that just exhibits a “long tail” does not, and that’s why it’s better to look at the PDF when trying to answer this question.

To show this, we can analyze the above example by following the logic of my previous post in reverse. So, let’s assume that the distribution of the number of bookmarks made by users fits a perfect bell curve. This means that the PDF is a Gaussian or normal distribution:

This is the ideal case where an average is *most* meaningful and informative. Now, this means that we can integrate to form the corresponding CDF, showing the percentage of users who have made b or more bookmarks:

Finally, we can then invert this to get the ranked graph of users in order of the number of bookmarks made:

Now, I’d certainly agree that most actual histograms I’ve seen on tagging data, etc. fit a power law much better than the above graph; but if you were faced with data that looked like the above, especially with a rescaled y axis, you might think something like “oh, it’s just the long tail of a power law but the top guys are lower,” and not necessarily realize that this changes the situation significantly; in particular, the underlying PDF might not even be a decreasing function.

The main point is this:

If you consider ranked histograms, it’s easy to see power laws everywhere.

But in many cases it may be that fitting another curve would be more informative, or that the ranked graph is not the right one to be looking at in the first place.

Returning to Clay’s post, the other original question that was asked was: Are most bookmarks made by a small core of heavy users? This can be answered by looking at the ranked histogram: if the median line dividing the area under the curve in half is far to the left, then most bookmarks are made by the top few users.

To get an even clearer answer to this question, it seems to me that the obvious thing to do would be to integrate the ranked graph, resulting in a curve from which you could easily read that the top u% of users were responsible for b% of all bookmarks made. For example, if the ranked histogram fit an exact power law b = 1/u, then the integrated graph would be a logarithm:

If we consider a ranked histogram that fits a higher order power law b = 1/u^2, the integrated graph shows an even higher dominance by top users:

In contrast, integrating the ranked graph resulting from a Gaussian PDF gives:

Here we can easily see that the heavy users do not have much of an outsized influence. I guess the purpose of all this, besides to get it straight in my own head, is to underscore the importance of this fact: the histogram you want to consider, and the usefulness of the curve that you fit to the data, depends very much upon the question you want to answer.

The issue is that long tail graphs have an x axis comprised of items ranked by their y axis value; e.g. for a social bookmarking site we can graph users ranked by how many taggings they’ve performed, and approximate it by an inverse power law:

One nice thing about a ranked graph is that the “area” under the curve is equal to the total value associated with the items spanned on the ranked axis; e.g. for the above graph, the area to the left of the median line represents 50% of the total number of taggings.

However, it seems as if both axes here are carrying essentially the same information (as Phil said, “that bar’s the tallest *and* it’s the first!”). But it turns out that the ranking and the value are in fact separate pieces of information, and a power law shape to the data carries over to more familiar distributions. As I go through how that works, I’ll add some notes on terminology, since it can get a bit confusing.

Firstly, a ranked graph like the one above that follows an exact 1/u inverse power law (i.e. a = 1) is often said to adhere to “Zipf’s Law.” This law originally described how the frequency of use of the nth-most-frequently-used word in any natural language is approximately inversely proportional to n. Subsequently the term Zipf has been used to refer to ranked data that can be fit to several different kinds of curves, but one thing’s for sure: a ranked graph is by definition always decreasing, and is not a probability distribution! (although as we’ll see below, it’s closely related to one).

The key to transforming a ranked graph into a more familiar distribution is this fact: saying that the nth ranked item has a value of y is equivalent to saying that n items have a value of y or more. So for the above graph, saying that the u-th user performed t taggings is equivalent to saying that u users performed t or more taggings:

Now if we invert this graph and turn the number of users performing t or more taggings into the percentage of such users, we arrive at a probability distribution, namely the probability that a randomly selected user performed t or more taggings:

This kind of distribution is called a Cumulative Distribution Function (CDF), and if it follows an inverse power law it is also sometimes referred to as a “Pareto distribution.” A Pareto distribution originally described the percentage of people owning more than x amount of wealth, and led to what is sometimes called the “Pareto principle” or the “80-20 rule”, e.g. that 20% of the population owns 80% of the wealth (note that 80 + 20 = 100 is a coincidence here, another common confusion!).

Finally, we can note that the number of users performing t or more taggings minus the number performing t+1 or more taggings is the number of users performing exactly t taggings, i.e. N(t) – N(t+1) = n(t) = -N’(t); here n(t) is the number of users performing t taggings, and is equal to the negative derivative of the number of users N(t) performing t or more taggings:

This is a familiar Probability Density Function (PDF), and is the more common setting to describing a distribution as fitting a “power law.” Note that if the original ranked graph follows an inverse power law of any kind, so does the corresponding CDF and PDF. The converse is not true, i.e. if a PDF follows an inverse power law, the other graphs only do so if the shape parameter b = 1 + 1/a is greater than 1. Otherwise, the integral of t^-b with b < = 1 does not give a power law.

Of course, after all this, the fact is that at least for me, the original ranked graph is the one that most directly and intuitively answers the question at hand: are most taggings by the most active users? I'd be interested in comments and thoughts, or other graphs that aren't covered above.

** Addendum: Some technical notes, mostly to refresh my own memory

- For a discrete set of outcomes, the function corresponding to non-zero probability values for each outcome is called a Probability Mass Function (PMF).
- A PMF is also sometimes called a Probability Function (PF).
- A histogram (AKA bar chart, step function) can be obtained from a PMF by binning the outcomes into ranges (usually equal or exponential) in a continuous sample space (set of outcomes).
- For a continuous sample space, a Probability Density Function (PDF) gives the probability of an outcome in a specified range by integrating the function over this range. A PMF or its histogram can be approximated by a PDF.
- For a PDF, the probability of any specific outcome is 0, non-zero probabilities only exist by integrating the PDF over an interval of outcomes.
- The "area" under a PMF or PDF must be 1; the area is the sum of all values for a PMF and the integral over the entire sample space for a PDF.
- A power law can only approximate a PDF and must be cut off at its extremeties, since its integral is divergent and so cannot integrate to 1.
- A Cumulative Distribution Function (CDF) F(x) is usually defined in terms of a PDF f(x) by F(b) - F(a) = integral from a to b of f(x), i.e. f(x) = F'(x). This means F(x) is the probability of any outcome less than or equal to x.
- A CDF is also sometimes called a Distribution Function (DF).
- Sometimes a CDF is defined with a different inequality; e.g. a Pareto distribution is a CDF F(x) which is the probability of any outcome *greater* than or equal to x, so that f(x) = -F'(x).
- A Pareto distribution is characterized by a shape parameter a and a minimum value parameter m, and takes the form F(x) = a m^a / x^(a+1) for x >= m, 0 otherwise.

What I’m picturing is a lot like Durl, except instead of history trends, URLs / tags / users would be associated with distribution graphs. There are many possible such graphs, so you have to pick the ones that answer the most interesting questions. The three that seem most interesting to me are:

So for example, upon entering a URL to get the first graph, each bar on the X axis represents a tag, with the Y axis giving the number of users who tagged the URL with that tag. The vertical line corresponds to the median, i.e. half of the taggings were done using tags on either side of that line; thus the long tail could be considered to be the portion to the right of this line, this “area” being equal to that to the left of this line. The horizontal line represents the average Y axis value (of arguable interest in most cases).

One could also generate graphs using the entire set of URLs, tags, or users. Again, many such graphs are possible; here are some that answer what seem to me to be interesting questions:

After looking at all this, I realize that my previous post was a bit muddled on why Clay’s graph bothered me. The problem isn’t that the users were ordered by decreasing tag usage (also done above), it’s that the Y axis represented the number of tags ever used instead of the number of taggings performed. This makes the “area” under Clay’s curve in the long tail difficult to define, and so it’s hard (I think) to find a question that it answers. Or who knows, maybe if I look at it tomorrow, it’ll make perfect sense.

So, anyone out there up for making these graphs real…?

In a system like del.icio.us where each person can tag a URL, value is created along two dimensions: the user who tagged it has presumably fit it into his/her own personal category scheme; and the aggregated tags of many users assign new semantic properties to the URL. Feedback can then be created between these dimensions via an auto-complete functionality (e.g. John Resig’s extension) that, at the moment when a user assigns a tag, displays tags assigned that URL by other users and/or algorithmically “related” tags based on what the user types.

Secondly, the fact that any user can assign a tag to any URL means that many more URLs can be tagged (at no cost) as compared to an expert categorization scheme, and that many more (weighted) meanings can be assigned to each URL. The tagging of obscure URLs addresses the “long tail” of URLs, while the aggregate “tag profile” of a given URL addresses the “long tail” of perceived meanings.

Clay’s essay covers these points in clear prose accompanied by some really helpful charts and graphs. One very minor issue I have, though, is with the “Tag Distributions” chart. Clay refers to “the characteristic long tail of people who use many fewer tags than the power taggers.” While this chart does exhibit a “long tail,” this is simply a result of the fact that the users were ordered by decreasing tag usage (also true of the following three charts) — the X axis here doesn’t represent a value, it is just a sequence of users.

The phrase “long tail” usually refers to the observation that for many distributions, the number of elements with outlying values (the “tail”) may be cumulatively significant compared to the number of elements clustered near the average. Clay might have not even been using the phrase in this way, but once a buzzword gets going, it’s best to use it as conservatively as possible (otherwise people start getting pissed off!).

Some “long tail” charts that would be interesting to see would be URLs by number of times a tag was assigned (showing whether the long tail of obscure URLs cumulatively comprises more tags assignments than the common URLs), or tags for a specific URL by number of times a tag was assigned (showing whether the long tail of obscure tags cumulatively comprises more tags assignments than the common tags). This last chart could also be averaged across many URLs to see if this long tail applies in general to arbitrary links.