Thursday, January 26, 2006
The Problems With 2.0, pt 345314
I luv Memeorandum and all it's reconstructor cousins. It's one of the first things of my reading list. It's hugely slashed my search costs in finding new stuff.
But there's a problem. Ever since I've started using it to the point where it replaces many of my other sources, I have gotten stupider.
I can feel it - I don't think as fast, flexibly, or freely.
This is a well-known phenomenon in psych and econ - I've been locking myself into a diet of reinforcing information. Nothing really challenges my beliefs, and so I get hyperpolarized, or echo-chambered, sure - but the deeper effect is that I also get stupid, fast.
Part of the reason is that all the attention markets, reconstructors, etc push all the same stuff to the top - they all converge to the same equilbirum. Paradoxically, it's an environment of incredible diversity, but incredible sameness at the same time.
But the larger reason is that none of these reconstructors are really broad enough yet, and so reconstructing microchunks into coherent wholes is still pretty shabby. This is the where discontinuous value will be created - and surprisingly, the market gap is still open.
The good news is that the solution, I think, is fairly straightforward - reconstructor guys, if you wanna chat, ping me.
I know what you mean. I have a feeling these kind of mechanisms might work best when a very deep pot of content or whatever is being filtered is available, and alongside the organised picks we see now, there should also be a bunch of tangents, a few long tail hops. I'm not sure how that might manifest itself tho.
Umair, I would even agree, depending on the meaning of "many". If you're getting alot out of reading sources that aren't covered well on tech.memeorandum, then clearly you should continue reading those sources. In fact, I never hype memeorandum as replacement for anything.
An interesting fact is people will notice the same effect even when using personalized "filters", though I guess that's a topic for another post...
I remember reading a critique of Pattie Maes's Firefly--and I suppose all collaborative filtering by extension--several years ago.
The point made was that after a short learning curve Firefly provides users with a lot of value. However due to the mathematics of feedback loops and lorenz attractors, that value rapidly drops on both individual user and community levels. And the system eventually stops revealing new stuff.
(if you're into math see http://mathworld.wolfram.com/LorenzAttractor.html for info on lorenz attractrs)
I supposed this is what you're experiencing with memorandum (and how FoxNews manages to stay in business).
One solution is to introduce a level of randomness. But this may not be sustainable as users learn to recgnize the random elements and overlook them.
It a very interesting design problem.
microchunks is correct. When Google, Apple, Yahoo, Sony get most of the attention in blogs and IBM, SAP, Oracle, EDS, Verizon get little attention that's what you end up with. The enterprise technology market may be boring but is by some estimates 10X bigger than the consumer side - lots more waste too and more to write about if people did...
I've noticed the "getting stupid" results myself after heavy following of Memeorandum. I think it's largely due to the limited number of topics important enough to rank on their pages.
There seems to be less of an effect when using digg, but I tend to stay off the front page there, as the topics that make it there commonly seem to be of little interest to me.
If you want to get into the technical aspects of it--
Recent research http://scholar.google.com/scholar?hl=en&lr=&q=cache:p9Q4rBsUAU0J:arxiv.org/pdf/cs.CY/0511005+%22egalitarian+effect+of+search+engines%22 seems to indicate that search has a reverse effect than you would assume, because encourages a higher volume of consumption: "in spite of the rich-get-
richer dynamics implicitly contained in the use of link anal-
ysis to rank search hits, the net effect of search engines on
traffic appears to produce an egalitarian effect, smearing
out the traffic attraction of high-degree pages. Our empir-
ical data clearly shows a sublinear scaling relation between
referral traffic from search engines and page in-degree. This
seems to be in agreement with the observation that search
engines lead users to visiting about 20% more pages than
I suspect that aggregators follow the same patterns and encourage even greater consumption than the input-required world of search terms. If you feel dumber it's probably a result of the content itself and not the aggregator.
As for the mathematical aspects of aggregation and search, I agree with niblettes
that the value of search and aggregators drops as the number of inputs (web pages) and community (user feedback) grows. Instead of following niblette's
suggested attractor model, I think the decreasing value is due to the nonlinearity of content contributed a big community (like digg's front page) which results in the creation of unstable bifurcations and the onset of chaos (http://mathworld.wolfram.com/Bifurcation.html). Nonlinearity may work stylistically for Tarantino, but it's hard to follow a newspaper if you can't tell the difference between ads, classifieds, and articles (...also why anthropologists must devote so much time to what they do).
Yes... I agree. The cool thing is that this is a major problem I intend on solving with TailRank. In the next week or so I'll be rolling out some features which I think will fix this so stay tuned...
If you have any ideas feel free to email me... I'll also give you early beta access if you want.
I think an often overlooked piece of del.icio.us that provides value in this area are the popular tags. The more obscure the tag, the more obscure the posts. It only takes a few users to tag something 'playlists' to show up on the http://del.icio.us/popular/playlists page.
Echo chamber tags tend to generate echo chamber results, but you always have the choice to use a more creative tag.
Umair, always intrigued by your blog. I always anxiously await reading what you have to say.
I would be very interested to hear your ideas as I've been playing around with the creation of a politically-oriented reconstructor and one of the main concepts I've been struggling with is how to fight that echo chamber effect in an already echo-chambery environment. My gmail address is marcslove.
chuquet.com seems - to me - to converge on different points to memeorandum and tailrank, even though the basic principles are the same.
Perhaps it's a combination of the sample set and the algorithms. I don't know.
I feel it too! I am not so sure it is about the depth of presented information, or the number of topics. I simply think that the lack of effort in acquiring and digging out interesting information is what dulls our senses. When you are browsing through different sources, you are constantly taking into account the bias of the source and how it relates to the story. You are also presented with stories of little apparent value that may not be picked up by aggregators but could be gold to your personally. All this is removed when you're just passively scanning stories.
At some point we realize that some of our information consumption has to be a "minority game".
The winners will be those who know things that other people *don't*. That's what makes you valuable to other people. Being an expert in the top ten most known topics makes you worth what?
By definition, there's no algorithm which can give this to you. You need to cultivate your own tastes and ideosyncrasies. Your outlier blogs.
Don't worry about being the first to follow the fashions. Something which is number one on memeorandum or digg today (and is actually important) will probably turn up on one of your regular blogs within a week. So decide how you'll trade-off your own personal timeliness for personal individuality.