Farm to Market

Aquifers Feed Community Development Beyond the Garden

Healthy civic engagement is often based around an emergent collective influence, a momentum from the coalescence of many individuals’ activity. As developers of civic software this is a concept we think of often and try to embrace, especially within the context of a dense urban environment. Such environments are made up of complex systems of networked activity, multitudes of input and output. Like the internet, these systems are really networks of networks. In fact, one might view a city as a macrocosm of its own biological systems – the nested ecologies and individuals that comprise it.  Viewed from such a framework, a biological system as advanced as the human body provides an excellent example of emergence in the way our senses are brought together into consciousness and perception of the world. Bringing the metaphor full circle, one might ask: What if we could bring consciousness to a city?

Now I don’t mean to suggest collective consciousness, but instead by analogy I’m alluding to a pragmatic unification of information about the activity within a community. This of course has been the aspiration of many for quite some time and it’s approached in various contexts at different scales: Everyblock/Outside.in, The Facebook Platform, The Open Stack for the Web, The Semantic web, the Noosphere, Kurzweilian Singularity, Lolcats, and so on. So how exactly does one go about collecting all the data of activity in a city and turn it into a cohesive and meaningful stream of intelligence? Get all of the data and filter it. Trivial? Not quite, but bear with me: What turns consciousness and human perception into intelligence is the careful filtering of an overwhelming abundance of sensory input and cognitive activity, finding meaning in the data. With the amount of information noise that’s been introduced in the world today, robust filtering must be a fundamental function of the social web and wired communities in general. Otherwise we will be mired in the fog of infoglut, salad sentences of schizophasia, silty streams of consciousness, mixed metaphors, overwhelming alliteration, annoying hyperbole, and the list goes on. This, I believe, is where a friendly little app called Melkjug has potential to change the world.

Melkjug can be used to curate and highlight content from a variety of sources.

Melkjug filtering content from a variety of sources.

Melkjug is a news feed reader like Google Reader, but focused on filters. Plus, it’s open source! Melkjug lets you tune your reading experience with a wide array of filters, avoiding Google Reader’s inbox model so you don’t have to stress about those 2493 unread articles. It also lets you collaborate with the Melkjug community by using the filters and filtered content produced by others. In the near future Melkjug will also employ true collaborative filtering where the preferences of others in the community can seamlessly act as a filter for you.

Since Melkjug is built around the Atom and RSS standards it can be used with anything publishing a feed, not just the feeds that we typically think of as news feeds. Another type of feed is an activity stream, a feed of a person’s online activity, as you would see aggregated by applications like the Facebook News Feed or FriendFeed. As you can see in the sidebar, this model has already been employed on this WordPress-powered blog by simply using a WordPress plugin that imports a Melkjug feed which is filtering and aggregating a variety of relevant activity sources – Twitter, Delicious, MediaWiki, SVN, Trac, etc. Activity streams (examples) are currently undergoing a standardization process where the grammar of each action is clearly defined. In addition to increased interoperability across the web, a standard with articulate grammar will make it even easier to filter activity streams with high fidelity.

What’s exciting about using Melkjug with activity streams is that it functions as both a muxer and a demuxer, it both combines and separates. This property and the way that Melkjug allows one to tweak their signals and EQ to get some cool feedback effects made me realize that the technology could be even more useful if used for both pre-processing and post-processing of feeds. For example, instead of advertising just the feed Flickr provides of your photos,  you could provide some custom preset feeds, then when you import those into Facebook you could further hone in, filtering further as needed. I see several advantages to distributed filters: they could simplify the user interfaces for filtering by keeping the appropriate UI in the most relevant context, they might help load balancing or caching on servers, and they might even help develop standard UI conventions for content filtering throughout the web. I don’t mean to suggest feeds should always be pre-filtered, after privacy has been taken into account raw data feeds should be available, but often it’s nice to have some well tuned presets to choose from.  A distributed input/output filtering model might also help us to consider the nebulous relevance issues and the sensitive privacy issues within the same framework. Bring on the OAuth.

The since removed Facebook News Feed filters.

The since removed Facebook News Feed tuners.

At this point in time, any discussion about activity streams is incomplete without referring to Facebook, as it is perhaps most responsible for creating and disseminating the concept of activity streams as we know them today. Activity streams are also an important part of Facebook’s success. The Facebook News Feed has significantly evolved in a short period of time and it’s been through several inspiration-innovation feedback loops with services like Friendfeed, Twitter, Tumblr, etc.. But the most recent Facebook “redesign” fundamentally re-engineered many of the activity stream concepts Facebook had pioneered. These concepts include publishing more ambient/implicit activity, using a relevance algorithm filter (with an equalizer fader UI for filtering similar to Melkjug) and some might argue – a well defined layout and visual hierarchy that made seeing a lot of information reasonably easy to digest. Since the default News Feed now contains every item from every contact, it can be somewhat difficult to get a grasp of what has happened in the past day – a single screen of the feed might only shows a few hours of activity whereas before the relevance algorithm typically filtered it down to about a day. The filtering options that we are now left with are somewhat subtle, fragmented throughout the interface, and overall provide few options. There are also a number of options that Facebook entirely removed with their redesign. These include the feed source equalizer UI which gave more flexible control than the current linear ordering of feed sources. I’m really not sure why this was so hidden before and why it’s now been entirely removed, it came across as a compelling feature. Also gone is the ability to edit the prominence of certain items in your feed – there used to be a summary versus detail option. Until recently Facebook had even taken out some of the implicit activity such as people tagged in photos. The filtering options that remain can be summarized as 1) creating groups of (whitelisted) contacts 2) filtering by a particular app/feed source 3) ordering these groups/sources and 4) creating a blacklist of contacts. For anyone who is subscribed to multiple mailing lists and manages a high volume of email traffic, it’s easy to understand the potential frustration of being limited by such a small set of filtering options. After all, filtering, whether by some well tuned algorithm like PageRank, personalized settings, recommendations, or editorial control, is the only way we’ve ever been able to make meaningful sense of the nearly infinite information available on the web.

After critiquing the current Facebook News Feed, I should acknowledge an important non-tech point regarding its development. Initially, the feed was perceived as being forced upon a community that really wasn’t asking for it, including many who vehemently opposed it’s introduction claiming that it was akin to a tool for stalkers. Yet quickly the complaints fizzled out and the News Feed ended up creating a dramatically broader culture of engagement and openness on the site. Society as seen through Facebook has stepped up to a new level of transparency and open dialogue. Yet even while many technologists and early adopters attempt to share absolutely everything online, it must be acknowledged that Facebook has largely been allowed to succeed because of its rigorous privacy settings. Some people have no qualms with radical transparency, but many more live within multiple communites that don’t always mix well (eg family/friends/colleagues). So again, the importance of balancing the privacy/relevant-openness spectrum speaks to the need for a unifying framework to filter both the input and output of our social data.

What’s also compelling about all the filtering metaphors (equalizers, faders, feedback loops, signal-to-noise ratios, etc) in regard to society is that they allude to the sounds of many mixed down into one and the rip-and-remix ethos of free culture. It could be argued that activity streams demonstrate an increasing openness of ideas in our culture throughout multiple mediums.

Similarly, with the advent of organizations like the Sunlight Foundation and people like President Obama, Aneesh Chopra, and Vivek Kundra in the White House the open flow of civic information will be quick to fill the airwaves, if not flood the activity streams.  New fire hoses such as these again present a challenge:  I’m really only interested in participating in the most  relevant interactions with our civic system – citing problems and suggesting solutions for my place in this democratic society.  Without fine tuning the filters on my neighborhood activity streams with a tool like Melkjug I don’t think even hyperlocal sources like Everyblock or Outside.in will be very relevant to me. In fact, without smart tools like Melkjug I’ll likely filter everything out.

Note: Neither Melkjug nor the activity streams standard have reached their 1.0 release, both are still in development. Melkjug does not currently provide any features that specifically target activity streams as that was not its original intent, but that didn’t stop us from creating a jug of our feeds from other sites for the activity stream you see on this site.