Help open the Big Apple

Easier to eat in slices. Apples photo by Craig via Flickr.

Delicious data inside. Photo by Anabadili via Flickr.

The effort to open municipal data is an initiative with momentum. Inspired in part by the transparency mandate on the federal level that gave us the first ever White House CIO and, cities across the country are opening up. One city in particular set the scene before this all hit the national stage: Washington D.C. has delivered precedents like the first online city data directory, the first open API for 311 service requests, and open invitations for developers to produce apps with city data using initiatives like Apps for Democracy.  D.C. has laid the groundwork for a national model by delivering some killer apps. Previously the CTO for D.C., Vivek Kundra is now the national CIO. This model has not yet been fully embraced by New York City, but it’s getting close. In fact the New York State Senate Office of the CIO has been paving the way for opening government data on the state level.

Historically a trend setting city, New York is trying to catch up and even exceed precedents for opening city data. A City Council bill sponsored by Gale Brewer, Int 991, mandates that every city agency will make its data easily and publicly accessible in raw structured digital form. As a mandate, this legislation would be the first of its kind. Even in D.C. there is only a general policy to release data, not an official mandate. At the public hearing for the bill, a plethora of supporting testimonies were given from the likes of the NY State Senate CIO, the W3C, and the Software Freedom Law Center. While a representative from the Mayor’s office stated that the Mayor supported the spirit of the bill, he expressed that it was too comprehensive and expensive to be practically possible in its current form. Int. 991 is set to be voted on in early fall.

At the Personal Democracy Forum just a few hours before the hearing, Mayor Bloomberg announced the creation of NYC Big Apps, a contest modeled after Washington D.C.’s Apps for Democracy that would let outside developers create applications on top of city data. Given some of the varying approaches to opening city data, the next day Vivek Kundra offered advice to help direct the effort in New York City.

According to the announcement for Big Apps, 80 of the city’s datasets will initially be made available. The city is currently seeking requests for expressions of interest (RFEI) with a deadline of September 1st to determine which datasets should be a priority to open. One problem with access to city data is that it can be difficult to see what data is useful without knowing what data is available. It can also be difficult to know what applications would be useful to build without knowing what New Yorkers want.  To help gauge interest, Peter Corbett of iStrategy Labs created a simple public venue to let people add or vote for the apps they’re interested in. If you’re interested in submitting a full RFEI, the Open Government NYC group has created a process for doing that while keeping it in the public record. The task of showing what data is available is more of a challenge.

New York City doesn’t provide a public directory of the data that each agency maintains. Much of this information used to be available online and the city once even published a full Public Data Directory. Unfortunately, the first and last version of this directory was published in 1993, but it may still provide some utility.  Much of the the city’s IT infrastructure is slow to evolve, so many parts of the 1993 directory are likely unchanged. Regardless of how current it is, the 1993 directory offers a solid framework for developing an up to date version.  Open Government NYC has an effort underway to digitize the full directory, allow the public to update it, and submit improvements to the soon to be launched National Data Catalog, a citizen maintained version of With the help of Eric Mill from the Sunlight Foundation’s Sunlight Labs, the challenge of digitizing the NYC Public Data Directory has been broken into small tasks that anyone can contribute to.

One other caveat to opening city data is that some of the agencies perceived to be run by the city are actually run as state agencies. The MTA for example is a quasi-public private enterprise overseen by New York State. While many of the most wanted apps relate to MTA transit, neither the City Council legislation nor the Mayor’s Big Apps program are able to provide MTA data. Because of some of the challenges of accessing and using public transit data, TOPP Labs hosted a New York Public Transit Data Summit to help develop clear and mutually beneficial policies for the relationship among transit agencies like the MTA, their riders, and application developers.

In many cases, the challenge of opening city data is coming up with a sensible policy that sets terms which ultimately provide superior city services. The task of drafting these policies is something that requires a proactive collaboration between developer communities and city government. With sound policy, the challenge then becomes for the city to provide the data in a well structured and timely manner. This too is a place where developer communities can contribute, even if it’s just by doing a task for digitizing the Public Data Directory.  Technology companies can also contribute. Google could help digitize the city’s 6 billion pages of documents much like it has been digitizing the world’s libraries. Amazon has already offered to host public datasets for free. What data would you like to see, what applications would you like to use?

With your help, we can open up the Big Apple.

If you’d like to suggest applications or data for the city to make available, you can add and vote for them at To submit your own RFEI and make it public, you can add it to To help update the Public Data Directory you can first help digitize the old version