open data Sharing Economy Transparency



In the absence of open data standards, companies like Airbnb can define their own terms for behaving in an “open and transparent” way.

  • In early December, Airbnb made headlines by releasing some data on how people are using the company’s platform in New York City.

    In doing so, the company has provided an object lesson in the critical role that data plays (and will continue to play) in government regulation of private companies in the 21st century, and highlighted how ill-equipped governments are to obtain and use this data.


    Over the past year, the use of Airbnb to rent properties in New York has received intense scrutiny from government regulators because of suspicion that a non-trivial amount of rentals listed on the site were in violation of state or city rules. In late 2014, New York State Attorney General Eric Schneiderman released a report examining Airbnb rentals in New York that concluded that “most short-term rentals booked [through Airbnb’s service] in New York violate the law.”

    The recent data release by Airbnb was meant to make good on a promise by the company in response to the Attorney General’s report to be “open and transparent,” and to underscore its contention that the vast majority of the users of its service do so in compliance with state and local laws.

    From the start, the release of this data was viewed with skepticism by some journalists, activists, and others that closely watch the “sharing economy.” In order to view it, interested parties needed to make an appointment to review the data in person at Civic Hall, where Airbnb is an organizational member. The data was highly redacted, was not published to company’s data website, and those viewing it were not allowed to copy it to review it more closely after their scheduled appointment.

    In the weeks that followed the release, outside reviews of the data seem to contradict Airbnb’s primary claim that most of its hosts are using the service lawfully. Others have cited the need for more detailed information to draw definitive conclusions. Though Airbnb has indicated that it would like to share similar data on usage for other cities, there is no indication that the company will release more detailed information for New York City.

    It seems clear that the ultimate question of whether Airbnb’s service is being used lawfully will be determined by examining data on how people use the service. State regulators and others know this and have used legal and other means to try and get this data. Airbnb also knows this, and it has carefully constructed its limited release of data (and its public outreach about this data release) to assert that they are in compliance with the law.

    But if data is at the heart of government’s ability to regulate Airbnb and other “sharing economy” companies, why are they so ill-equipped to obtain and use it?


    At its core, the sharing economy—whether defined by the example of UberAirbnb, or one of the other standard-bearers of this emerging class of business—represents a change in the way that people consume goods and services that is enabled by advances in technology. This change in consumption results in a challenge to an existing, entrenched market actor (for Uber, the taxi industry; for Airbnb, the hotel industry) that is subject to existing government taxes and regulations.

    The challenge for public officials and regulators is to avoid stifling new innovation that can result in better services but at the same time to ensure the fair and equitable application of rules that have been set up to regulate business operations and protect consumers. This is an issue that can have significant political implications. Striking the proper balance between competing needs can indeed be tricky.

    But this challenge is not a new one for governments—tension between regulators and private interests spans the history of collective governance.

    Supporters of the sharing economy commonly criticize existing government regulations as outdated and ill-suited to support innovation by 21st century technology companies. Reputation systems are often pointed to as a 21st century alternative to traditional government regulation, to ensure that sharing economy firms and similar types of companies act in the best interest of consumers.

    But there are a number of common examples where consumers engage in transactions for which they have an abundance of reputational information where the company providing the services is also highly regulated by government. Consider the example of dining out at a restaurant—never before have consumers had as much information as we do now about the quality and cleanliness of food service establishments. And yet, restaurants and food service establishments are highly regulated by multiple levels of government.

    In addition, recent research specifically examining reputation systems used by technology companies suggests that reputation systems and more traditional central regulation can work beneficially in tandem. A recent study from the Ohio State University examined the reputation system used by eBay in conjunction with a buyer protection program—a centrally managed program to provide protections to buyers and recourse if they are dissatisfied with purchases (an approach strikingly similar to the notion of centralized regulation). The study concluded that:

    “[W]e estimate that the total welfare rises by 2.9% after the introduction of the buyer protection program. This increased welfare demonstrates an efficiency gain by having the two mechanisms, the eBay Buyer Protection and eBay Top-Rated Sellers, in place.” [Emphasis added]

    Sharing economy supporters also claim the the growth of companies like Airbnb, Uber, Taskrabbit and others has led to a fundamental change in the economy, with more people opting to become freelancers who—to paraphrase the words of sharing economy companies—become masters of their own destiny. But another recent study from George Mason University suggests that the freelancer phenomenon began long before the advent of sharing economy companies like Airbnb and Uber:

    “Our data support the claim that there has been an increase in nontraditional employment, but the data refute the idea that this increase is caused by the sharing-economy firms that have arisen since 2008. Instead, we view the rise of sharing-economy firms as a response to a stagnant traditional labor sector and a product of the growing independent workforce.”

    The George Mason study helps to clarify the role that sharing economy companies can play in a changing economy—as opportunities for traditional employment become scarcer, the sharing economy may play an important role in providing employment for a changing workforce.

    But we should not allow these potential benefits to be offset by other negative consequences that may arise if sensible regulations are not applied sharing economy companies.


    The exercise of regulating private interests by government almost always involves an information asymmetry. Governments seek to discover the occurrence of specific activities that are subject to regulation and to apply relevant rules and taxes to those activities.

    Prior to the digital age, governments sought to (and still seek to) address this asymmetry by hiring groups of trained employees like auditors, inspectors and agents. The job of these individuals is to investigate certain kinds of activities and transactions to determine if the activity in question falls under the authority of a specific regulation or regulating entity, and then to enforce any applicable rules or taxes.

    With the rise of the internet and the dawn of the digital age, governments have employed a wide range of new tools to help ensure that the activities of private interests comply with rules that have been adopted by elected and appointed bodies. The steady march of technical advancement has also made compliance with government mandated taxes and rules easier and more efficient for businesses than ever before.

    The tension between how technology alters production and consumption patterns, and how these new patterns square with existing government rules is not new.

    In the 1960’s, the rise of mail order retailers began a protracted debate on the application of state and local sales taxes to remote sales—a debate that has raged through the time when internet retailers like Amazon developed and flourished, and that still rages today.

    In the early 2000’s, a new class of business—bolstered by the increasing availability of broadband internet access—began to offer consumers new options for telecommunication services that bypassed the Publicly Switched Telephone Network. In a scenario that is remarkably similar to the sharing economy, these new VoIP companies competed against large entrenched industry incumbents that were heavily regulated by government by offering customers improved service, a better customer experience, and lower prices.

    The tension between the new service being offered to consumers VoIP companies and existing government rules was ultimately resolved by an order from the FCC.

    So while the tension with existing tax and regulatory requirements created by technical advances is not new, and existing government institutions have shown they are capable of resolving these tension to the benefit of consumers, things are a little different when we consider the companies that make up the sharing economy.

    Sharing economy companies self identify as “technology companies”—not dispatch companies (in the case of Uber) or hoteliers (in the case of Airbnb) that happen to make heavy use of internet technologies. They position themselves not as providers of a service, but as enablers or connectors that bring together individuals that want to transact with each other.

    The issue with respect to government regulations as they relate to sharing economy companies is not so much that existing regulations are outdated as some have claimed. Instead, it is that the infrastructure for ensuring compliance with these regulations was not constructed for the 21st century—or, at best, the infrastructure has been minimally and very unevenly built.

    Government regulations need to speak the native language of these companies—in the 21st century, this language is almost always data in JSON or CSV format delivered over HTTP.


    Consider Airbnb’s recent data release—what if the State of New York, as part of its request to the company to share data, had offered Airbnb the ability to publish data to its open data portal?

    The state could have provided the company with a user account and requested that they publish their data using that account at regular intervals to allow for scrutiny by regulators and other interested parties. They could have provided Airbnb with existing guidelines for ensuring privacy as well as metadata guidelines to help ensure data quality.

    Even if Airbnb didn’t opt to publish data to the state’s open data portal, the request to do so would have helped to better qualify the deficiencies in the data the company actually did release. The state’s portal already contains scores of data sets with detailed information from dozens of agencies. Are technology companies like Airbnb less equipped to publish data on their core business operations than the State Liquor Authority? Really?

    With very few exceptions, government open data portals are one-way vehicles—transporting data unidirectionally from government agencies to external consumers. Governments largely don’t view their open data portals as platforms for comingling data from different data producers, much less as vital instruments for successful 21st century regulation.

    But what if they did?

    With complete enough data, the question of whether an Airbnb host is in compliance with the law is fairly easy to spot. The issue is that in the absence of a standard mechanism for sharing this data openly, companies like Airbnb can define their own terms for behaving in an “open and transparent” way.

    We need to expand the way we think about open data so that it’s not just about agencies publishing data to an open data portal, but instead is an integral way that we collectively help ensure the health and stability of our communities. We need to expand our notion of “government as a platform” to go beyond just building new civic apps to helping ensure efficient compliance with rules that are adopted collectively through democratic processes.

    We’ve started to assemble some of the building blocks for this new infrastructure, but we now need to put the pieces in place.

    For example, some governments publish data on zoning rules as open data. But we need to go beyond simply publishing this data and expose these rules through an interface that allow them to be encapsulated in a transaction. This work has only just begun and is,for now,largely driven by actors outside of government.

    Imagine an infrastructure that would allow companies like Airbnb to instantly determine if a potential short-term rental was authorized under local zoning and rental regulations, and to determine if a rental tax was due on the transaction and the amount.

    We are woefully short of this goal.

    In fact, in jurisdictions that specifically require short-term renters to register with the local government, there is no interface that supports an automated check to determine if a specific property has a permit and is authorized to conduct such a rental. The absence of this essential infrastructure to enforce local regulations may go a long way toward explaining the dismal rate of compliance.

    It seems clear that governments will not be able to successfully regulate these new companies without the infrastructure necessary to do so.

    Building the infrastructure for 21st century regulation will require us to expand our ideas of open data and government as a platform. Checking for compliance with tax and regulatory requirements—by either party in a sharing economy transaction—should be as simple as making an API call.

    Constructing this infrastructure won’t be done overnight, and it probably won’t be inexpensive. But the stakes for state and local governments have never been higher.

Crowdsourcing International Development Transparency



"Citizens of Sarpallo VDC in Mahottari district waiting for the VDC office to open."

The money was supposed to buy bicycles for ten girls attending public school in Mahottari, Nepal. The head of the village government allocated 150,000 rupees—about $1,400—for the project, which he said would encourage girls from the marginalized Dalit (so-called “untouchable”) community to attend school. An auditor signed the paperwork and the money left the government’s coffers.

“Not a single student in the village got a bicycle,” says Pranav Budhathoki, the founder and CEO of the Local Interventions Group. Budhathoki received an anonymous report of the missing funds during the pilot of an anti-corruption project he is looking to launch throughout the country of Nepal. He immediately sent his regional representative, a respected journalist in Mahottari, to find out what happened to the money.

Budhathoki is trying to curb corruption in Nepal by publishing information about local government budgets—about where money should have gone—and then eliciting feedback from citizens about what projects actually happened. In urban, wealthy contexts, people can use smartphones and apps to crowdsource information about corruption, but Budhathoki knew that wouldn’t work in rural, undeveloped parts of Nepal. So he came up with an alternative: People could submit reports of corruption using very simple tools—text messages and phone calls—and he would use technology on the back end to aggregate, analyze, and publish the data. This is the key to development, he says: not to increase funding or develop new projects, but rather to give citizens the information and the tools they need to demand accountability.

“The problem is not resources,” he explains. “The problem is citizens not knowing how much the government has allocated in their name.” To solve the problem of corruption, he says, it’s necessary for people to be “engaged and involved and reporting what they see around them, the malpractices and corruption issues.”

According to Budhathoki, the Nepali government sends money to Village Development Committees to address every imaginable social injustice, but “up to 60 percent of that allocated budget gets sent back to national coffers because the local governments don’t spend it,” he says. Local news reports corroborate that claim. As a result, Budhathoki says, “health centers are running without medicines and doctors, and kids are attending schools with no books or teachers.”

Budhathoki founded the Local Interventions Group in September 2011. Since then, he’s launched an app aimed at reducing police abuses and increasing police effectiveness and has used the open-source software Ushahidi to address cheating and violence in the 2013 Nepali elections and to ensure that earthquake relief services and supplies reached the right people after this year’s earthquake. His newest project tackles corruption head-on by publishing information about budgets and then crowdsourcing reports of corruption. He ran a pilot project last year in two of Nepal’s 75 districts, and, over the next two years, he plans to scale up the program so it will reach the entire country.

Here’s how it works: First, the Local Interventions Group’s regional representatives, all well-connected reporters, ask government officials how much money was earmarked for, say, schools or clinics in a certain area. Then, the Local Interventions Group crowdsources reports about what is needed, what is being done, and the quality of the work.

During a three-month pilot, Budhathoki solicited reports in each of three categories: absenteeism of local officials, misspent or disappeared development funds, and missing pension payments. At “Mobile Help Desks,” volunteers gathered information from text messages and phone calls. Budhathoki received 1,300 citizen reports. Then, the regional representatives verify the reports with local government officials and threaten to make the information public unless the grievance is resolved. Over the course of the three months, the Local Interventions Group verified about 600 to 700 genuine, specific, and actionable reports.

Take the bikes for female students, for example. Budhathoki’s contact in Mahottari took the report of the stolen bicycle funds to the district government. “Within three days, he got a call from the chief district official,” Budhathoki says. The Village Development Committee secretary admitted responsibility and promised to return the money. It took three more months, but “the 150,000 rupees was finally returned back to the national coffers,” Budhathoki says. No girls have gotten bicycles, since there’s a new government with new priorities in office, but the money is back where it belongs.

As Budhathoki scales up his anti-corruption project to the entire country of Nepal, he hopes to streamline his interactions with the government. His NGO will collect, vet, and verify grievances; map them onto a GIS platform; and then share the reports both with government officials and civil society organizations. That way, “each district official, head of the village government, [will] see these reports at his desk, on his computer, early morning every day so that he can talk to his colleagues on the ground and do something about that,” Budhathoki says.

Several researchers who have investigated anti-corruption programs say such collaboration is the key to success. Carla Miller is the founder and president of City Ethics, a nonprofit that aims to help local governments develop programs to prevent corruption, and has prosecuted federal corruption cases involving elected officials. She’s found that anti-corruption programs entirely internal to the government with no line to citizens can themselves be corrupted, while citizen groups can lose momentum and don’t necessarily understand the inner workings of government. “Ultimately, you have to have communication, collaboration,” she says.

The spirit of collaboration with government officials was behind Budhathoki’s decision to take down a map of crowdsourced corruption reports he’d initially published on the Local Interventions Group’s website. “We put it up for a month and then we shut it down because the government was getting antsy because they didn’t want citizens to know there’s a lot of corruption in the government agencies,” Budhathoki says. “Of course everyone knows that there’s a lot of corruption in government agencies at the local level but they didn’t want that to be up on the website for everyone to see.”

So, in the end, Budhathoki came to the conclusion that taking the map down was the right decision. “We’re okay with that because my approach has been collaborating with government officials rather than confronting them,” he says—and, now, he adds, he can walk into government offices and say, “Alright, let’s try to change something on the ground.”

This is a controversial approach. Some analysts, such as Miller, agree that taking down the map was the right call. “Maps of corruption and press and then a reaction from the government” is just a short-term solution, she says, that “doesn’t integrate citizens into long-term planning and identifying those issues faster.”

Yuen Yuen Ang, an assistant professor of political science at the University of Michigan, has looked at the platform I Paid A Bribe, which crowdsources reports of corruption in India, and similar platforms in order to identify what works and what doesn’t. She agrees that working with the government should be Budhathoki’s priority.

“Budhathoki made a pragmatic choice in agreeing to the government’s request—which, importantly, was a request, not a demand,” she wrote in an email. “By doing so, he avoids antagonizing the government, making it more likely for state authorities to work with his organization to take concrete steps in fighting petty corruption.”

“The purpose [of crowd-sourcing initiatives] is not to shame the government or individual officials, because, as Budhathoki himself points out, everybody already knows there is a lot of corruption,” she adds.

The Local Interventions Group isn’t combating corruption within elite circles, which Ang says would call for a political response, but rather what Ang refers to as “petty corruption”—that is, “low-stakes, diffused corruption that directly affects the lives of regular citizens.” Changing out government officials has no effect on small-scale corruption, Ang says, since it’s a systemic problem. Instead, the best way to approach the problem is through procedural changes: for example, “reducing bureaucratic discretion, centralizing budgetary management, reforming public compensation practices, etc.”

Budhathoki was able to make such a collaborative procedural change when it came to paying pensions. During his pilot, he found many people didn’t get their pensions on time, or at all. The government officials decided that, when the next month of pension payments came due, they’d have the Local Interventions Group’s regional representatives oversee the process, verifying who was owed a pension and who received one. That way, “we know how much money is being given to whom,” Budhathoki says. “The whole process [is] transparent.”

That’s the kind of change he hopes to continue to make in the future. “That happened because we’re seen as partners,” he concludes.

Campaign Finance Civic Hall Transparency



“There are thousands of potential front page of reddit stories hidden in 25 years of campaign finance data,” and Solomon Kahn wants to help unearth them.

  • In the hours he’s not working at Paperless Post or spending time with his wife and baby, Solomon Kahn is a crusader for campaign finance transparency. Earlier today, Kahn stopped by Civic Hall for a brown bag lunch to share the a visualization tool he built using OpenSecrets data that lets users dive into politicians’ campaign finance records, all the way down to individual-level donors, if necessary. Kahn is currently raising money on Kickstarter to help pay for hosting and to support outreach and training for journalists who can best make use of the tool.

    Using his own local congressperson, Rep. Joseph Crowley (D-NY), as an example, Kahn demonstrated how the visualization made it easy to see that Crowley went from getting 27 percent of his campaign funds from labor groups and 14 percent from finance/insurance/real estate in 2000, to getting 6 percent of funds from labor and 38 percent from finance/insurance/real estate in 2014. Digging a bit deeper, Kahn found that one of Crowley’s largest contributors is the Blackstone Group, and that in 2007, 14 people affiliated with Blackstone, who had never contributed to Crowley before, all gave $2,300 on the exact same day.

    Kahn says that users will be able to submit “scoops” like this which, once vetted for accuracy, will be displayed at the top of a politician’s page so that journalists and other interested folks will see notable facts like the above right off the bat. (An audience member asks if he’s thought about annotation, to which Kahn replies: “That’s complicated,” and reiterates the limits of his time and money.)

    Helping journalists make the most of the tool is one of Kahn’s priorities. “For a reporter in Sunnyside [Queens, New York] at the Tribune to get a story like this would just be impossible without this kind of tool,” Kahn tells those gathered at Civic Hall.

    “There are thousands of potential front page of reddit stories hidden in 25 years of campaign finance data,” Kahn writes to potential Kickstarter backers. He wants to make sure those stories see the light of day.

    Speaking of reddit, Kahn says he’ll be promoting the tool in a reddit I Am A—, Ask Me Anything session soon, so keep your eyes peeled for that.

    For more information, see Kahn’s Kickstarter video here: