From reuters.com, a piece I wrote on the rise of open data in arguably more interesting, or at least directly impactful, circumstances. In this version I’ve added some links to the source material where available.
Aug 28 (Reuters) – ‘Open data’ – the trove of data-sets made publicly available by governments, organisations and businesses – isn’t normally linked to high-wire politics, but just may have saved last month’s Indonesian presidential elections from chaos.
Data is considered open when it’s released for anyone to use and in a format that’s easy for computers to read. The uses are largely commercial, such as the GPS data from U.S.-owned satellites, but data can range from budget numbers and climate and health statistics to bus and rail timetables.
It’s a revolution that’s swept the developed world in recent years as governments and agencies like the World Bank have freed up hundreds of thousands of data-sets for use by anyone who sees a use for them. Data.gov, a U.S. site, lists more than 100,000 data-sets, from food calories to magnetic fields in space.
Consultants McKinsey reckon open data could add up to $3 trillion worth of economic activity a year – from performance ratings that help parents find the best schools to governments saving money by releasing budget data and asking citizens to come up with cost-cutting ideas. All the apps, services and equipment that tap the GPS satellites, for example, generate $96 billion of economic activity each year in the United States alone, according to a 2011 study. [PDF]
But so far open data has had a limited impact in the developing world, where officials are wary of giving away too much information, and where there’s the issue of just how useful it might be: for most people in emerging countries, property prices and bus schedules aren’t top priorities.
But last month’s election in Indonesia – a contentious face-off between a disgraced general and a furniture-exporter turned reformist – highlighted how powerful open data can be in tandem with a handful of tech-smart programmers, social media savvy and crowdsourcing.
“Open data may well have saved this election,” said Paul Rowland, a Jakarta-based consultant on democracy and governance.
CULTURE CLUB
Indonesia, home to 247 million people and some of the world’s largest Facebook and Twitter populations, has been a few steps ahead in embracing open data. It’s one of eight founding members of the Open Governance Partnership (OGP), a government-led initiative to free up data that now has more than 64 members.
The embrace of open data has had few tangible benefits, but created a buzz and fostered a culture that prodded Indonesia’s election commission to tweak the way it handles vote results.
“There was nothing in this OGP stuff that said you had to put up results from each village,” said Kevin Evans, a Jakarta-based governance consultant. “But it provides a culture where the commission says, ‘why don’t we try a bit of transparency?'”
While it was not allowed to speed up or ditch the manual tabulation of votes – where much of electoral fraud takes place – the commission provided equipment for tallies from nearly half a million polling stations to be scanned and uploaded to its servers, and from there to its website.
Those scans prompted some volunteers to start the laborious process of sifting through them to look for signs of fraud. As the country waited for an official vote count, the rival candidates both claimed victory, based on whichever unofficial poll suited them. Indonesians feared at best a stalemate, at worst fraud and a collapse in public confidence in the election process.
A THIRD WAY
This worried Ainun Najib, an Indonesian IT consultant based in Singapore. “I could see a situation where both sides claimed they’d won, based on their quick counts,” he said, adding he feared a quick descent into confrontation. “I saw we’d need a third alternative.”
Ainun reached out to two friends at Google, and between them they cobbled together a solution: a way to automatically download all the scanned files from the election website, and a website where volunteers could easily transcribe the key numbers from each tally into a spreadsheet.
This marrying of open data, programming savvy and crowdsourcing was the key. Within a few days – well before the official result was available – 700 volunteers were able to tabulate more than 90 percent of the vote on a website that could be viewed by anyone.
It was decisive in convincing Indonesians that the election had been fair – a verdict upheld by the country’s highest court last week.
By short-circuiting the long and fraught manual tallying process, it played a “very important” role in restoring public faith in the process and deterring fraud, said Marcus Mietzner, associate professor at the Australian National University. “It allowed the media and citizens to check whether data was manipulated as it travelled from the polling station to the centre. This had never happened before.”
At a ceremony honouring inspiring young leaders last week, Indonesia’s President-elect Joko Widodo thanked Ainun and his colleagues for helping make the election more transparent.
HASHTAGGING A TYPHOON
It’s not the first time open data and crowdsourcing have been used in elections, but it’s probably the most decisive. And it hints at the potential if officials, geeks and others play their part in leveraging open data to take on the big issues of the day.
“The idea is we can see the data of the state and do things with it, and that expertise doesn’t just live inside bureaucracies,” said Tim Davies, a researcher at the World Wide Web Foundation who is compiling a report on open data in developing countries. “These are powerful ideas and important ones for building new visions of how we do governance.”
It might not just be politics.
Neighbouring Philippines, another founder member of the OGP, combines open data with social media and crowdsourcing to minimise the impact of typhoons and storms that ravage its shoreline. Twitter users are encouraged to label their messages with specific hashtags, for example, making it easier for relief officials to quickly identify those who need help.
“That’s been a very big deal,” says Patrick Meier, who works with UN and Red Cross organisations on using crowdsourcing and open data in crises, and is now working with Manila-based start-up Rappler to better predict where help might be needed.
POLITICISATION
But Indonesia and the Philippines are outliers in Asia.
The Asian Development Bank website, for example, contains a single reference to open data, against the World Bank’s more than 85,000.
Singapore, despite describing data as a “natural resource”, has been cautious, and it has taken open data evangelist Daryl Arnold a couple of years to help persuade officials and executives to free up significant chunks of data for programmers to build apps around.
He points to dozens of apps as evidence of progress, including a recent ‘hackathon’ where programmers explored 65 million rows of data released by port-related companies and agencies to figure out how to make Singapore’s port more efficient. One problem, he said, was that despite technological advances there is often an 8-hour delay between when a ship is due to arrive and when it actually berths – disrupting the mini-industry of tugs, cleaners and caterers that serve such vessels.
While Singaporean agencies and businesses warm to the idea of freeing up data, Arnold cautions that it must be used responsibly. “It’s still critical that people use it in a respectful fashion,” he says.
Indeed, the very success of open data in Indonesia’s election may give pause for thought among more conservative countries of Asia.
Waltraut Ritter, co-founder of Opendata Hong Kong and a researcher on the knowledge-based economy, reports a tension over open data in some less developed countries, where officials worry advocates would focus solely on issues like corruption.
“In developed countries, data and information is seen more as an ingredient or commodity for everything you do whereas in Asia everything around information and data can easily be politicised,” she said.
“There’s so much more baggage in terms of about how people think about information.” (Editing by Ian Geoghegan)