A trade journal of a still-emerging field, written by Adam Tinworth.

Archive for

Dr Laura James

Dr Laura James, foundation coordinator of the Open Knowledge Foundation

What is open data? asks Dr James. They have the Open Definition. Open data is open. You can use it whatever organisation you’re in. Data? Not personal data. There shouldn’t be any personal privacy issues (most of the time).

We have lots of data, and powerful IT systems. To many eyes, all data problems will be fixed. Data needs to be open to be interwoven. It doesn’t need to be semantic – it can be – but it does need to be well enough described to weave it with other information. You’ll need a range of skills: developers, designers, ethnographers, to get the most from the data. But open data means we can scale.

We’re in really early days for this. It’s still new in government – it’s barely heard of in business. There are whole new sets of activities – like data wrangling – that bring new costs. Many of those costs are because we’re retrofitting open onto existing data.

This is scary disruption. We get to experiment, take risks and occasionally succeed.

Open data leads to more sustainable cities. You understand more about what’s happening – so it’s easier to get all kinds of organisations to work together to solve the problems.  Who can make the city data useful to people? Startups, SMEs, schools, arts groups, libraries – open data brings them all together. The UK government now shares transactions over £25k monthly. Before they released this data, civil servants couldn’t access it. Now they can – so civil servants can see where better deals are to be had.

But it’s not a magic potion. It’s got to be used  – it needs individuals and organisations to build apps and services to allow it to be accessed and used. We need to collect advice around using it. And if it’s open data – you want open tools. And that means open source software. Select a robust open source software project, and you have a sustainable project. You don’t need to worry about a propriety vendor putting up prices or going bust. The code is free – services probably won’t be. That’s fine. Pay more, get better service.

We went from hand-writing html to content management systems, like WordPress. Now we need to go from hand-managed data to data management systems. And there will be a whole range of them, from propriety to open source, from basic to expert. They’ll allow management, analysis, proven ace checking, data cleaning… They’ll be used right through the lifespan of that data.

CKAN – an open data management system. What’s it good for? Sharing, finding and using data. For example –, dataGM.orgmentioned earlier – was built on it. And it’s not just the Open Knowledge Foundation – you can download it yourself, work with other partners, etc.

Opening data can be good for your organisation. had a lot of downtime when they first launched – and the treasury kept phoning to complain. It was the best way of accessing their own information…

Ian Holt

Ian Holt, senior developer programme manager at Ordnance Survey

Two years ago the OS was asked to release mapping data as open data for the first time.

They have data, tools and a network, GeoVation, which is underpinned by their data. They offer location data for Great Britain which supports web and mobile access models by developers. Comes under the “very permissive” open government license. You can grab the data from OS OpenData. When it was first launched, all you could do was get the data. In the years they’ve been running the API, they’ve realised you need more than that – like examples of how people have used it, and forums for discussing working methods and problems.

The data’s a mix of contextual products – which look like maps – and more analytical data, like boundary lines.

Examples of use:

icoast – created a product with activities along the Dorset coast.

Free iPhone app for Winchester Hat Fair. – game using the data

Most used data sets: OS Street View, OS VectorMap District Raster, and OS VectorMap District Vector.

The analytical stuff is accessed less. How can they encourage more use? Or is it a phased thing? When another data set is released, suddenly the others might become more relevant.

Biggest learning? Just releasing the dataset is not the same a getting people to use it. You need to communicate about it, and you need to provide tools. And data is not just for developers – you need to think about who else might want to use it, and how they will need to engage with it. For example, a wizard that allows you to build a map with markers on it…

You can incentivise people to use the data through global competitions. They’ve also run workshops called “Open Data Master Classes” – they encourage use of both OS and other government data sets.

GeoVation has a challenge process, which encourages people to pitch solutions to problems. The best ideas are invited to GeoVation camps, and the winners get money to produce their solutions. They’ve just closed a transforming neighbourhoods project. One around the Welsh Costal Path is running now.

Remember: publication is not the same as communication. And that goes both ways; they like to hear (and publicise) how people are using the data.

Enhanced by Zemanta

Emer Coleman

Emer Coleman, Deputy Director of Digital Engagement at the Government Digital Service

Some reflections from her experience in building the London Data Store.

Lesson 1 – it was never about data

The conversations are not about data – they’re about the threat it was to the public sector. It creates a totally different approach to governance, so you gave have mature conversations with the electorate, who have the same data that government does. You move from the tyranny of the experts to the wisdom of the crowds. But that’s uncomfortable for back office statisticians who have not been used to being in the public gaze.

Lesson 2 – the state cannot do this alone

Politicians are often open to data release – but officials aren’t. So she ended up as internal lobbyist, and also working with digital disruptors both inside and outside the business to get what they needed done. There’s also the Goverati – people outside the traditional structure who are prepared to pitch in at events like GovCamp 2011.  If the public sector can be less defensive, there are plenty of people to bring into projects. Some of those are real experts – like Jonathan Raper –  and can give you the information you need to push back against internal resistance. Get ugly early – release stuff quickly in the from people need, and iterate.

Lesson 3 – are we there yet?

People always ask about RoI. Developments aren’t linear – they’re emergent and unpredictable. People want to try and control things, but you need to allow a thousand flowers to bloom… You can’t predict what people will produce, from expert blogs, to apps, to one off activities to support people during city problems.

But the focus needs to move away from just the data cities produce. How does the state harness the data from social activity to improve the lives of citizens?

Lean Doody

Lean Doody, associate at Arup

10 years ago she did a masters at LSE  – then the only talk about IT and cities was about the death of place. Ubiquitous broadband would mean people could work from anywhere – why would they come to the cities.

Ed Glazer’s book: Triumph of the City: How Our Greatest Invention Made us Richer, Smarter, Greener, Healthier and Happier argues that they allow face to face conversations with a diverse number of people – with a range of skills. They enable large numbers of small company sot collaborate, and provide an excellent experience of space – both in safety and in the range of spaces available.

How do smart cities contribute to this?

  1. Economic development
  2. great places to live and work
  3. Growth in the ecological age – they’re resource efficient

Report: Information Marketplaces, the new economics of cities. They wrote the report because there had been a lot of discussion of smart cities, but they weren’t seeing that translated onto the ground in cities. They wanted to try and help to fill that gap.

ARUP did a report with the C40 – the 38 largest cities in the world committed to making changes around climate change. Many cities are doing things like smart grid, energy metering, water metering and real-time transport information. That all yields data. But these efforts are very silted. They’ve developed a framework for viewing cities from Level 1 to Level 4 based on how integrated their technology efforts are. San Francisco is a 3.5. Most UK cities are a 1 to 2.

San Francisco has mayoral support for initiatives, and a CIO in place. The look strategically at the problems facing the city, and how technology might eb able to solve that. Congestion is one example. One big driver of congestion is people looking for parking space. So they put sensors in every parking space owned by the city – giving them real-time availability. And now they have variable pricing of parking, to manage demand.

Aarhus in Denmark launched a smart city program in January. No chairs at the launch event – they all stood for 90 minutes… “They’re vikings…”  They have a partnership between the University, academic institution, the mayor and local government. And they’re spending a year thinking about what they want the city to be. This is a bottom up approach, rather than SF’s top down.

Birmingham City council are looking at how they can better connect silos of information with technology, and create a more integrated system of thinking. It’s too easy to end up with multiple, unconnected tech projects. Can you use common infrastructure? Common apps? Where can you get efficiencies. All European cities are having to do more with less… Open data fits in positive externalities. Can we release data to let the public achieve economic aims with it.

Cautionary Tales: Sydney had an open hack day, and the winner built a bus arrival app. It was popular and successful – but it was withdrawn two weeks later, because the back end couldn’t cope. The city should have thought about that in advance. What will council IT have to put in place to support apps from open data? Developers are going to want SLAs around the data – and you could charge for that. When you procure a service from outside entities, what’s the contractual agreement around the data? Cities need to think about this.

The number of open APIs around cities has exploded in recent years. Access to public data is estimated to be worth around €27 billion across Europe.

Three steps for cities:

  1. Set a vision and metrics. Think about what you’re doing and how technology can fit on.
  2. Appoint someone with an overview and vision of what the city is trying to achieve. They need to be an informed client for vendors.
  3. Create an information marketplace, and the back end systems to allow it to work.

Drew Hemment

Drew Hemment, founder and chief executive of FutureEverything

The project started in 2009 and posed the question: how might cities evolve in open data environments? He’s not interested in efficiency and transparency – he’s a bit suspicious of them. They’re about control. He’s more interested in openness – and an ecology of open data that allows you to do cool stuff.

They targeted useful data, and showed that useful things could come out of it – which led to DataGM to free up Greater Manchester’s public data.

But they were faced with challenges – they didn’t want it to just be an inwards thing within local authorities. The innovation argument wasn’t taking hold, and a market wasn’t opening up. Do cities like Brighton and Manchester have the scale to build useful things of open data exchanges? Maybe not. Now they looking at CitySDK – an European project for an open data markets based on standardised civic data.

Does open data lead to an open society? Maybe. You need data literacy, so citizens can make more use of it. It’s the preserve of an elite right now. Data arts is one approach to this. It can help demystify data. Emoto – a data visualisation for London 2012.

What they’re working towards is a Digital Public Space. The Creative Exchange is a step in that direction.

If we don’t shape the future we want, we’ll get the future we deserve.

Bill Thompson, head of partnership development, archive development at the BBCBill Thompson

What is the nature of this shared hallucination we’re all about to engage with? Where the physical and virtual space merge… We’ve seen closed data cities. Facebook is one. Do we have a good vision of an open data city? What will that liminal space we occupy be like? We’re at the start of the process of building those cities.

We’re at the stage of identifying the swamp, cutting down the trees and putting logs into the swamp – to make a comparison with Venice.

Bill spent 15 years as a freelance hack, but he was seduced by an offer to help with the BBC’s archive. What’s the most they could get out of the stuff that the BBC forgot to throw away? He got to play with the archive – boxes of documents, records marked “not to be played”. The BBC was set up as a cartel between six manufacturers of radios, to sell more radios. The government required everyone to have a license for it, to regulate it – and then decided to nationalise it. It needed to be looked after. Since then the BBC has been there to act in the public interest – but the detail of how it hass done it has changed. Not everything has been kept – until about 1980 the BBC viewed the magnetic tape as more valuable than the programmes, and wiped them.

In that archive they may have footage of you as a child. Or of your parents. But can you find it now? No.

A lot of it is on paper – and there are now plans to have it digitised. There’s an enormous amount of stuff in the BBC no-one outside knows exists, so how can people ask for it? That’s his group’s job.

And the archive isn’t just “old stuff” – it’s everything that it’s recorded about itself, up until the show that just finished transmitting. A lot of the thought has been about “outputs” – complete shows. But there will be tracking shots of buildings that no longer exist, or dead people’s voices within those programmes. It’s more interesting to think of it as a collection of data – frames, chunks of programmes. If they can be digitised and catalogued, it becomes a data repository with an API. The BBC becomes a massive factory for making cultural product that people who understand RDF and XML can make use of.

Wouldn’t it be great if you could match BBC film or photos with local authority data about buildings? Wouldn’t it be great if you could match politician data with every appearance they’ve made on the BBC? And then you can find out how often they contradict themselves…

They want to bring all this into the Digital Public Space, while respecting the copyrights that exist. At least the catalogue can be there, and then the content as the rights issues are resolved. They’re working with partner organisations for a couple of years now. Lots of public cultural organisations – the British Museum, the BFI –  are having similar ideas.

But… the BBC doesn’t know what it has broadcast. So they’re creating Genome, by scanning and digitise the Radio Times – the best record of what the BBC planned to broadcast, back to its 1923. It’s been written about extensively on the BBC blogs.

They’re building a prototype digital public space navigator with partners, to try and navigate user journalists through the digital public space. This is a way of proving internally that it makes sense to work with those partner.

BBC Redux is iPlayer on steroids – with an API. It records the whole digital multiplex, so they can identify problems. Each show has an individual URI, which allows them to build tools on top of it. Snippets allows you to do a full text search of five years of BBC programmes, based on subtitles data. But the rights are complicated- because the BBC doesn’t own many of the programmes it broadcasts. – a joint BBC/ACE digital art intervention launching in May 2012. There’s an underlying data structure under all the art works – so it’s catalogued properly. Every piece of work will have a data feed as well as an asset field. They’re using ACE funding to try out BBC open data theories, that will feed into the digital public space.

Tom Steinberg

Tom Steinberg, founder and director of mySociety

mySociety is a charitable enterprise that exists to make people more powerful by giving them access to democratic process – like being able to contact their MPs. WriteToThem allows you to find out who all your political representatives are. It needed data: postcodes, where postcodes are in Britain, and which political unit each geographic point is in. And you also need who has been elected – and the government has the first two bits, but it doesn’t have the last in an organised fashion, so they had to go to third parties.

Intially, they had to scrape – steal – some of the data, because it just wasn’t available publicly.

FixMyStreet – which allowed people to pinpoint local infrastructure issues – another site they had to steal data to get the product to work. They were committing crimes to create a charitable, public good organisation going. The parts of government that should have been facilitating this just weren’t working. And that’s how he became passionate about open data. And he’s ended up writing policy for both Labour and Collation governments about this.

What has he learnt?

Don’t expect to win arguments on economic grounds. Economic decisions are not made on evidence – they’re made on evidence and prejudice. And most open data research is new, and doesn’t have clear economic evidence yet. By all means, mention economics, but don’t expect it to convert the unconverted. Instead, show them tools that will improve their lives.

Don’t emphasise making things more accountable. People who are already busy trying to deal with cuts and politics, don’t want you to make their lives harder. Instead point out that open web tools -like Google – are often better at finding information than their internal tools are. That makes their lives better. If you can persuade them than a website will stop the phone ringing, with people asking for things, you’ll persuade them.

But what happens when you have someone ready to have a go? While the likes of hack days, and data stores are useful, it’s far more useful to be good at requests that are already coming through all sorts of channels. Having a hack day while freedom of information requests are building up is an issue. – makes it easier to submit freedom of information requests. It has 20 people accessing the data there for every person submitting. it gets many more times traffic than the US initiative – because it’s all data people want, rather than data people have chosen to release. You need to empower someone to go looking for these requests, and making sure they happen. They need to bribe/flatter/lunch the relevant people until the information emerges. They should be looking for ways of responding to FoI data requests that’s better than asked – if they want a spreadsheet, give them a feed that’s already up to date.

Councils need to get into the business of collaborating to build tools that help author the information the needs to come out in a structured way. MySociety is working with a system to help author that sort of structured data. It’s easier to author a page on a politician with their system than it is to write it on paper… That’s how easy it needs to be.

MySociety is now legit. He had to ask and lobby and campaign. If you can get a button on your site that allows people to ask what they want – and then you to go an away and provide it, then you’re there. If your city has a button that says “give me the data I need:” and you have an 8/10 chance of getting it after you press it – you’re an open data city.

Jonathan Carr-West

Jonathan Carr-West, director of Local Government Information Unit

How do we deal with the financial crisis? With climate change?  How do we train our children for jobs that don’t exist yet. Think about ageing: there are 10,000 people over 100 in the UK. 2071? there could be a million. Life expectancy increases by five hours a day. These are real problems.

And solutions happen at a local level. These are existential problems – societies don’t always survive. Ask the East Islanders. Or the Maya. Some fail to adapt and innovate. There’s nothing given about the society we live in. We need to solve these problems.

It’s also a question of democracy – giving people the power to respond to these problems – and to respond creatively to local context. We need adaptation and selection of innovation – how do ideas connect? How can people pool their resources? Connected localism – local projects connecting with one another, so we can learn from others’ successes and failure. That’s how innovation can spread from one neighbourhood to another. We need to be cosmopolitan and local at the same time. For that to work we need other things – a field of exchange.

Historically the city has functioned as a field of exchange – goods, commodities and thinking are exchanged. But new see new digital fields of exchange are emerging. Mumsnet is a brilliant example. You can connect with a paediatric nurse in Australia – or a mother in the same street as you.

Let’s think about openness in that context. Field theory – a structured social space in which people interact. These spaces are defined by individual habits and social rules, both informal and formal, We need to adopt openness as part of the habit – and as part of the rules that underpin our interactions.

We don’t always know what will be useful. Whatever size of organisation you need to adopt the principle of openness, and that creates a field of exchange, which allows innovation to happen. Don’t worry is data is useful – people will find uses for it. Open data is a new city – and allows us to radically transform social services and the way we live.

Leigh Dodds, chief technology officer of Kasabi

Leigh Dodds

Leigh is talking about the foundations of digital cities. But he’s starting with Robert Hooke – an early member of the Royal Society – a natural philosopher, inventor, surveyor and architect. he had the hacker ethic. He was closely involved in the rebuilding of London after the great fire.  It was an organic city before the fire, a mix of tangled street and random building. The fire was seen as a chance to rebuild the city, after a terrible period of plague and fire. If their plans had gone ahead, it would have looked much more like Paris: big, wide boulevards. But there was a lack of funds to support the work and the relocations it would involve. Charles II called the rebuilding project to a halt, and moved to an iterative process based on legislated city standards.: width of streets, bricks for construction and so on. And it empowered Hooke to go and work with the community.

Now the fire is data. We have almost an embarrassment of riches in data available: but there’s more that could and should be opened up. We need some better infrastructure: ubiquitous wireless, better broadband. And local services need to become more timely and efficient. Data underpins all of that.

We’ve created lots of data hub – but he has a problem with them. They’re about governments publishing to a single community: developers. They need to be more multi-tenent. More organisations need to share their information through it, and more communities interact with the published data.  We need to make them more engaging places for the whole community.

A key aspect of a linked data approach is about giving everything unique identifiers: places buses, organisations, trains. Everything you want to describe. And they already have them – in propriety systems or behind paywalls. So it’s hard to combine data because of those barriers. A lot of the open data push is about open identifiers. Tim Berners-Lee is advocating a set of technologies, but the important thing is a common code. An URL for each thing is one approach. Context make sit easier to maintain and link data.

We need to encourage digital graffiti – if there’s a standard for identification, then the community can come along and start annotating it.

Data isn’t just data sets – it’s a useful piece of infrastructure we can build upon. And it’s an evolution of the role public authorities have always done: defining spaces, and building within it, digitally or physically.

An audience member asks about privacy concerns around personal data. Leigh replies that there’s an initiative around creating data hubs around yourself: managing your own data and who can access it.

Who should publish identifiers? Leigh suggests that whoever is mandated to create them, should own them and have responsibility for making sure they stay around and are sustainable.

Data quality? There’s been lots of examples of data quality being improved with lots of eyes on it. Maybe the solution is not to clean it before publication, but to publish and let collaborative cleaning happen. We’ll see an Open Street Maps type collaboration happen.

I’m at the Open Data Cities conference all day. And this is the opening session:

Sadly, John Barradell, chief executive of Brighton and Hove City Council isn’t with us, as he’s sick. He’s got stand ins: Charlie Stewart, strategic director and John Shewell, head of communications at Brighton and Hove City Council.


John Shewell

Busy week: 3rd Brighton marathon last weekend. This conference comes at an important time for public services. We’re at an interesting change – not just in the UK, but across the globe. The future of local government – and how opening up public data – can nudge people into opening up about what’s important to them. That’s our goal: connecting the city with the place. Inovolving people in design and delivery of public services.

It’s an iterative process – a journey that will continue for years to come. In May, we’ll have the Brighton festival, which has become one of the great cultural festivals of the world. It was proposed in 1964, and was first held in 1967. It’s brought together by collaborate – private and public, community groups and cities. It gets more ambitious every year – the spirit of the city. Collaboration works. It reaches into every part of the city – from schools to housing estates. That sort of collaboration lies at the heart of open data cities.

The concept of cities as systems: harnessing human and social capital, with infrastructure, to fuel sustainable economic development and a high quality of life. Democracy really matters – not just at elections, but a conversation every day. Open data is part of that process.

750,000 people are members of a political organisation. 7.5m want to be more involved in decisions that effect their lives. The challenges is in finding ways of allowing that that suit them. This is about creating a democratic organisation; one that is open to discussion and scrutiny, and which is accountable. It needs to protect the functions of the state we value the most, while allowing more social action. We need to understand the difference between the state as provider and the role of arbitrating democratically between equally valid needs. The second role is connecting the citizens: facilitating dialogue so we can shape the future together. We need to create space to allow those conversations to happen.

How do we unlock data so citizens han influence decisions before they’re made? There is a golden triangle:

  • public serives
  • citizens
  • innovation

We want to connect these to create a healthier city. We want to cede control to citizens in return for active collaboration. It’s as much a shift in attitude as the way we want to work.

Companies as ecosystems, not as top down machines. Strong values. Responsive to needs of customers. These are the principles we feel are right for the city council.


Charlie Stewart

Two years into a change process at the council. It’s not finished – and it may never be: it’s a journey. The  main aim is to bring the council closer to the community. They’ve been involved in initiative alike CityCampyBrighton. Another project is Patchwork: using social media to join up professionals to support families. They’re trying to work collaboratively across agencies. The worked with practitioners to design an app allowing the agencies to see who is involved with which family.

We Live Here – building systems using traditional and new media, and the basis for neighbourhood councils.

Beside sthe publication of various data sets on the website, there’s the Brighton and Hove  Local Information Service (BHLis), for use of everyone across the city. They want to improve their transparency and therefore their accountability. There at the start of a journey, to become a curator of the city’s needs and data, and to work with all organisations in the city to improve the outcomes. It’s about loose connections and networked organisations.

They need to move beyond FoI requests to really accessible datasets.


Citizen reporting at work

Laurie Penny:

As more and more ordinary men, women and children without degrees in journalism acquire the skills and technology to broadcast text and video, the media has become another cultural territory which is gradually being re-occupied. Those on the ground do not have to wait for the BBC and MSNBC to turn up with cameras: they make the news and the reporters follow. They have grown up in a world of branding and they know how to create a craze and set the agenda. They occupy the media. And the media is starting to worry.

I only disagree with that last sentence. The media has been worrying for years.