Jan 04
lindsayrgwattNYC, Technology Gov2.0, NYC Big Apps, NYC Data Mine, Open Data
I was part of a team that recently submitted an app for the NYC BigApps contest and therefore had the opportunity to play with a lot of the data sets in the NYC Data Mine. In the spirit of continuous improvement, I wanted to suggest how the city can take a great resource and make it even better.
After combining nine data sets (and being unable to combine many others), we feel that we can talk to some of the challenges in working with the data and the overall opportunity to improve it. Our main suggestion is that the City should think of itself as a platform that provides standardized data and end users like ourselves add value by bringing together the different data sets and 3rd party services. We can’t claim to have created the “government as a platform” idea, but we can tell you some of the issues New York City faces if this is the path you (hopefully) take.
With that, here are a few suggestions:
- Make all your data machine readable. Some of your data is incredibly rich, but buried in formats that are hard to parse. The best example is probably the community district demographic info: there are over 100 spreadsheet pages of rich data there, but it’s incredibly difficult to use in CSV form. Ideally, this would all simply be XML and it could be reused in seconds.
- Open source formats please. Some of your data is locked in proprietary formats, accessible only to users with multi-thousand dollar software packages (a great example are the gdb files for ArcView shape data). If you want random citizens like us to mash up your data, please provide the data in exclusively open source formats as our software budget is $0. (It looks like this is changing – I noticed that some of the gdb files at the Data Mine now include shape files as well. Kudos)
- Add lat/longs (the commercial standard for mapping services) to every address/point. Some of your address data consists just of addresses without lat/longs. Some of these (e.g., certain school locations) are improperly formatted, so we can’t use commercial systems like the Yahoo Geocoding API to calculate lat/longs and put them on a map or assign them to a district/neighborhood. Other files contain lat/longs but they’re in a different coordinate system: this means that developers need to convert them into lat/longs in order to map them – and this process introduces errors and takes unnecessary time.
- More documentation. Some of your files contain lots of data but lack the documentation necessary to make them usable. The best example are the health info shape files: they’re full of statistical data for different areas, but there’s no way to figure out exactly what each of the statistical values means. Similarly, the shape files for each community district contains their area – but there’s no unit of measurement (it’s something called ‘internal units squared’).
- More data. You gave us some great data to play with, but there’s still a ton more. What about Compstat for crime data? School scores and their trends over time? The location of different alarm boxes so we can see how response times vary by area?
These suggestions are meant entirely as an opportunity to build off of a great start. We – the people of New York – want to work with the city to help make it a better place. However, the value we add is in creating tools and finding hidden relationships in the data – not in standardizing it; that’s where you can make the system work.
If you provide us with well-formed, machine-readable, standardized data, we’ll help build the services that citizens need and free up the city’s resources to focus where they’re needed most. We’ll also find new relationships in the data that might help you rethink policy initiatives (see how your data suggests an interesting link between ‘education’ and ‘going green’; I’ve no idea if you knew this).
2009 was a great start for a more open NYC.gov. Can’t wait to see what you do in 2010.
Dec 21
lindsayrgwattFood, NYC new amsterdam market, tourtiere
Yesterday was the year’s last installment of the New Amsterdam Market. It’s New York’s most charming farmer’s market and is held in one of it’s most charmless locations (underneath the FDR by the old Fulton fish market). The market is a celebration of the passion and craft of local and regional food artisans. Whether you’re looking for bread, meat, cheese, chocolate, coffee or olive oil, the common thread is an insatiable passion in the topic by whoever is selling to you.
This leads to some outlandishly enjoyable situations: below is a photo of the folks from Prime Meats selling duck confit hot pockets. Unbelievably delicious and yet ridiculous at the same time.
The other great find at the market was Sarah Obratis’ & Hugue Dufour‘s meat pies. Technically, the weren’t actually at the market, which meant that you had to literally buy a pie out of the trunk of their Corolla (it all felt very surreptitious). The tourtiere’s have a mouth-watering mix of meat – brisket, pork and turkey – and come with their own Cranberry-based tomato sauce. Ours lasted two days (really, more like 28 hours).
Dec 19
lindsayrgwattHistory, NYC
Today I went to an exhibition at the Met called Art of the Samurai: Japanese Arms and Armor, 1156-1868. It has given me a new appreciation for how absolutely insane Japanese culture is. First, let’s take a look at some of the different helmets that Samurai might wear into battle:
If you were to ask “Lindsay, isn’t one of those helmets done up to look like a swallow’s tail? And is the other one a set of crabs claws?” you’d be absolutely correct. Apparently the Samurai helmet was the early equivalent of taking your shoe off and banging it on your desk at the UN – an attempt to intimidate your enemy by convincing them that you were crazier than they were.
Their armor certainly suggests that. Granted, to a jaded 21st century dweller, it looks like the sort of costume a psychopathic midget who worships the Village People might wear (the armor dates back to the Middle Ages and the Japanese were quite short back then), but I can only imagine how exceptionally nutty it would look when combined with a very sharp sword.
If you look closely at the images below you’ll notice a few things. First, everyone’s wearing a mask to look a little crazier – although the mustaches undermine it. Second, the guy with brown hair actually does have hair coming out of his helmet-there’s some sort of animal pelt there; a different set of armor (not shown) actually had long, flow, dyed red hair attached to it. Finally, the red outfit suggests where Darth Vader comes from.
Go to the exhibition if you get the chance!
Dec 17
lindsayrgwattNYC NYC, statistics, UncoverYourCity
If you follow this blog, you’ll know that we launched UncoverYourCity to help people better understand New York. Here’s another example of interesting things we can learn about the city by playing with the data.
Let’s say you wanted to see the relationship between people renting and their household income. A few things emerge here: if you’re, on average, poor, you rent. If you’re rich, you buy. The poorest Districts of New York are in the southern Bronx and a shocking 92% of people rent; just up the Harlem River, 94% of people in Morris Heights rent. The average family income here is around $17-22K per year.
The converse is also in the south: in southern Staten Island, 85% of people own and the household income hovers near $83K. Here’s a comparison of all the districts side by side.
However, if you look at the plot of household income vs. rental rates you’ll notice a bunch of outliers in the upper right corner:
These are a couple of districts towards the tip of Manhattan (districts 1, 2, and 4-8) plus district 6 in Brooklyn (Cobble Hill, Carroll Gardens, Park Slope).
What’s going on here? Well, if you asked me to pick the areas in New York where people who are passing through (i.e., getting experience before they move to a different city or start a family in the suburbs) live, I would pick these neighborhoods. These are places where younger professionals move to get experience yet maintain a high quality of life (yes, there are lots of people there who are neither young nor professional, but we’re talking ‘on average’ here) and this may be why we see this litter cluster of outliers.
What’s interesting is what’s not in the set of outliers: Williamsburg (Brooklyn District 1) and the Lower East Side (Manhattan District 3). My bet is that these neighborhoods are both subsumed by the averages of their districts and their more bohemian younger residents may have a lower average family income.
This data doesn’t tell the whole story, but it does help us understand the city a bit better. More fun examples in future posts.
Dec 16
lindsayrgwattNYC, Technology Gov2.0, NYC Big Apps, UncoverYourCity
So last week, Wendy, Jill and I unveiled UncoverYourCity. This is a site that we put together as part of the NYC Big Apps competition (you can vote for us here; you’ll need to create an account). We want to share a bit of background on what the project is and why we did it.
Why?
There’s a nascent movement called Government 2.0 which seeks to apply the principles of the web to government and make it more open and efficient. One of the first steps in governments becoming open is making their data available online for any citizens who want to use it. The government has some of the most interesting data out there – everything from demographic info to build permit locations to school scores – and they’ve got more information than just about anyone else.
This movement has gained a lot of traction at the municipal level and San Francisco, Vancouver, Toronto and New York are some early cities to start putting municipal data online. New York has gone a step further by creating the Big Apps competition to get people to showcase what could be done with the data. We decided that we wanted to create an app to support the the city and also learn what could be done with the data.
So What Is It?
We wanted to create an app that would compare the quality of life in different New York neighborhoods and help people find the neighborhood that was perfect for them. If you know this city, you know that there are 8 million people that exhibit remarkable diversity. It’s what makes the city magical but also makes it hard to grasp. We wanted a tool to help people grasp it.
However, we quickly realized that this was way too hard to do (more on that in a future post) and that we weren’t comfortable placing a “quality of life” ranking on different areas. Instead, we decided that the right thing to create was an app that would let people learn more about the neighborhoods they live in and compare them with others.
The result is UncoverYourCity. We’ve combined almost a dozen different data sets (sounds easy, but it’s not) so that people can see how their neighborhood squares with others. You can use it to discover the leafiest streets in NYC, compare the neighborhoods with the highest and lowest murder rates (bet you don’t guess either one) or see interesting relationships like that between poverty and renting.
This isn’t a gimmick, rather, we believe it’s got the potential to help you see the challenges facing the city in a new way. Take the Mayor’s plan for making the city greener. I’ve no idea how the city is thinking of making the city greener, but one hypothesis might be that if we increase population density we might be able to increase recycling rates (if you live in condos, etc. they usually have recycling designed into the building). However, our stats suggest that there’s no relationship between recycling and population density:
However, there’s a pretty strong relationship between education levels (% population holding bachelor’s/graduate degree) and recycling rates (graph below). This suggests that making the city greener may need to include elements to improve education. It’s a similar story if you compare recycling rates with medium household income or poverty rates.
The tool can also show us outliers that may represent opportunities to learn new approaches to apply elsewhere in city. One of my favorites is the relationship between Median Household Income and Family Poverty. There’s a big outlier in the bottom left of the graph: Brooklyn Community 13 – if it was like other districts, based on its income it should have a poverty rate of about 28% but instead its holding out at 18%.
Is this due to the housing projects of Coney Island working as planned? Maybe it’s the tight Russian community of Brighton Beach taking care of their own and making sure that everyone’s doing okay. Or maybe Sea Gate’s population is so affluent that it skews the poverty level down. I don’t know, but if I were trying to reduce poverty in the city I’d try to find out.
So give the app a try. It’s not perfect – the site’s a bit slow (we’re not great programmers) and the navigation can be awkward (we ran out of time to get it polished) – but there’s something there for everyone. If you want to learn more about how we built it and why the Gov 2.0 movement is important, stay tuned to this blog (we’re also open sourcing all the code; stay tuned for links to code and data). And, when you’ve got a moment free, vote for us.
Nov 09
lindsayrgwattNYC google, maps, mta, subway
Last week Google added subway lines to the list of items that it shows in Google Maps for NYC. What’s neat about this is that we can now plot where the subway lines are in the real world vs. where they appear on the MTA subway map:
A couple of themes emerge:
- Manhattan is ridiculously oversized in the MTA version. Note how much smaller it should be
- If you live in East Brooklyn or the entire borough of Queens you’re pretty much out of luck when it comes to subway transport
- It is ridiculous how far the subway lines are from both JFK and LaGuardia
Nov 02
lindsayrgwattNYC
Here are a bunch of random thoughts/comments from yesterday’s marathon:
- As we were waiting for the race to start on Staten Island, we couldn’t help but notice the sniper on roof of one of the buildings. And it was definitely a sniper as he was defiantly placing his rifle on the ledge so that anyone could see it
- Tonnes of people showed up late for the race and therefore had to jump fences to get into the corrals we were held in before the start. Imagine training for four months and then blowing the race because you couldn’t be bothered to get to the race on time?
- The start was sudden: a message from the Mayor, race instructions and then the howitzers (race starts at Fort Wadsworth) went off, followed by us! Immediately, New York, New York began playing; a great way to start a race
- Running across the Verrazano Bridge gives you one of the best views of the city you’ll ever get. I suspect it might be the highest point in the entire city. It’s also the only bridge in NYC you can’t walk over so it’s a treat to run on. While we were running over it we were also being buzzed by at least four choppers: at one point one was above us while another was moving beside us; I almost caught the glare off the camera. I also noticed two massive Chinook helicopters headed towards us, coming in low over the harbour, but the broke away at the last moment and turned to Staten Island
- When we hit Bay Ridge (just across the bridge), the cheering began. At the first fluid station, some cheeky future entrepreneur was offering Gatorade for $0.25 (in case you’re wondering, it’s free; every mile from mile 3 onwards)
- Along the way there are lots of bands/djs. You can tell a neighbourhood’s character from the music: Bay Ridge = classic rock, Sunset Park thumps to a Hispanic beat while Harlem and the Bronx are a little more hip hop
- Fort Greene was the best neighbourhood. For the first few miles, the runners are split into three, then two groups. They come together near Fort Greene, so for a mile or so it gets quite intimate. The street, Lafayette St., narrows at this point and at the same time Fort Greene has some locals who really like to cheer. It was best summed up by the guy next to me who turned to me and exclaimed “This is fucking amazing!”
- Shortly after, we hit Williamsburg. After an inspiring run past a church with a gospel choir outside, the neighbourhood became incredibly quiet. The neighbourhood is bifurcated by the bridge and to the south it is all Hasidic Jews (it’s literally like running through Once Upon A Time In America). They don’t really cheer for marathoners too much. To the north is hipsterville and there you’ll find bands and people eating brunch while cheering you on
- When we passed the halfway mark I let out a cheer, but unfortunately I was surrounded by Europeans who didn’t seem too excited. In fact, in general, the Euros weren’t too friendly. No idea why
- Crossing the Queensboro Bridge from Queens into Manhattan was a little surreal. We were running on the lower level, so it was a bit like being in a sooty tunnel. As we ran, the trains were running beside us and I think they were going slower so that the passengers could gawk at us. I also noticed that when we got to the middle of the bridge they hadn’t yet cleaned up the bottles of the lead runners (these guys actually set up their own water bottles at spots on the course) and they were still lying on the course
- I’d heard that when you come off the bridge and hear the crowd it’s unbelievable…and it was. The roar started with a few hundred meters to go and then suddenly a crowd appeared. Their cheers became a roar and every runner picked up their speed; the cheers continued up 1st Ave
- At 81st street or so I saw another set of cops on a roof watching the crowd
- Somewhere between 90th and 100th, there was a DJ cranking Empire State of Mind. Hearing Alicia Keys sing “Let’s hear it for New York” was sonic adrenalin and made me accelerate
- As I swung back into Manhattan from the Bronx, I saw my coworker Jorgette on the corner of 137th. Very cool to see someone I recognized
- With about five kilometers to go I felt my hamstring start to twinge and new that there was no way I was going to make my goal time of 3:20. In fact, I’d been dropping time since coming hitting kilometer 30. Next year I’ll have to run longer long runs
- After the race, I got stuck next to this guy on the subway:
Before he fell to the left, he was leaning left on me:
We made a great team. I’m pretty sure I smelled because at least one person moved away from me after sitting down next to me- Finally, Timex gave out the best swag: a fridge magnet where you could put your time. Here’s mine:
Oct 31
lindsayrgwattNYC drum and bass, ltj bukem, mc conrad, Music
Last night Rich and I went to see LTJ Bukem and MC Conrad at Club Love. It’s one of my favourite places to see a dj in NYC – a bit dive-y, great sound system and a totally unpretentious crowd:
The other great thing about Love is there light show. They’ve got human-powered lasers: a woman literally fires a laser through a set of prisms to create the visual effect she wants. Definitely a job that I never would have guessed existed:
Here are a couple more snaps:
Oct 11
lindsayrgwattFood, NYC
Do you have an alcoholic Russian (or Russian-loving friend) who is filthy rich and you’re struggling to find a gift for them? Maybe they own a football or basketball team and already have a chateau in the French countryside. Fortunately, the good folks at Astor Wines understand your challenges and offer Imperial Collection Faberge Egg:
This bit of conspicuous consumption/unnecessary luxury can be your for the astonishingly low price of $2,399.99 (note that they ran out of space for the last ’9′ in the shot below):
Your oligarch awaits his gift.
Oct 09
lindsayrgwattNYC, Random memory, perception, reality
Last night there were fireworks over the harbour and Wen and I sat on our deck watching them. This morning I went for a run and almost bumped into a shackled prisoner going into the police station around the corner.
When I hit the bottom of Union, the Queen Mary 2 was parked in the cruise ship terminal and was glowing with all its lights turned on in in the pre-dawn light.
Upon my return, our corner had turned into the staging area for a SWAT team raid. Cops in body armour and machine guns prepped their shields and set up a convoy that also included an ambulance and a fire truck.
And now it’s evening. The last wisps of firework smoke drifted away hours ago. The accused has been processed and sent to Rikers. The Queen Mary has sailed and somebody who I’ll never know had one hell of a surprise this morning.
But if you were to visit my apartment today you’d never know that any of this ever happened.
Older Entries Newer Entries
RSS
Recent Comments