Mar 11
lindsayrgwattTechnology algorithms, fail, youtube
Yesterday I was listening to a podcast that referenced Cajmere’s Coffee Pot - commonly referred to as “Percolator” or “It’s time for the percolator”. I was wondering how the original song went, so I figured I’d listen to in on YouTube.
All good, but check out that “featured video” in the upper right corner. I’m guessing that YouTube noticed that people who looked at this video looked at that video – and therefore I might want to as well. But rest assured, I have little desire to copulate with animals.
Recommendation algorithms are hard, and this one’s not working yet.
Mar 02
lindsayrgwattTechnology google, search, test
You may or may not know, but Google tests hundreds of different versions of their search service every day. They’ve turned their users into a giant set of unwitting testers who are constantly providing them feedback on how to improve their product. This unparalleled ability to conduct tests is one of the skills that makes them currently unsurpassed in search.
Yesterday I turned into one of those testers. While searching for a particular term at work, I came across this design:
Here’s what the same search looked like when performed in a different browser:
What can we glean from this? Well, a few things:
Google’s test index (the number of documents is queries against) may be much bigger than it’s current index. The test page returned 5.2M documents vs. 1.3M for the normal version
Location is going to become more important in your search (no surprise in an increasingly mobile world). Note that in the test version, I can change my location from NYC. This is important, as if I search for “Zanzibar”, I get returned the bar in Hell’s Kitchen as the first result, not the beautiful island off the coast of Africa
Finally, Google thinks that they type of content you’re looking for is as important as what you’re looking for. If you were to click “More” under “Everything” in the test version, a list showing Images, Videos, etc. would have opened up (this normally appears at the top of the page). Just like my employer or Best Buy, Google’s trying to make it easier for you to find info using faceted search.
Why type “Zanzibar photos” when you could type “Zanzibar” and then click “images”. While that takes two steps, it allows you to easily flip between different types of info about Zanzibar, rather than having to re-type your query.
This is an interesting way for them to start to integrate all their different search properties together (Google, images, YouTube, scholar, books, etc.), and I hope it makes it into the real world.
Jan 31
lindsayrgwattTechnology google maps, maps
In my opinion, one of the greatest innovations of the past five years is how location has become a part of everyday life. Google Maps or its equivalent has become a standard tool in many people’s life. When you’re looking for directions, a place, etc. you simply call it up; no more guessing where you are.
We’ve come a long way in the past five years, but a couple of recent experiences reminded me of just how far there still is to go.
First one: I can walk from NYC to Ottawa in 5 days.
As Wen and I were going home for Christmas, I decided to see what the directions would look like ‘by foot’. I was impressed that it would only take about five and half days to do that. That’s about 72 miles a day.
You get a great sense of how Google’s algorithms work here. The average human can walk about 3 miles per hour. There are 24 hours in a day. Ergo, 72 miles per day and 5.5 days to Ottawa.
My second moment came when I stumbled upon this nameless street when trying to find a cafe:
Of course, this street does have a name (Greenwich). And this example speaks more to the power than limitation of online mapping tools: they’ve become such a part of my life that a part of me almost questions why the street has no name, rather than thinking there’s something wrong with the program.
Jan 20
lindsayrgwattBusiness, Technology apple, payments, speculation, tablet
It’s official. Apple’s releasing something next week; maybe the table computer that’s going to revolutionize everything. Lots of people have speculated as to what’s going to be in it. A lot of people expect that it might save journalism and media.
Let’s speculate about how it might save journalism (or your industry of choice). It’s going to have to provide a great experience that’s fundamentally different from how it works today (check out this Sports Illustrated mock for one hypothesis). However, I think it’s going to have to make it really easy to pay for this content, too. Ads alone aren’t keeping these folks in business, so a new format alone (which is just more ad inventory) isn’t going to be enough.
So what could this mean? Well, imagine a monthly subscription to all New York Times content – across your tablet, iPhone and the web – with the first 100 articles or so free.
Similarly, right now it’s tough to do subscriptions for an iPhone app. You have to put the app in the store and then the subscription occurs within the app (e.g., it sells for $0.99 in the store but then it’s $9.99/month within the app to use it). I’m hoping that with the tablet, Apple will continue switching the store to allow transparent per month pricing at the time of purchase, with a free trial.
Ideally, two things come out of the tablet:
- New ways to convince users to subscribe to content. Ideally, providing a ubiquitous payment system that works across all digital channels
- An improvement to subscription pricing in the iTunes store
These sound totally banal, but if this happens it will suddenly be financially viable for a whole new type of software to be developed for the iPhone. Apple can build its mobile lead and really revolutionize a lot more than just journalism.
Jan 04
lindsayrgwattNYC, Technology Gov2.0, NYC Big Apps, NYC Data Mine, Open Data
I was part of a team that recently submitted an app for the NYC BigApps contest and therefore had the opportunity to play with a lot of the data sets in the NYC Data Mine. In the spirit of continuous improvement, I wanted to suggest how the city can take a great resource and make it even better.
After combining nine data sets (and being unable to combine many others), we feel that we can talk to some of the challenges in working with the data and the overall opportunity to improve it. Our main suggestion is that the City should think of itself as a platform that provides standardized data and end users like ourselves add value by bringing together the different data sets and 3rd party services. We can’t claim to have created the “government as a platform” idea, but we can tell you some of the issues New York City faces if this is the path you (hopefully) take.
With that, here are a few suggestions:
- Make all your data machine readable. Some of your data is incredibly rich, but buried in formats that are hard to parse. The best example is probably the community district demographic info: there are over 100 spreadsheet pages of rich data there, but it’s incredibly difficult to use in CSV form. Ideally, this would all simply be XML and it could be reused in seconds.
- Open source formats please. Some of your data is locked in proprietary formats, accessible only to users with multi-thousand dollar software packages (a great example are the gdb files for ArcView shape data). If you want random citizens like us to mash up your data, please provide the data in exclusively open source formats as our software budget is $0. (It looks like this is changing – I noticed that some of the gdb files at the Data Mine now include shape files as well. Kudos)
- Add lat/longs (the commercial standard for mapping services) to every address/point. Some of your address data consists just of addresses without lat/longs. Some of these (e.g., certain school locations) are improperly formatted, so we can’t use commercial systems like the Yahoo Geocoding API to calculate lat/longs and put them on a map or assign them to a district/neighborhood. Other files contain lat/longs but they’re in a different coordinate system: this means that developers need to convert them into lat/longs in order to map them – and this process introduces errors and takes unnecessary time.
- More documentation. Some of your files contain lots of data but lack the documentation necessary to make them usable. The best example are the health info shape files: they’re full of statistical data for different areas, but there’s no way to figure out exactly what each of the statistical values means. Similarly, the shape files for each community district contains their area – but there’s no unit of measurement (it’s something called ‘internal units squared’).
- More data. You gave us some great data to play with, but there’s still a ton more. What about Compstat for crime data? School scores and their trends over time? The location of different alarm boxes so we can see how response times vary by area?
These suggestions are meant entirely as an opportunity to build off of a great start. We – the people of New York – want to work with the city to help make it a better place. However, the value we add is in creating tools and finding hidden relationships in the data – not in standardizing it; that’s where you can make the system work.
If you provide us with well-formed, machine-readable, standardized data, we’ll help build the services that citizens need and free up the city’s resources to focus where they’re needed most. We’ll also find new relationships in the data that might help you rethink policy initiatives (see how your data suggests an interesting link between ‘education’ and ‘going green’; I’ve no idea if you knew this).
2009 was a great start for a more open NYC.gov. Can’t wait to see what you do in 2010.
Dec 16
lindsayrgwattNYC, Technology Gov2.0, NYC Big Apps, UncoverYourCity
So last week, Wendy, Jill and I unveiled UncoverYourCity. This is a site that we put together as part of the NYC Big Apps competition (you can vote for us here; you’ll need to create an account). We want to share a bit of background on what the project is and why we did it.
Why?
There’s a nascent movement called Government 2.0 which seeks to apply the principles of the web to government and make it more open and efficient. One of the first steps in governments becoming open is making their data available online for any citizens who want to use it. The government has some of the most interesting data out there – everything from demographic info to build permit locations to school scores – and they’ve got more information than just about anyone else.
This movement has gained a lot of traction at the municipal level and San Francisco, Vancouver, Toronto and New York are some early cities to start putting municipal data online. New York has gone a step further by creating the Big Apps competition to get people to showcase what could be done with the data. We decided that we wanted to create an app to support the the city and also learn what could be done with the data.
So What Is It?
We wanted to create an app that would compare the quality of life in different New York neighborhoods and help people find the neighborhood that was perfect for them. If you know this city, you know that there are 8 million people that exhibit remarkable diversity. It’s what makes the city magical but also makes it hard to grasp. We wanted a tool to help people grasp it.
However, we quickly realized that this was way too hard to do (more on that in a future post) and that we weren’t comfortable placing a “quality of life” ranking on different areas. Instead, we decided that the right thing to create was an app that would let people learn more about the neighborhoods they live in and compare them with others.
The result is UncoverYourCity. We’ve combined almost a dozen different data sets (sounds easy, but it’s not) so that people can see how their neighborhood squares with others. You can use it to discover the leafiest streets in NYC, compare the neighborhoods with the highest and lowest murder rates (bet you don’t guess either one) or see interesting relationships like that between poverty and renting.
This isn’t a gimmick, rather, we believe it’s got the potential to help you see the challenges facing the city in a new way. Take the Mayor’s plan for making the city greener. I’ve no idea how the city is thinking of making the city greener, but one hypothesis might be that if we increase population density we might be able to increase recycling rates (if you live in condos, etc. they usually have recycling designed into the building). However, our stats suggest that there’s no relationship between recycling and population density:
However, there’s a pretty strong relationship between education levels (% population holding bachelor’s/graduate degree) and recycling rates (graph below). This suggests that making the city greener may need to include elements to improve education. It’s a similar story if you compare recycling rates with medium household income or poverty rates.
The tool can also show us outliers that may represent opportunities to learn new approaches to apply elsewhere in city. One of my favorites is the relationship between Median Household Income and Family Poverty. There’s a big outlier in the bottom left of the graph: Brooklyn Community 13 – if it was like other districts, based on its income it should have a poverty rate of about 28% but instead its holding out at 18%.
Is this due to the housing projects of Coney Island working as planned? Maybe it’s the tight Russian community of Brighton Beach taking care of their own and making sure that everyone’s doing okay. Or maybe Sea Gate’s population is so affluent that it skews the poverty level down. I don’t know, but if I were trying to reduce poverty in the city I’d try to find out.
So give the app a try. It’s not perfect – the site’s a bit slow (we’re not great programmers) and the navigation can be awkward (we ran out of time to get it polished) – but there’s something there for everyone. If you want to learn more about how we built it and why the Gov 2.0 movement is important, stay tuned to this blog (we’re also open sourcing all the code; stay tuned for links to code and data). And, when you’ve got a moment free, vote for us.
Dec 12
lindsayrgwattTechnology algorithms, facebook, fail
Wow, I haven’t even been married six months and Facebook already thinks that I’ve got marriage issues. I’d like to think that my marriage is going really well. In fact, I thought that the day we spent today, traipsing around town, was some of the most fun in ages.
But Facebook apparently thinks otherwise. In fact, when I logged in tonight, it invited me to reconnect with Wendy Franks, my wife:
Apparently, their algorithms have determined that we’re drifting apart and need to ‘reconnect’. I guess Wendy keeping me in her profile photo is just a charade (and a particularly cruel one, given that it’s one of our wedding photos) – but Facebook knows better. In fact, it’s completely unimaginable that the reason that I don’t write on her wall could be the fact that I live with her and spend more time with her than anyone else on earth, right?
Time to tweak the algorithms folks.
Oct 19
lindsayrgwattProgramming, Technology arduino, hacking, nyc resistor
Yesterday I spent the afternoon at NYC Resistor making my own Arduino (technically a Freeduino) microcontroller. Say what? That basically means that I spent the day soldering together a bunch of pieces of electronics that let me connect my computer to the physical world.
Here’s what the darn thing looks like:
The best way to look at this board is to look at the narrow black slits at the bottom and the top on the right hand side: they’re pins that you can connect to external electronic components (e.g., lights, sensors, etc.). The big black chip controls the signals to those ’slits’ and ultimately connects to your computer via the silver piece on the upper left; you can alternatively plug the board right into the wall via the lower left.
The board itself will not do too much: it has four LEDs and you can blink one of them to make sure it’s working:
I’m going to play around with this some more and if I create anything interesting I’ll put some photos up.
Sep 24
lindsayrgwattBusiness, Technology
There’s a great two-part interview with Marc Andreessen up at the Business Insider (although it’s occasionally painful to watch as their streaming is buggy beyond belief).
Marc talks about Netscape and what he’s learned about creating technology businesses, strategy, etc. Here are a couple of gems:
- Jim Clarke spent four years in the wilderness at Silicon Graphics before he had a product that people actually thought was useful. This helped him mentor Marc while Netscape was in its wilderness period
- Before starting Netscape, Jim & Marc had a variety of ideas, including an idea for XBox Live-but it was 1993 so it couldn’t work.
- When they started Netscape they had two elements that helped them:
- The idea worked: Mosaic as a browser was a research success. They just had to rewrite the code and commercialize it
- They defined their business model (sell servers and commercial browsers; give non-commercial browsers away for free)
- Note that the above basically says “create a great product and then build a company”, not the other way around
- Business plans are useless – but the process of planning is very valuable. It forces you to think critically about your business and show that you understand the issues
- Every tech company starts off doing something different than one it originally planned to do. Microsoft started writing compilers but made money in OS. Apple started with the Apple II but made its money on the Mac. Silicon Graphics started selling computer chips and ended up making supercomputers.
- “There’s not a tension between product and business [in the technology industry] – there’s a staging. You don’t have a business until you have a product – especially a product that lots of people want.” It’s about product/market fit. Make a killer product and then scale up. Marc seriously wants to avoid creating companies that grow to 75 people, go nowhere and everyone just collects a salary.
Marc answered one big question I’ve always wondered: why did AOL buy Netscape? After all, the business tanked immediately after they bought it.
It turns out that AOL never really cared about the business and actually never even bothered to really run the company. They bought the company so that they would be perceived as “the blue chip internet company,” the “GE of the Internet”. It worked – there stock subsequently quadrupled and they were able to buy Time Warner.
Finally, he was asked what three things made him the success that he is. His answers might surprise:
- Luck
- Timing – which he considers luck
- Place – being in Silicon Valley has forced him to be better as he is always surrounded by so many smart people
Here are the videos:
Sep 22
lindsayrgwattHistory, NYC, Technology gis, google earth, maps, nypl, open street maps
I went to the Conflux Festival on Saturday morning and attended a talk by Matt Knutzen entitled Rebuilding the Historical City. Matt’s a cartographer working at the NYPL and he was explaining a new tool they’ve built – and the Very Big Idea behind the tool.
The NYPL has over 60,000 maps of NYC in their Digital Gallery, but they’re simply digitized images. They lack any actual mapping points: latlongs, etc. that can actually be use to project the map onto other maps. As a result, they’ve decided to build a tool – the Map Rectifier – that allows anyone to convert an image of a historical map into an actual working map and share the results with the world.
The process is simple: you find an old map you want to convert into a working map. You put it side by side with an OpenStreetMap map of New York. You then click on a point in the old map and click on a corresponding point in OpenStreetMap. Once you’ve done at least four points, you click “rectify” and the system warps the image of the old map to fit it onto the real map. At this point, the old map is converted into a set of latitude and longitudes and can be used elsewhere (the system is also smart enough to tell you if you did a bad job). I’d show you screen shots, but I can’t get a login to the system; it’s still in invite only mode 
There’s some other cool functionality in the tool: it’s got the ability to crop maps (so you can skip parts of the map) and you can also trace out buildings and add data about them (e.g., it’s a public house, etc.).
This gets better and better because once you’ve converted the map into a set of KML coordinates and you can view it in Google Earth. For example, here’s a projection of a 1924 aerial set of photos vs. what’s there today (a lot more farmland back then):
Here’s another example, from the 20th Ward’s fire insurance map. You can see what Madison Square looked like before the Garden and the Farley post office were built:
This technology is impressive as you can start to tell and visualize the history of the city. Moreover, once the system launches, it’s going to be open to the public and anyone can rectify a map (that’s a sea change in how libraries work). Also, kudos to the NYPL for making the entire system open source: you’ll be able to install the software on your own server and start rectifying your own maps.
Older Entries
RSS
Recent Comments