Technology – Random Dispatches

Finding Nearby Things in QGIS (Tutorial)

If you get excited by maps and places, here’s a reasonably common problem:

I’ve got a set of locations (e.g., landmarks)
I’ve got another set of locations (e.g., points of interest) and those points have additional metadata (e.g., a star rating for each place)
I’d like to find all the points of interest with at least a certain star rating that are within a fixed radius of my landmarks

I want to walk you through one way to solve this problem. (And you can find all the files in this Github repo).

Step 1: Get your data

I’m starting with two .csv files: base (my landmarks) and nearby (my points of interest). Each contains a name of a place, a lat/lon and, for nearby, a rating:

Base

Nearby

Step 2: Create your QGIS project

Open up QGIS and create a new project. I want to measure distance in meters and the locations I’m using are in the United States to I have set my default QGIS projection to EPSG 2163 – which is measured in meters. This is going to be critical: if you use a projection like EPSG 4269 (NAD 83) then you’ll end up doing all your work in degrees – which is meaningless.

Step 3: Import your .csv files as new layers

Use the command Layers >> Add Layer >> Add Delimited Text Layer. Use “Browse…” to locate the files and import similar to the image below:

After you import both files, you should have a (boring) map like the following:

Here’s a key thing to understand: your .csv files did not have a projection associated with them so QGIS adds the default of EPSG 4326 – which is in degrees, not meters. We need to fix that.

Step 4: Convert the layers to EPSG 2163

Right click on the name of each layer and select “Save As…”. Make sure you set the file name of the new layer and change the CRS to EPSG 2163. I also end the name of my files in the coordinate system (e.g., base_2163) so that I can remember it unprompted.

Step 5: Start a new QGIS project

I now start a new QGIS project so that I can be in the correct coordinate reference system (CRS) – EPSG 2163. Add both the new _2163 shapefiles by using the Layer >> Add Layer >> Add Vector Layer… command.

I added labels to each layer to understand what I’m looking at. You can show both the name and rating on nearby_2163 by using the expression concat( "location", ' | ', "rating"). Your map should look something like the following:

Step 6: Draw radius around landmarks

Use the command Vector >> Geoprocessing Tools >> Fixed Distance Buffer. Set your input layer as base_2163 (and verify that it says [EPSG:2163] next to it. Set your distance to a value in meters (I’m going to use 1,000 for a kilometer). Under “Buffer”, save this a new shapefile by clicking the … :

After you hit the “run” button and rearrange the layers on the map, you should see something like the following:

Step 7: Add landmark info to points of interest

This is where the magic happens. Go to Processing >> Toolbox and the click on SAGA (2.2.3) in the new right hand tab and open up “Vector Point Tools” and double click on “Add polygon attributes to points”.

Your “Points” is nearby_2163 and you want to add the nearby landmark: set “Polygons” to Buffer and select the “Attribute” location. Per earlier, save the new layer as a file:

Your map now has a new layer called “Results” (or similar). Right click on the layer and select “Open Attributes Table” and you should see that each point now has the corresponding nearby location attached:

Step 8: Visualize or export

If you want to see a subset of ratings (e.g., 10 or above), then just right click on “Results” and select Filter… Set the value of ratings to above 9 and you’ll see that one of our locations is now gone:

Here’s the map. For some reason, the names aren’t showing properly on the labels but you can see that only the places rated 10 are showing:

If you save the layer (right click) as an Excel spreadsheet, you’ll get the values in a form you can play with:

Understanding The Now

While 2016 has been an annus horribilis overall (unless you’re a racist, in which case, it’s probably a magical time to be alive), it has a been a great year for books helping us attempt to understand the heady times we live in.

Most people would agree that we live in a state of rapid change. In their book Whiplash, Joi Ito and Jeff Howe call it “exponential times” due to asymmetry (small can now beat big), complexity (interconnectedness at scale) and uncertainty (our existing institutions haven’t evolved to handle current situation). This provokes many questions, but one immediate one is: will this continue?

In his book The Inevitable, Kevin Kelly answers this with a resounding “yes.” The book outlines the twelve technological forces that will keep the rate of change high for the foreseeable future. Kelly, an original Wired editor, is an accomplished writer and technologist, leading to spare sentences full of insight such as:

We are moving away from the world of fixed nouns and toward a world of fluid verbs.

In the intangible digital realm, nothing is static or fixed. Everything is becoming.

Technological life in the future will be a series of endless upgrades.

In this era of “becoming,” everyone becomes a newbie. Worse, we will be newbies forever. That should keep us humble.

Kelly’s thesis is that we are moving into a world called Protopia where there is a constant, steady accumulation of positive changes, but similarly an increase in new problems. The net is positive, so society advances. We all get mobile phones with the internet but have to listen to occasional idiots shout into their Bluetooth headsets while walking down the street.

Kelly’s book dives deep into the twelve forces and backs each up with a wealth of sometimes obscure information. Printing drove the number of word available from 50,000 in Old English to over 1,000,000 today! Dematerialization means that one kilogram of inputs produced $1.64 in GDP in 1977 and $3.58 in 2000!

These stats are neither cocktail party fodder nor petty stats, rather an attempt to demonstrate the inexorable march of technology and illustrate that we are nowhere near the end of the path.

Kelly also gets full points for offering predictions of what the future will look like – and acknowledging both that futurists are almost always comically wrong and he is essentially guaranteed to be incorrect. This leads to paragraphs like the following:

If you’d like to have a vivid picture of someone interacting with a portable device in the year 2050, imagine them using their eyes to visually “select” from a set of rapidly flickering options on the screen, confirming with lazy audible grunts, and speedily fluttering their hands in their laps or at their waist. A person mumbling to herself while her hands dance in front of her will be the signal in the future that she is working on her computer.

If you had to distill what this means for humanity, Kelly attempts to summarize it in a few sentences:

We are marching inexorably toward firmly connecting all humans and all machines into a global matrix. This matrix is not an artifact, but a process. Our new supernetwork is a standing wave of change that steadily spills forward new arrangements of our needs and desires.

A key notion here is that of a “matrix.” This is a reference to networks and Joshua Ramo wrote The Seventh Sense to describe networks and their implications for the K Street crowd.

His thesis is that networks represent a fundamental change in the distribution of power. A small group of terrorists called ISIS can use networks to take on all of Western society in a completely asymmetrical game. ISIS is small and geographically isolated but as they tap into networks like Facebook, Twitter, or Telegram a phase change occurs and a more complex organization with disproportionate reach emerges.

Ramo has one critical insight for policy makers: whoever has the biggest network wins. The reason, is that networks compress time, enabling the biggest network to sustain its advantage as its members save time, creating a virtuous cycle where the big get bigger.

This is a simple sentence but it has profound implications. Like Kelly, Ramo offers a few speculative suggestions, but his are focused on how America can create networks to maintain political advantage. (As a reader I love these; speculative narratives are a great mechanism for reinforcing what the author means)

Examples? Retool the NIH and global affiliates to create a network of learning centers. Membership in the network means you share info and get access to that of others. Or have the State Department create a digital currency (a la Bitcoin) backed by US dollars (not a al Bitcoin). Give the digital currency directly to people in need and simply cancel digital dollars if taken by middlemen/corrupt officials. The network trumps all and no physical dollars need ever leave America.

The book is thought-provoking as it is the first attempt I have ever seen to apply the lessons of the Internet to government policy. We’ve seen the use of the Internet to get politicians elected but I’d argue it hasn’t materially changed their policy. Ramo’s book portends a networked political future.

This networked future isn’t just complicated, it’s complex. Samuel Arbesman’s book Overcomplicated outlines what this means and how to handle it.

A complicated system has a lot of parts but is predictable (a jet engine) whereas a complex system has feedback loops between the components making it unpredictable (the weather). He shows how systems everywhere – the stock market, air traffic control, Toyota’s brake systems – have accumulated so much interrelated cruft that complex systems pop up everywhere.

He cites Danny Hillis (a favored practitioner and philosopher of all the authors) in stating: “Our technology has gotten so complex that we no longer can understand it or fully control it. We have entered the Age of Entanglement. . . . Each expert knows a piece of the puzzle, but the big picture is too big to comprehend.”

Arbesman cites example after example of modern complexity and then offers an approach for managing it. Essentially, look to biological systems and embrace the fact that you will not know how everything works:

We must work to maintain two opposing states: mystery without wonder and wonder without mystery. The first requires that we strive to eliminate our ignorance, rather than simply reveling in it. And the second means that once we understand something, we do not take it for granted.

If this sounds a little Zen-like, then you will hate the recommendations of Ito and Howe in Whiplash. They offer nine principles – each of which is almost a koan – to navigate the exponential age. Emergence over authority. Pull over push. Compasses over maps. Practice over theory. Resilience over strength. etc.

The lens through which they see the world is the MIT Media Lab; Ito is the director and Howe a visiting scholar. They consider it-not unreasonably-a window into a possible model for the future and have tried to distill what they have learned into a series of principles.

There principles are definitely the most controversial part of any of the four books mentioned in this post. Since they are offering a way to navigate the future they lack the tested empiricism of the other authors. You are left with a sense of “trust me” with some of their recommendations but I see several of them embedded in what has made my employer successful so I am more than willing to give them the benefit of the doubt.

I appreciate the authors’ humility when they closed their book with an admission that they’re not prophets but guides:

We’re not trying to sell you a way to organize your workdays or an exercise regimen, and we’re definitely not trying to make you believe in our vision of the future, because we don’t have one, other than a ﬁrm belief it will be very, very different from the world we inhabit right now. We do have an argument to make: Innovation isn’t about learning how to use social media to generate sales leads. And modifying a business for a networked globe will require more than buying fancy teleconferencing gear for your management team. Instead, we think it requires a deeper, more fundamental shift: an entirely new mode of thinking-a cognitive evolution on the scale of a quadruped learning to stand on its hind two feet.

There is an apocryphal Chinese curse that “may you live in interesting times.” We certainly live in interesting times and I’d encourage you to embrace it. These four books will help you do so.

Debunking the “Steve Jobs Ignored Customers” Myth

Today at work we were fortunate enough to have Mark Hurst come in and talk about his updated book Customers Included. It’s all about working backwards from customers’ needs – which is a pretty easy sell in Amazonland (see “Customer Obsession“) – and hence recommended reading.

What I most enjoyed about his talk was his thorough destruction of the pernicious “Steve Jobs didn’t listen to customers so I don’t need to” myth. Few things in life are as soul-crushing as hearing someone (usually someone who has never created anything) tell you all about how they don’t need to listen to customers because Steve Jobs didn’t.

Any person who has spent any time trying to create new products knows in their bones that if you’re going to be successful, you’ve got to have some key insight as to what customers want – and the most successful way of doing that is working backwards from a customer need.

If the greatest trick the devil ever pulled was convincing the world that he didn’t exist, then Steve Jobs’ Kaiser Soze impression was convincing folks that he didn’t listen to customers. Fortunately, Mark Hurst is here to help us dive deep and discover the truth.

Article #1 – The 1997 WWDC

1997 was not a great year to work at Apple. They were almost out of money and turned to Steve as the only guy who could (or would) save the company. At the 1997 WWDC he did a fireside chat with Mac developers – who are the most invested people in the entire Mac ecosystem. It’s their livelihood and they’re the most diehard converts. So, it was probably a bit of a surprise when this question got asked:

Here’s the lightly edited transcript for you:

Questioner: Mr Jobs, you’re a bright and influential man. It is sad and clear that on several counts you’ve discussed you don’t know what you’re talking about… And when you’re finished with that, perhaps you can tell us what you have personally been doing for the last seven years.

If you’re watching the video, you’ll notice a quick joke as a response followed by a long, awkward pause as Steve tries to think of the following capable response:

One of the hardest thing when you’re trying to effect change is that people, like this gentleman are right, in certain areas. …The hardest thing, what does that fit in, to a cohesive, larger vision that’s going to enable you to sell eight billion, ten billion dollars of product a year. One of the things I’ve always found is that you’ve got to start with the customer experience and work backwards to the technology. You can’t start with the technology and figure out where you’re going to sell it. And I’ve made this mistake probably more than anybody else in the room. And I’ve got the scar tissue to prove it and I know that it’s the case. And as we’ve tried to come up with a strategy and a vision for Apple, um, it started with what incredible benefits can we give to the customer. Where can we take the customer. Not starting with, let’s sit down with the engineers and figure out what awesome technology we have and how are we going to market it. And I think that’s the right path to take.

There you have it: the greatest tech legend of recent years talking about working backwards.

Article 2: The Creation of the iPhone

So Jobs said he worked backwards, but how did he do that when creating products that never existed before? Well, the iPhone came about by working backwards from the compromises made by consumers in using existing (circa 2005) cellphones.

From the Walter Isaacson biography (page 466):

He [Jobs] began talking to Ed Zander, the new CEO of Motorola, about making a companion to Motorola’s popular RAZR, which was a cellphone and digital camera, that would have the iPhone built in. Thus was born the ROKR. It ended up having neither the minimalism of the iPod nor the convenient slimness of a RAZR. Ugly, difficult to load, and with an arbitrary hundred-song limit, it had all the hallmarks of a product that had been negotiated by a committee which was counter to the way Jobs liked to work. Instead of hardware, software, and content all being controlled by one company, they were cobbled together by Motorola, Apple, and the wireless carrier Cingular. “You call this the phone of the future?” Wired scoffed on its November 2005 cover.

The next paragraph gets to the interesting part:

Jobs was furious. “I’m sick of dealing with these stupid companies like Motorola,” he told Tony Fadell and others at one of the iPod product review meetings. “Let’s do it ourselves.” He had noticed something odd about the cell phones on the market: They all stank, just like portable music players used to. “We would sit around talking about how much we hated our phones,” he recalled. “They were way too complicated. They had features nobody could figure out, including the address book. It was just Byzantine.” George Riley, an outside lawyer for Apple, remembers sitting at meetings to go over legal issues, and Jobs would get bored, grab Riley’s mobile phone, and start pointing out all the ways it was “brain-dead.” So Jobs and his team became excited about the prospect of building a phone that they would want to use.

This is classic working backwards from the customer. Identify all the compromises faced by the customer. Heck, test your hypotheses with your target market (rich lawyers with lots of disposable income for phones) before you build anything. But don’t pretend that this was just willed out of thin air.

Hurst called out one important corollary to the above: when deciding what the compromises are, there’s no mention of “multi touch” or “gorilla glass,” etc. All those fancy design features came later once the team had identified the customer problems. They were the “how” to solve the “what” which was crappy address books, etc.

Thanks Mark for a thorough debunking of one of Silicon Valley’s most pernicious, lasting myths.

What’s Getting Backed on Kickstarter: Technology Edition

I’m a huge Kickstarter fanboy. The creativity that they’ve unleashed is mind-boggling and embodies exactly what the Internet is capable of.

The projects I’ve backed tend to skew tech-heavy and I recently wondered if there were any identifiable trends as to what sort of tech projects get backed. I hear all the time about companies that started with a Kickstarter campaign; are there any patterns?

Kickstarter is really open about what projects get backed (note that it’s not clear how accurate the project counts are; they seem to appear/disappear/dramatically change based on when you access the site), so I went out and scraped some data (code here).

I managed to get data on 1700 successful tech projects, which raised a cumulative total of $212,472,913. That’s an average of $124,984 per project-but this is no even distribution. The winners (like the Pono music player, Reading Rainbow or Zano drone) raise millions while Kymira smart sports apparel is limping in at $8,000 or so (but still successful; kudos). The median raised is $45,992.

I wrote some code to try and cluster these projects and found a few categories that people like to back:

Physical Computing: Arduino clones and shields; Raspberry Pi accessories galore. Examples include Microview, RFDuino and the Touch Board.

3D Printers: Every type you can imagine, including the Micro, the Form1 and the 3Doodler.

Home Automation: Smart plugs, dimmers, remotes – Kickstarters want a connected home. Sample projects are Ube Wifi dimmer, the NEEO remote and the Ninja Sphere controller.

Lighting: Make it glow-whether lights attached to your stereo, fancy bike lights or an enhancement to your GoPro. Examples are the Notti Smart Light, Lume Cube flashbulbs or Playbulb candles.

Phone Accessories: Anything that can pimp your phone. Check out the Jorno foldable keyboard, Thermodo thermometer or Chipolo item finder.

Solar Powered Gizmos: Kickstarter backers seem to really want to take their electronics outside. Witness the WakaWaka Base, SPOR and Solarpod Pyxis chargers/lights.

Here’s how many projects fall into each category:

That’s a high of 433 for Home Automation versus 181 3D printers.

(Hate that there are no numbers in these graphs? I do, but can’t figure out how to add labels. All the raw data is here.)

The total funds raised varies dramatically across the categories ($M):

Surprisingly, Physical Computing has clocked almost $50M ($46.4M), closely followed by 3D Printers at $41.4M. Solar Power is half this at $21.5M.

There’s a similar difference in the the amount raised per category-both the average and median ($K):

On a per-project basis, 3D Printers have captured peoples wallets (most likely have a much higher per unit cost than other projects) and clock in with a $229.0K/$79.9K average/median raise. This median is almost as high as the average for solar power projects: $89.4K/$42.3K average/median.

A couple of closing thoughts:

I was amazed at how much money has gone towards backing Physical Computing. I imagine that most of these devices were bought by geeks to make geeky devices, so I’m guessing that we’re only at the starting of a big revolution in Internet-connected devices
There was no major cluster for robotics or drones. This surprised me as I would have guessed more based on the buzz in the press. Big difference between what is bought vs. what is talked about
Some of the major success stories (like Pono or Reading Rainbow) don’t fall into an of these categories. I don’t yet know how to interpret these “one hit wonders” but its interesting to think about why they succeeded as a product but didn’t launch a category

The Blinky Box

Last year this awesome story by Miria Grunick appeared in one of my feeds. I was immediately inspired to create one for Cam and this year I finally built him his very own Blinky Box.

I made a few changes to her design: a switch in the upper left toggles a solid color vs. flashing lights mode. Plus the knob in the upper right rapidly changes the frequency of flashing.

I had the top laser cut as I have no idea how to cut a plastic box. (If you’re in Seattle, get it done at Metrix Create Space so you can watch them do it – and it only costs $7 or so)

Here’s a video of it in action:

If you want to make your own, the code’s on Github; feel free to fork as it’s definitely not optimal. I also included the .SVG for the laser cutting; I recommend making the holes for the arcade buttons a little bigger as mine stick – 0.5mm should be enough.

You can get the bill of materials on Miria’s Github repo; if you make my version you’ll also need a switch.

For anyone who tries it and gets stuck, here are some photos of some of the major breadboarded items. Should be enough to get you over any hump: