– Nikhil Naik
Transcript from TEDx Beacon Street November, 2015

Here we see a picture of a little girl walking to school in one of the many rundown neighborhoods in America. The windows are boarded up. The walls are filled with graffiti, and there is trash on the street. It makes you wonder how living in this neighborhood affects her life, her health, or her performance in school. It makes you wonder if the windows were cleared, if the trash was cleaned, would her life be any better – even if she had the same family and the same school.

Screen Shot 2016-02-23 at 11.05.49 AM

Social scientists have grappled with the following question for more than a century. How does the appearance of a neighborhood affect the people who live there? Inspired by the famous broken windows theory studies have linked neighborhood appearance to criminal behavior. Similar studies have shown that neighborhood appearance affects even health, mobility, and school dropout rates.

Screen Shot 2016-02-23 at 11.17.08 AM
These studies are limited in the scope due to a lack of data on how cities look. At the moment, the researchers spend many months doing field surveys to measure how neighborhoods look but the field surveys can only cover so much area. They are limited to a few neighborhoods and a handful of cities at most. So, how do we increase the speed and the scale of these field surveys? The solutions is to replace people surveying streets with computers looking at images from Google Street View.

Screen Shot 2016-02-23 at 11.19.58 AM

In the past decade, more than 3,000 cities from 91 countries have been photographed by Google Street View alone. It is now possible to virtually field survey many, many cities without leaving your home. Specifically we want to teach the computer to be able to look at a picture of a street block and tell us how safe it would look to a human observer. We can do this using machine learning.

Screen Shot 2016-02-23 at 11.46.29 AM

In machine learning, the computer learns to do a task from examples. The first goal is to create examples of how safe street blocks look to humans. To get these examples, we created an online game called Place Pulse. In this game, we show people images from Street View side by side, and ask them to choose one which looks safer. This game was online for about four months. It went viral. We had 3,000 images from New York and Boston. Eight thousand participants from more than 90 countries played this game and we received more than 200,000 clicks on the site.

Screen Shot 2016-02-23 at 11.20.23 AM

Now we can take these clicks and assign scores between zero and 10 to images based on how safe they look. The higher the score, the safer the image looks to a majority of our users. We call this score Streetscore. Now that we have these examples, we can train the computer to do this task. The computer learns from the shapes, textures, and colors present in the image and learns to assign a score to a new image it has never seen before.

Screen Shot 2016-02-23 at 11.20.40 AM

The computer is quite good at it. It assigns low scores to street blocks that have empty streets, rundown buildings, and industrial warehouses. On the other hand, it assigns a high school to suburban houses with manicured lawns and clean streets lined with trees.

Screen Shot 2016-02-23 at 11.20.48 AM

Not just that, we can look at the same location and take pictures from two different times and study how the place has changed. We can do this by simply taking the difference of the streetscore of the place at two different times. This can be used to identify places undergoing gentrification or decline, which is very important at the moment in cities.

Screen Shot 2016-02-23 at 11.20.55 AM

So, we use streetscore to calculate the perception of safety of more than 2 million street blocks from 21 American cities to create the largest database of how cities look. We can use this data to create incredibly detailed maps of how neighborhoods look, like this map of New York City. Now researchers across the country are using our data to measure how neighborhood appearance impacts your health, crime, mobility, and education.

Screen Shot 2016-02-23 at 11.21.04 AM

This tool is useful to city governments as well. City governments spend millions of dollars trying to clear graffiti, fix sidewalks, and plant trees to improve their neighborhoods. Streetscore can help them make data-driven decisions on how the roads look and assign the resources accordingly. Streetscore can also help them to measure the impact of neighborhood improvement initiatives by using images before and after a neighborhood fixup.

Screen Shot 2016-02-23 at 11.33.51 AM

Tools that can use computers to measure city appearance are even more useful in developing countries. The governments like the resources to measure the condition of their cities. As they say, what you cannot measure, you cannot understand. What you cannot understand, you cannot improve.

Screen Shot 2016-02-23 at 11.34.19 AM

Now, our goal is to teach the computer to be able to look at the pictures from a city in a developing country and make inference about it. To be able to do so, we created a new game that has images from 56 countries across six continents. More than 200,000 people have played this game so far and contributed 1.3 million clicks, but we need more data. We invite you to visit this website and help us play the game.

My work with streetscore is helping improve our understanding of how neighborhood appearance impacts people. It is also helping us to study gentrification, and decline of cities. In the long run, big visual data and machine learning can help us improve the life of more than 3.5 billion people who live in cities across the world.

We need data! Play the game:
See safety ratings:

Read the paper:

Screen Shot 2016-02-23 at 11.36.55 AMNikhil Naik is a PhD student at the MIT Media Lab. He is interested in computer vision and computational imaging. His recent work focuses on understanding cities using computer vision. His research has been featured in The Atlantic, The Economist, The Guardian, and Harvard Business Review, among others. He is also a decent ping-pong player.


1 2 3
September 13th, 2016

Ramesh Raskar Lemelson-MIT 2016 Winner

Congratulations to Professor Ramesh Raskar and the Camera Culture Group  – Winner of the 2016 Lemelson-MIT prize Ramesh Raskar and […]

September 8th, 2016

Can computers read through a book page by page without opening it?

Terahertz time-gated spectral imaging for content extraction through layered structures A. R. Sanchez, B. Heshmat*, A. Aghasi, M. Zhang, S. […]

November 21st, 2018

Society of Autonomous Vehicles – Part 1

This is part 1 of a multi-part blog series about the Society of Autonomous Vehicles course held in Spring of […]

March 20th, 2018

Seeing Through Realistic Fog

A technique to see through dense, dynamic, and heterogeneous fog conditions. The technique, based on visible light, uses hardware that […]

March 29th, 2017

Efficient Lensless Imaging with a Femto-Pixel

Lensless Imaging with Compressive Ultrafast Sensing Guy Satat, Matthew Tancik, Ramesh Raskar Traditional cameras require a lens and a mega-pixel […]

September 28th, 2016

How to see through tissue

All Photons Imaging Through Volumetric Scattering Guy Satat, Barmak Heshmat, Dan Raviv, Ramesh Raskar We demonstrate a new method that […]

September 12th, 2016

The World is Our Lab

by Roger Archibald Photo credit: John Werner Ramesh Raskar, head of the Media Lab’s Camera Culture Group, takes measure of […]

March 27th, 2016

Handheld 3D Imager to visualize features in the throat like tonsils!

3D visualization of oral cavity and oropharyngeal anatomy may play an important role in the evaluation for obstructive sleep apnea […]

February 24th, 2016

How to use computer vision to improve cities

– Nikhil Naik Transcript from TEDx Beacon Street November, 2015 Here we see a picture of a little girl walking […]

February 12th, 2016

Optical Brush

Optical brush is an open-ended bundle of optical fibers that is enabled with time of flight technology to image and […]

December 3rd, 2015

Innovating for Billions – Ramesh Raskar’s UIST Keynote

28th ACM User Interface Software and Technology Symposium Charlotte, NC November 8-11, 2015 Watch the presentation on Youtube HERE Keynote […]

December 2nd, 2015

Making 3-D imaging 1,000 times better

Algorithms exploiting light’s polarization boost resolution of commercial depth sensors 1,000-fold. See MIT coverage here. Read more about the work […]

November 26th, 2015

Engineering Health Class Final Presentations on 4 Dec

Great projects and demos- wearables, 3D imaging, novel stethoscopes, imaging the eye, oral imaging.  Followed by health night with guest […]

November 2nd, 2015

Engineering Health Class featured in Medtech

Our Engineering Health class was featured in Medtech Boston. Read the full article here.

October 26th, 2015

OPEN POSITIONS: Technical Assistants

Technical Assistant # 1 Date: December-4-2015 The Camera Culture group at the MIT Media Lab focuses on making the invisible […]

October 15th, 2015

Time-of-Flight Microwave Camera to See Through Walls

New Camera Culture research in Nature Scientific Reports shows a prototype time of flight camera working at microwave frequencies. In […]

September 30th, 2015

How to Make a City Smart? The Indian Context

By: Pranav Chandrasekaran   Introduction – by Ramesh Raskar, Associate Professor Media Arts and Sciences, Head of the Camera Culture […]

September 11th, 2015

Kumbh Mela – The World’s Largest Moving City

Kumbh Mela in Nashik Article and Photography by John Werner   “The world’s largest city has no permanent address.”   […]

September 8th, 2015

Engineering Health Fall 2015 Course

MAS.S62: Join us for the first class on Friday, 11 Sept, 1-4 pm in E15-341(Click to go to class webpage)

June 26th, 2015

Vahe Tahmazyan Graduates with Best Thesis