Arizona State University researcher talks about the approaches and impact of NASA Acres and NASA Harvest to tackle planetary-scale challenges using satellite data.
Hannah Kerner is an assistant professor of computer science at Arizona State University (ASU). She leads the development and deployment of AI and machine learning tools for the NASA Acres and NASA Harvest consortia. Here's an excerpt of an article on Full Circle, the magaxine of Ira A. Fulton Schools of Engineering.
Lack of labeled data is a common challenge to interpreting remote-sensing data worldwide. The reason is that creating labels is slow and expensive, especially in vast and remote agricultural regions.
“You can’t just look at a satellite image and determine what is growing at a given site,” Kerner says. “For example, a field of corn and a field of soybeans are going to appear the same to our eyes in that image.”
Someone needs to physically visit each location to record what is being cultivated. Also, identifying crops requires specialized knowledge beyond the training of most computer scientists and their students. Accurate labels therefore require informed input from farmers or agricultural authorities, which in turn requires time to establish relationships and trust with them.
Kerner and her colleagues, including agricultural remote sensing expert and NASA Harvest Africa lead Catherine Nakalembe, wanted to create a faster way to gather this ground-level, or “ground-truth,” labeled data. They developed a method called Street2Sat in which GoPro cameras were mounted on cars and motorcycles driven on roads alongside farm fields to capture geotagged and timestamped images, similar to Google Street View. These images were then supplied to computer vision algorithms to automatically detect crops and create a ground-truth crop type label that could be paired with corresponding satellite data for each location.
“Most of automating how we processed these surveys went as expected. But the difficult part was managing the distance offset,” Kerner says. “We didn’t want location data for the vehicle-mounted cameras on the roads. We wanted to record the locations of the crops that were photographed by those cameras.”
Kerner says off-the-shelf computer vision models for depth estimation do not work very well for open environments where objects might be far away. She says they work for objects less than 4 meters from a camera, but beyond that range — such as 10 to 12 meters away, as crops often are from a road — the models generated significant errors. Kerner and her team tried a variety of approaches to correct that problem, but without success.
“Then we took a step back and asked what we were really trying to do here — as human beings — to solve this problem,” she says. “We’re looking in Google Earth at where this camera is located on a road, and we know it’s pointed in ‘this’ direction. It’s ‘this’ field that we want the model to predict. So, maybe we should just have the model solve that specific problem.”
They then trained a segmentation model to use an image with an arrow on it to predict which field it was pointing out, and it turned out to be a successful approach. Enough so that the team has now used their Street2Sat pipeline to process more than 5 million images from countries including Kenya, Nigeria, Rwanda, Tanzania, Uganda and the United States.
