Constant Coverage: Nexar’s Consistent, Visual Inventory of Roads

Nexar’s software architect, Robert Harrison, explains what stands behind Nexar’s Constant Coverage Technology.

Providing consistent, quality coverage of roads and highways across the US means providing the necessary information that helps us understand the world around us. Collecting data from our car camera network - through an annonymization process - enables us to apply our vast data repository to various applications that generate insight about blocked lanes, road signs, free parking spots and dangerous road conditions.
At Nexar, meeting these needs means being ahead of the curb. A great example for that is our Constant Coverage methodology, where our dataset quality is improved on an ongoing basis.
In this post, I’ll explain how we at Nexar keep this massive coverage maintained with fresh, quality images.

But First, Terminology
Nexar stores our coverage by using “Hex Seg” which is a combination of “Hexagon- Segments”:
The combination of a Uber’sH3 geospatial indexing system (where we “hold” our images of the road within a Res 12 Hexagon) to OSM road segments.

While the Hexagon at Res 12 is 20 meters across, a “Hex Seg” might only be 5 meters long, resulting in multiple Hex Segs per Hexagon. Conversely, a “Hex Seg” could be 200 meters long, spanning 10 distinct Hexagons.
Nexar has imagery in millions of Hexagons across the US, and some Hexagons contain up to 1,000 images.

The Constant Index of Quality Frames  
The constant coverage method provides an ongoing, quality-updated repository.
It requires keeping an account of the coverage, where for each Hex Seg we’re keeping a stack of frames built up from the different inference types that we run, and is limited by a pre-defined cap.
Once the stack is filled up it will rotate: as new frames come only images that are considered “better” will replace an existing frame within the Hex Seg.

Let’s get into the technical details:
Nexar’s index of the latest ‘good’ frames per Hexseg means the ability to hold a limited number of ordered frames per Hexseg. New frames should be inserted into the index and old frames bumped off to maintain a bounded size.
There are about ±3 million updates a day - a mixture of insert, delete and update operations on the index. The frames added and bumped are based upon a frame’s ‘score’ - the ‘Universally Good’ frames filter:

• Speed is lower than 30 miles per hour
• Camera heading is within 40 degrees of car heading
• A minimal localization error
• Visual quality, includes whether an image is flipped or not, corrupted or not, does it have a minimal view of the dashboard and maximizes the external view from the car.

If an image doesn’t pass the filter, it will not be offered to the index. Once an image passes the filter, the frame is then scored based on these 3 criteria- where the better the score the better chances of the image to be added to the index:

• Lighting/time-of-day: A mixture of daytime and non-daytime frames in the index, biased towards daytime.
• Image quality: based on the average of all of the visual characteristics of the image described above. The better the image quality - the sharper it is, the more stable it is and better shows the exterior- the higher the score.
• Timestamp: When was the frame taken, as newer is better

The score is calculated using a weighted geometric mean, which has a huge advantage of allowing multiple properties to be combined with different scales and without the need for normalization. If an incoming frame has a lower score than the lowest existing item in the Hexseg, it should be ignored. Otherwise it will be inserted and the lowest existing item bumped from the Hexseg. Each Hexseg should contain about 50 frames, divided between different inference types and constantly updated.
The index is stored in a Scylla Database, to provide excellent query performance by hexseg for online API usage. This means faster performance, enabling us to select a big amount of data in parallel by key - in this case hexseg. This is now the database serving Nexar’s Virtual Camera  app on the CityStream platform.

What We See in The Hex Segs

Here’s an example of Yosemite park, where you can actually see the park:

This image from our inventory, as well as similar ones from this area, emphasizes which images are considered “better” ones: the quality of the frame is higher based on being universally good, as well as a frame that scores highly, in addition to being relatively recent. In a city which we “see” often, such as New York, frames will be replaced much more often, and not include images as old as this Yosemite one.
We also use a heat-map (based on a Keppler map), where the color shades show how many frames we hold per road segment. The darker it gets the more frames we have. This gives a nice visual indication on the density of our imagery in different areas as well as the coverage we have in this area.

The Future: Dynamic Framework on Demand
After we understand the impact of constant coverage, we plan to make this ability dynamic, providing more data and even better insights. Currently, we limit frame uploads to the inventory, but with Dynamic Frame on Demand which we plan to add later, we will be able to get more frames, improve their quality faster, and close gaps - all without affecting costs and even reducing them.
Such an improvement of upload economics, on a national scale, means a massive leap and a step closer to creating a digital twin of the world around us.