A new Computer Vision Model including 4,717 new taxa

It’s September, 2022, and we’ve released a new computer vision model for iNaturalist. This follows updates in August and April 2022. The iNaturalist website, iNaturalist mobile apps, and API are all now using this new model. Here’s what’s new and different with this change:

• It includes 65,000 taxa (up from 60,000)
• It is the second model to be trained in our new, faster approach

Taxa differences to previous model

There are 5,811 taxa in the new model (v1.2) that weren’t in the old model (v1.1).

4,717 of those represent newly added choices. For example, of the 3 species of Carpillus, the old model only included Spotted Reef Crab and Convex Crab whereas the new model also includes Batwing Coral Crab.

907 of those taxa represent more refined replacements. For example, the old model included the Crestless Curassow genus Mitu which contains 4 species of birds. None of these species had enough photographs to be included in the model. The new model includes the species Razor-billed Curassow as a more refined replacement for Mitu. Because genus Mitu was replaced by Razor-billed Curassow, the number of choices was not increased by this refinement.

Lastly, 187 of thetaxa in the new model but not in the old model result from taxon changes. For example, in the old model Wedge-rumped Storm-Petrel was represented by the taxon Oceanodroma tethys, but due to a taxon change the new model has replaced that taxon with Hydrobates tethys. This is the same species but in a different genus so again the number of choices was not increased by this refinement.

There were also 1,165 species in the old model which are not in the new model. 31 of these were lost because of a decrease in the amount of data. For example, the old model included 2 species of genus Aplysilla, Encrusting Rose Sponge (Aplysilla rosea) and Aplysilla glacialis. However, due to new identifications added by the community, many of the observations that were identified as Encrusting Rose Sponge now represent other taxa. As a result, the new model no longer includes this taxon as a node.

As described above, there are also taxa in the older model not in the newer model because they were refined (e.g. genus Mitu) or because they were the inputs of taxon changes (e.g. Oceanodroma tethys)

The charts below summarize these taxa. We can use these categories to filter out just this set of 4,717 new taxa added to the new model that aren’t the result of refinements or taxon changes.

By category, most of these 4,717 new taxa were insects and plants.

Here are species level examples of new species added for each category:

• Other Animalia
• Mollusca
• Plantae (sample of 200)
• Amphibia
• Protozoa
• Mammalia
• Actinopterygii
• Insecta (sample of 200)
• Chromista
• Arachnida
• Aves
• Fungi
• Reptilia

Click on the links to see these taxa in the Explore page to see these samples rendered as species lists.

You can find an entire list of all the species added to the new model here.

Remember, to see if a particular species is included in the currently live computer vision model, you can look at the “About” section of its taxon page.

This is our new vision model release tempo

Our previous goal for releasing models was twice a year, and we struggled to even meet that. However, with the new transfer learning approach that vastly speeds up training, we now plan to release a model every month, with the caveat that our schedule could grow longer as the number of photos continues to grow. This means that there will be much less taxonomic drift between the taxonomy that the model knows about and the taxonomy at the time the model is showing suggestions to a user.

We will still be training a full model once or twice a year, which we’ll then do transfer learning from in order to make release models. Extra hardware provided by NVIDIA and donations from the iNat community have made it possible to have a training strategy that combines both full model training and transfer learning.

Future work

First, we are still working on new approaches to improve suggestions by combining visual similarity and geographic nearness. We still can’t share anything concrete, but we are getting closer.

Second, we’re still working to compress these newer models for on-device use. The in-camera suggestions in Seek continue to use the older model from March 2020.

We couldn't do it without you

Thank you to everyone in the iNaturalist community who makes this work possible! Sometimes the computer vision suggestions feel like magic, but it’s truly not possible without people. None of this would work without the millions of people who have shared their observations and the knowledgeable experts who have added identifications.

In addition to adding observations and identifications, here are other ways you can help:

• Share your Machine Learning knowledge: iNaturalist’s computer vision features wouldn’t be possible without learning from many colleagues in the machine learning community. If you have machine learning expertise, these are two great ways to help:
• Participate in the annual iNaturalist challenges: Our collaborators Grant Van Horn and Oisin Mac Aodha continue to run machine learning challenges with iNaturalist data as part of the annual Computer Vision and Pattern Recognition conference. By participating you can help us all learn new techniques for improving these models.
• Start building your own model with the iNaturalist data now: If you can’t wait for the next CVPR conference, thanks to the Amazon Open Data Program you can start downloading iNaturalist data to train your own models now. Please share with us what you’ve learned by contributing to iNaturalist on Github.
• Donate to iNaturalist: For the rest of us, you can help by donating! Your donations help offset the substantial staff and infrastructure costs associated with training, evaluating, and deploying model updates. Thank you for your support!

Posted on September 13, 2022 07:29 PM by alexshepard