New computer vision model

We’ve released a new computer vision model for iNaturalist. This is our first model update since April 2022. The iNaturalist website, mobile apps, and API are all now using this new model. Here’s what’s new and different with this change:

• It includes 60,000 taxa (up from 55,000)
• It was trained using a different approach than our previous models, which made it much faster to train

To see if a particular species is included in this model, you can look at the “About” section of its taxon page.

It’s bigger

Our previous model included 55,000 taxa and 27 million training photos. The new model was trained on over 60,000 taxa and almost 30 million training photos.

It was trained using a transfer learning strategy

During previous training runs, our strategy was to train the entire model on the dataset. This means that all of the model weights were candidates for being updated, in order to learn the most efficient and useful visual features for making suggestions for the taxa in that dataset. When training this model, we froze most of the model weights (thereby freezing the visual feature extraction) and only trained the very last layer of the model, the layer that makes the taxa suggestions. This is a machine learning strategy known as transfer learning.

One way to think about this is to imagine that someone was asked to learn all about different kinds of cars. Later, that person was asked to differentiate between two different kinds of pickup trucks, but only using distinguishing characteristics they learned from their study of cars (for example, color, size, visual shape, branding, engine size, etc), without learning anything new about pickup trucks (for example bed capacity, towing limits, etc). Chances are, that person could distinguish between most kinds of trucks without needing to learn anything new specifically about pickup trucks. They may not perform as well as someone who learned about trucks from the beginning, but they have strong foundational knowledge to draw upon for the task.

Our new model was trained using a transfer learning strategy. We used the internal weights and visual features from our previous model which was trained on 55,000 taxa. The advantage of this approach is that we didn’t need to learn all of those internal model weights and visual features again, so training was quite a bit faster. It’s only been four months since our last model was released, which is the shortest time between model releases so far.

As with the pickup truck analogy, it could be that this model trained with the transfer learning approach is slightly less accurate overall than if we had trained the entire model again. However, in our testing this new model appears to achieve nearly the same accuracy as the previous model while containing more taxa. Our plan going forward will be to spend the time fully training a model about once a year to maximize accuracy with new photos and taxa, and to use the faster transfer learning approach in between full training runs so we can release models more frequently than we have in the past.

Future work

First, we are still working on new approaches to improve suggestions by combining visual similarity and geographic nearness. We still can’t share anything concrete, but we are getting closer.

Second, we’re still working to compress these newer models for on-device use. The in-camera suggestions in Seek continue to use the older model from March 2020.

We couldn't do it without you

Thank you to everyone in the iNaturalist community who makes this work possible! Sometimes the computer vision suggestions feel like magic, but it’s truly not possible without people. None of this would work without the millions of people who have shared their observations and the knowledgeable experts who have added identifications.

In addition to adding observations and identifications, here are other ways you can help:

• Share your Machine Learning knowledge: iNaturalist’s computer vision features wouldn’t be possible without learning from many colleagues in the machine learning community. If you have machine learning expertise, these are two great ways to help:
• Participate in the annual iNaturalist challenges: Our collaborators Grant Van Horn and Oisin Mac Aodha continue to run machine learning challenges with iNaturalist data as part of the annual Computer Vision and Pattern Recognition conference. By participating you can help us all learn new techniques for improving these models.
• Start building your own model with the iNaturalist data now: If you can’t wait for the next CVPR conference, thanks to the Amazon Open Data Program you can start downloading iNaturalist data to train your own models now. Please share with us what you’ve learned by contributing to iNaturalist on Github.
• Donate to iNaturalist: For the rest of us, you can help by donating! Your donations help offset the substantial staff and infrastructure costs associated with training, evaluating, and deploying model updates. Thank you for your support!

Posted on August 19, 2022 12:43 AM by alexshepard