A new Computer Vision Model including 4,717 new taxa

It’s September, 2022, and we’ve released a new computer vision model for iNaturalist. This follows updates in August and April 2022. The iNaturalist website, iNaturalist mobile apps, and API are all now using this new model. Here’s what’s new and different with this change:

  • It includes 65,000 taxa (up from 60,000)
  • It is the second model to be trained in our new, faster approach

Taxa differences to previous model

There are 5,811 taxa in the new model (v1.2) that weren’t in the old model (v1.1).

4,717 of those represent newly added choices. For example, of the 3 species of Carpillus, the old model only included Spotted Reef Crab and Convex Crab whereas the new model also includes Batwing Coral Crab.

907 of those taxa represent more refined replacements. For example, the old model included the Crestless Curassow genus Mitu which contains 4 species of birds. None of these species had enough photographs to be included in the model. The new model includes the species Razor-billed Curassow as a more refined replacement for Mitu. Because genus Mitu was replaced by Razor-billed Curassow, the number of choices was not increased by this refinement.

Lastly, 187 of thetaxa in the new model but not in the old model result from taxon changes. For example, in the old model Wedge-rumped Storm-Petrel was represented by the taxon Oceanodroma tethys, but due to a taxon change the new model has replaced that taxon with Hydrobates tethys. This is the same species but in a different genus so again the number of choices was not increased by this refinement.

There were also 1,165 species in the old model which are not in the new model. 31 of these were lost because of a decrease in the amount of data. For example, the old model included 2 species of genus Aplysilla, Encrusting Rose Sponge (Aplysilla rosea) and Aplysilla glacialis. However, due to new identifications added by the community, many of the observations that were identified as Encrusting Rose Sponge now represent other taxa. As a result, the new model no longer includes this taxon as a node.

As described above, there are also taxa in the older model not in the newer model because they were refined (e.g. genus Mitu) or because they were the inputs of taxon changes (e.g. Oceanodroma tethys)

The charts below summarize these taxa. We can use these categories to filter out just this set of 4,717 new taxa added to the new model that aren’t the result of refinements or taxon changes.

By category, most of these 4,717 new taxa were insects and plants.

Here are species level examples of new species added for each category:

Click on the links to see these taxa in the Explore page to see these samples rendered as species lists.

You can find an entire list of all the species added to the new model here.

Remember, to see if a particular species is included in the currently live computer vision model, you can look at the “About” section of its taxon page.

This is our new vision model release tempo

Our previous goal for releasing models was twice a year, and we struggled to even meet that. However, with the new transfer learning approach that vastly speeds up training, we now plan to release a model every month, with the caveat that our schedule could grow longer as the number of photos continues to grow. This means that there will be much less taxonomic drift between the taxonomy that the model knows about and the taxonomy at the time the model is showing suggestions to a user.

We will still be training a full model once or twice a year, which we’ll then do transfer learning from in order to make release models. Extra hardware provided by NVIDIA and donations from the iNat community have made it possible to have a training strategy that combines both full model training and transfer learning.

Future work

First, we are still working on new approaches to improve suggestions by combining visual similarity and geographic nearness. We still can’t share anything concrete, but we are getting closer.

Second, we’re still working to compress these newer models for on-device use. The in-camera suggestions in Seek continue to use the older model from March 2020.

We couldn't do it without you

Thank you to everyone in the iNaturalist community who makes this work possible! Sometimes the computer vision suggestions feel like magic, but it’s truly not possible without people. None of this would work without the millions of people who have shared their observations and the knowledgeable experts who have added identifications.

In addition to adding observations and identifications, here are other ways you can help:

  • Share your Machine Learning knowledge: iNaturalist’s computer vision features wouldn’t be possible without learning from many colleagues in the machine learning community. If you have machine learning expertise, these are two great ways to help:
  • Participate in the annual iNaturalist challenges: Our collaborators Grant Van Horn and Oisin Mac Aodha continue to run machine learning challenges with iNaturalist data as part of the annual Computer Vision and Pattern Recognition conference. By participating you can help us all learn new techniques for improving these models.
  • Start building your own model with the iNaturalist data now: If you can’t wait for the next CVPR conference, thanks to the Amazon Open Data Program you can start downloading iNaturalist data to train your own models now. Please share with us what you’ve learned by contributing to iNaturalist on Github.
  • Donate to iNaturalist: For the rest of us, you can help by donating! Your donations help offset the substantial staff and infrastructure costs associated with training, evaluating, and deploying model updates. Thank you for your support!
Posted on September 13, 2022 07:29 PM by alexshepard alexshepard


So the cutoff is no longer must have 100 obs?
This has only 71, and is newly included - from your link

Posted by dianastuder about 1 year ago


@dianestuder - I think it was mentioned a while back it was 100 photos now rather than 100 obs

Posted by sbushes about 1 year ago

the threshold is kind of complicated and involves the presence of the community ID and keeping observations used for test, train, and validation sets separate, but it uses number of photos as the core threshold, not number of observations. So complications aside, 100 photos would be closer than 100 observations.

Posted by loarie about 1 year ago

Oh how I love this! Thanks so much for the detailed update! Love looking at the newly involved species! Some in there I have worked on as well... and found one or two I need to check out.. might be wrongly included. More food for ID-thought for me for the coming weekend :-)

Posted by ajott about 1 year ago

Thanks so much! Every species added helps the community out so much!

Posted by yayemaster about 1 year ago

Excellent job, I love seeing these granular statistics on which taxa were added or removed.

Posted by radrat about 1 year ago

I love reading these details about how the system works, and am very happy to see the more frequent updates.

Posted by sullivanribbit about 1 year ago

Wow, that was fast! Great to see, even if the choices presented fluctuate more frequently.

Posted by jdmore about 1 year ago

Great to have new taxa from splits, e.g. https://www.inaturalist.org/taxa/1286913-Saxicola-stejnegeri for what before cv suggested unrelated species and now only have this species as a suggestion!

Posted by marina_gorbunova about 1 year ago

Great, took a long time for the next update during COVID times, but I hope next updates are coming more frequent. And I love the links to the last two blog posts about the Computer Vision model, in case I missed one...

Posted by optilete about 1 year ago

@loarie we can't see how many photos per taxon?
The no of obs is on the taxon page and gives us a visible target. As a ballpark figure across iNat ... how many obs to get 100 pictures?

Posted by dianastuder about 1 year ago

Freaking hell yeah. I just checked on one one taxon that wasn't included in CV, that I have spent a good amount of time fishing out of IDs under a very similar looking taxon that was in CV, and it is now included in CV! Yeehaw. Crassula colligata movin' on up in the world. :)

Posted by grmorrison about 1 year ago

I'd like to know more about the "transfer learning approach", how that differs from full training, and how it eases the workload enough that it can be done more frequently. If it's sustainable, a monthly release cycle would be awesome!

Posted by xris about 1 year ago

dianestuder, we don't have an view that displays a number of photos count unfortunately. We do have this view which displays photos per taxon directly (as opposed to observations per taxon) https://www.inaturalist.org/taxa/555170-Dendropsophus-reticulatus/browse_photos
but it doesn't give a count and the display excludes photos non verifiable/captive etc photos which we do include in training if certain thresholds are met

Posted by loarie about 1 year ago

@xris you can read more about transfer learning in the previous release announcement here: https://www.inaturalist.org/blog/69193-new-computer-vision-model

Posted by alexshepard about 1 year ago

@grmorrison sounds like your ID time was well-spent!

Posted by jdmore about 1 year ago

Great! And so soon after the last one!

Posted by kemper about 1 year ago

@alexshepard Thanks! I had missed that announcement. That explained it perfectly for me.

Posted by xris about 1 year ago

Congratulations and my best wishes.

Posted by drnamgyal04 about 1 year ago

I'm glad Narthecium americanum is in the database! It's endangered, beautiful, and rare. I hope this helps increase observations of it to protect it better.

Posted by yayemaster about 1 year ago

This is great, thank you iNaturalist team. The examples and figures really help to illustrate the changes. A suggestion for a future update: is there a way to make it easier for observers (and identifiers!) to check which newly-added taxa they have contributed to? I can get some idea by clicking on the taxon-specific links you gave, and filtering by user, but as stated, many insects and plants are missing from those. It would be really neat to be able to see more easily which taxa I've contributed to. Thanks!

Posted by deboas about 1 year ago

Really cool that y'all are able to update it so much more quickly and thoroughly. Very exciting!

Posted by pttrsn about 1 year ago

Wonderful update, rate and level of detail! Thanks for that :) As @ajott mentioned, having the updated taxa list is going to be very useful to refine it further. Keep them coming!!!

Posted by dgilperez about 1 year ago

Congratulations with this fast update. Very nice. Is it possible to see if anything did change e.g. Netherlands, Canary Islands (spain), Greece, Korfu (greece), Lesbos (Greece), Kreta. Crete, Kreta (Greece)

Posted by ahospers about 1 year ago

Great stuff! Thanks everyone!

Posted by susanhewitt about 1 year ago

Thank you, i was more interested in which areas (maps geographic) the new computer vision model made great progress (Africa, SAmerica, Asia,) so i will take a look.

Thanks although i have no clue how to make this for my self. In the past molluscs (Shell, schelpen) for Thailand were a bit disappointing. But it seems this group has had some progress in Canary island. Good to see this.

Plants Canary https://www.inaturalist.org/observations??nelat=29.463514&nelng=-13.31543&place_id=any&swlat=27.425414&swlng=-18.391113&verifiable=any&taxon_ids=50794,56011,56113,56405,56481,56487,56506,56517,56738,56791,57136,57270,57733,57797,57925,58005,58026,58034,58048,58166,58181,58209,58239,58264,58267,58460,58793,58821,58868,58946,58983,59064,59278,59298,60088,60166,60226,60248,60455,60459,61877,62261,62263,62991,63510,63684,64064,67839,68424,68579,68580,68582,68588,71222,75248,75259,75269,75336,75337,75339,75465,75471,75543,75601,75628,75632,75725,75745,75799,75876,75879,75880,75886,75887,75888,75900,75948,76064,76099,76122,76144,76152,76160,76163,76167,76179,76230,76298,76310,76347,76407,76411,76438,76448,76463,76473,76519,76548,76574,76681,76736,76851,76873,76905,76927,76952,77012,77015,77077,77165,77173,77178,77219,77273,77296,77359,77385,77426,77452,77499,77515,77526,77536,77657,77706,77792,77849,77875,77881,77909,77973,78012,78079,78113,78121,78156,78191,78202,78209,78210,78254,78289,78331,78370,78380,78417,78418,78421,78457,78459,78575,78614,78617,78702,78708,78779,78926,78983,79199,79253,79254,79265,79375,79454,79552,79558,82696,82748,82766,82854,82935,82948,83580,83899,84407,85468,117455,120061,120519,121637,121645,121652,122181,122977,123551,125199,125423,125665,126747,127265,127512,128089,128723,129002,129399,129621,129622,129711,129832,131580&view=species

Plants Netherlands https://www.inaturalist.org/observations?verifiable=any&taxon_ids=50794,56011,56113,56405,56481,56487,56506,56517,56738,56791,57136,57270,57733,57797,57925,58005,58026,58034,58048,58166,58181,58209,58239,58264,58267,58460,58793,58821,58868,58946,58983,59064,59278,59298,60088,60166,60226,60248,60455,60459,61877,62261,62263,62991,63510,63684,64064,67839,68424,68579,68580,68582,68588,71222,75248,75259,75269,75336,75337,75339,75465,75471,75543,75601,75628,75632,75725,75745,75799,75876,75879,75880,75886,75887,75888,75900,75948,76064,76099,76122,76144,76152,76160,76163,76167,76179,76230,76298,76310,76347,76407,76411,76438,76448,76463,76473,76519,76548,76574,76681,76736,76851,76873,76905,76927,76952,77012,77015,77077,77165,77173,77178,77219,77273,77296,77359,77385,77426,77452,77499,77515,77526,77536,77657,77706,77792,77849,77875,77881,77909,77973,78012,78079,78113,78121,78156,78191,78202,78209,78210,78254,78289,78331,78370,78380,78417,78418,78421,78457,78459,78575,78614,78617,78702,78708,78779,78926,78983,79199,79253,79254,79265,79375,79454,79552,79558,82696,82748,82766,82854,82935,82948,83580,83899,84407,85468,117455,120061,120519,121637,121645,121652,122181,122977,123551,125199,125423,125665,126747,127265,127512,128089,128723,129002,129399,129621,129622,129711,129832,131580&view=species

I am a bit slow but i think i understand it..Molluscs Thailand https://www.inaturalist.org/observations?place_id=6967&subview=table&taxon_ids=49909,51171,61909,82796,84255,104028,120967,121708,123415,142963,151405,151508,193946,193982,196280,233409,253958,254662,255205,292304,292433,292440,328508,328630,336816,336837,347352,349044,353605,355742,362576,368129,368165,368698,369215,370314,370413,380353,394296,395007,397161,397832,398433,408586,416761,418333,418369,418378,418421,418454,428790,446723,460286,464979,466906,466912,467977,470152,471070,471649,476205,485954,489130,489137,489151,493947,496843,497147,497150,502072,504108,504268,509466,516620,519516,533797,538768,538828,556907,574548,628401,637083,680522,701970,737954,808261,823934,869564,874244,878167,898531,898566,1262869,1278719,1279523,1318822,1393781&verifiable=any&view=species

Molluscs Greece https://www.inaturalist.org/observations?place_id=7094&subview=table&taxon_ids=49909,51171,61909,82796,84255,104028,120967,121708,123415,142963,151405,151508,193946,193982,196280,233409,253958,254662,255205,292304,292433,292440,328508,328630,336816,336837,347352,349044,353605,355742,362576,368129,368165,368698,369215,370314,370413,380353,394296,395007,397161,397832,398433,408586,416761,418333,418369,418378,418421,418454,428790,446723,460286,464979,466906,466912,467977,470152,471070,471649,476205,485954,489130,489137,489151,493947,496843,497147,497150,502072,504108,504268,509466,516620,519516,533797,538768,538828,556907,574548,628401,637083,680522,701970,737954,808261,823934,869564,874244,878167,898531,898566,1262869,1278719,1279523,1318822,1393781&verifiable=any&view=species

Posted by ahospers about 1 year ago

@loarie any chance of putting each batch of new species in an iNat project? Then we could use familiar iNat tools to search. We are eaten up with curiosity (the spreadsheet is not so fun to use)

Posted by dianastuder about 1 year ago

Add a Comment

Sign In or Sign Up to add comments