Vision Model Updates

iNaturalist currently uses vision models in two main places: 1) a private web-based API used by the website and the iNaturalist iOS and Android apps, and 2) within the recently updated Seek app. When Seek 2.0 was released in April, it included a different vision model than we were using on the web. At that time the web-based model was a third-generation model we started using in early 2018. That web-based model was trained with the idea it would be run on servers, and servers can be configured to have far more computing power than a mobile device. As a result that model was far too large to be run on mobile devices.

Early this year, with an updated Seek in mind, we started another training run with two main goals: shrinking the file size of the model, and allowing it to recommend taxonomic ranks other than species (e.g. families, genera, etc.).

Smaller

The mobile version of the model needs to be small in terms of file size to minimize the amount of data app users would need to download. Smaller models can also be used by more devices as they need fewer resources to run (e.g. memory, battery), and can generate results faster, which is important for Seek's real-time camera vision results. These models take a lot of time and money to train, so we also wanted a model that could be simultaneously trained to produce a large web-based version and a smaller version for use in mobile devices.

Unfortunately, shrinking the file size like this slightly decreased model accuracy compared to the larger web-based version (kind of similar to image compression), and we found that was an unavoidable tradeoff. We take this into account when processing the model results, and on average for a similar error rate, the mobile version might recommend a taxon at a higher taxonomic rank than the web-based version. The taxon results we show to users shouldn't be less accurate, but they may be less specific.

More Species Represented

We wanted the model to include more species data, even when some species don't have enough photos to be recognized as species level. There are some species with a small amount of photos that, if we trained on that small set of photos, likely wouldn't have enough information for the model to reliable recognize those species.

Our 2018 model only included taxa at rank species. We set a threshold for number of photos, and species below the threshold were not included. We could still recommend higher taxa by doing some post-processing of results, but the model itself would only assign scores to species. In our latest training run we allowed the photos from species under the threshold to be rolled up into their ancestor taxa until the threshold was reached, and we allowed the model to assign scores to these non-species nodes. This allows more species to be represented in this newer model, sometimes at the genus level mixed up with photos of other species in the genus under our threshold. Now instead of not knowing anything about these species, the model can at least identify the genus or family, etc.

Replaced Web-Based Model

On May 24 we replaced the existing web-based model from 2018 with the web-based version of the 2019 model used by Seek. You may have noticed taxa higher than species starting to appear in vision suggestions, and this is why. That means the iNaturalist iOS and Android apps, and web vision suggestions have all been using this new model for the last few weeks. As of right now we are using the same model everywhere. The model has a web-based version (used by the website and iNaturalist apps) and a mobile version (used by Seek). The two versions are from the same training run, but can produce slightly different results because of compression in the smaller mobile version, and because the web-based version can weight results based on nearby species occurrences.

Evaluating

By adding photos of more species, we hoped to see an increase in accuracy for observations for species less represented in iNaturalist observations. One way we thought to get insight into that is to look at identifications made through vision suggestions that are current or improving (good) vs withdrawn or maverick (maybe less good). Here's a chart of such identifications for flies (Diptera) and spiders (Arachnida). The dotted red line represent with the web-based model changed. The difference isn't glaring, but it does look like for these groups vision-based IDs are sticking more.


The model was also trained more recently, so we hoped to see an increase in accuracy for observations in countries that have added a lot of observations since 2018. Here's a chart comparing the accuracy of the older model to the new model for a sample of random observations in California, Europe, Oceania, and South Africa (we could/should have included more countries with different observation and species densities, but these still show the trend). In this chart, "Right" means the model's best suggestion was consistent with the actual observation taxon (it suggested the right taxon or its ancestor), "Wrong" means the best suggestion was inconsistent with the actual taxon, and "Unable to determine" means the model was not confident enough to make a suggestion.

For observations from California, the models get nearly the exact same number right, with the older model making more specific suggestions. For observation from Europe, again the two models get nearly the same amount right, but this time the new model makes the more specific suggestions. In Oceania, the new model is getting more right and is more specific. And looking in South Africa, the new model gets a lot more right, with nearly double the amount of correct species recommendations (and fewer wrong).

Finally, the new model can now suggest taxa higher than species beyond the suggested common ancestor we've always had. This means the vision suggestions will be a mix of species and higher taxa. Here's a chart of the rank of all vision-based identifications over the last month or so. Nothing really stands out in this view of the data - there are maybe more identifications recently, and you can maybe see more at rank genus (the larger yellow bar).

By removing identifications at rank species, it’s clearer there have been more vision-based identifications of higher taxa since the new model went into effect.

By going even further and remove ranks below order, it's clear that there's been a vast uptick in vision-based identifications at these ranks. It should be noted that on May 31 the Seek app started to allow observations to be submitted to iNaturalist, and as explained above Seek will tend to recommend taxa at higher ranks than the web-based vision of the model, and we can see these higher taxon identifications really pick up after that. Better to have accurate but broad identifications, than inaccurate species identifications.

So that's an update on iNaturalist's vision models, and why some things may have appeared to change recently, like the addition of non-species recommendations. We think these changes are overall for the better, and hopefully you agree. We're interested to hear you thoughts. Did anyone notice the change? Now that you know the model should perform better for less observed species, I'd be interested to hear people's experiences with different taxa, or in different places.

Posted by pleary pleary, June 14, 2019 19:07

Comments

Thumb

Thanks for continuing to improve iNat. I will pay more attention to visi oil n recommemdatiij and share my feedback later.

Posted by gancw1 12 months ago (Flag)
Thumb

Bravo! This is a huge improvement in Arachnids for at least some families and genera. I was ecstatic when I started noticing correctly identified Pardosa wolf spider observations, as suggested by the computer vision! These are spiders that we can generally only identify to genus level, but they are one of the most commonly observed genera in the world. So thank you, thank you, this is wonderful! 💗

Posted by tigerbb 12 months ago (Flag)
Thumb

I definitely had a huge sigh of relief when I noticed some CV genus-and-coarser IDs rather than species while doing quality control on some of the (previously) common computer vision traps. Thanks for your work on this and letting us know about the updates. The graphs are great to see the improvements.

Also, in case you didn't see this yet, I started a FAQ section about computer vision here if you wanted to edit/expand: https://www.inaturalist.org/pages/help#computer-vision

Posted by bouteloua 12 months ago (Flag)
Thumb

Oh sure I noticed the changes here in the Northeast for arthropods. I was very glad to see more genera and even families appear as suggestions.

I am talking in general here, not specifically about Mollusca, which I don't think has changed much, although I must say that very often with mollusk IDs I am not using the Computer Vision.

Posted by susanhewitt 12 months ago (Flag)
Thumb

I continue to appreciate the candor and transparency from the developers. Aside from just being just basic good human communication, it helps general users like me "see the big picture" and appreciate/understand considerations I might not otherwise. Keep up the great work!

Posted by colincroft 12 months ago (Flag)
Thumb

Great work!

Posted by jay 12 months ago (Flag)
Thumb

Thank you! - happy to expect better suggestions for our South African species.

Posted by dianastuder 11 months ago (Flag)
Thumb

I wonder why California didn't improve as the other regions did, maybe because there were already sufficient observations to accurately train the software, thereby reducing any potential gains in accuracy by way of a more computationally-intensive model?

That said, I have noticed a noticeable increase in correct assignments for less commonly observed taxa, and taxa (such as ferns) which can be difficult to identify to genus for myself and the model at first, while some (e.g. Arctostaphylos) continue to be elusive!

Posted by yerbasanta 11 months ago (Flag)
Thumb

I can confirm the genus option is offered and, in this case, is made a lead option:

Awesome update - having the genus first should encourage uncertain observers to ID to the genus!

Posted by danaleeling 11 months ago (Flag)
Thumb

I've definitely noticed the improvement in Ontario. The algorithm can often get sedges and grasses correct, something it was really bat at before.

Posted by reuvenm 11 months ago (Flag)
Thumb

Still struggling with - not clover, it's Oxalis
https://www.inaturalist.org/observations/23630819

Posted by dianastuder 11 months ago (Flag)
Thumb

@dianastuder the first ID was not a computer vision suggestion. The top computer vision suggestion is Oxalis purpurea.

See https://www.inaturalist.org/pages/help#cv-select

Posted by bouteloua 11 months ago (Flag)
Thumb

Thank you so much for this journal post and a little more of the nitty gritty details of the visual algorithm and taxon suggestions. Sharing this with several folks.
Keep up the great work.

Posted by sambiology 11 months ago (Flag)
Thumb

Just curious if there was any change in not showing recommendations below a certain match score. I ran the CV on a record just now, and it only pushed out 3 recommendations (none of which was right, but that is a separate issue), something I have never seen before, I've never not seen it spit out the full list of 8 or 10.

Posted by cmcheatle 11 months ago (Flag)
Thumb

@cmcheatle can you share the link for the observation where that happened?

Posted by carrieseltzer 11 months ago (Flag)
Thumb
Posted by cmcheatle 11 months ago (Flag)

Add a Comment

Sign In or Sign Up to add comments

Is this inappropriate, spam, or offensive? Add a Flag