The Latest Computer Vision Model Updates
We’ve released a new computer vision model for iNaturalist. This is our first model update since July 2021. The iNaturalist website, mobile apps, and API are all now using this new model. Here’s what’s new and different with this change:
- It includes 55,000 taxa (up from 38,000)
- Hybrid taxa are excluded
- It’s more likely to suggest the correct taxon
To see if a particular species is included in this model, you can look at the “About” section of the taxon page.
Here’s more details about what changed and why:
It’s a lot bigger
Our previous model included 38,000 taxa and 21 million training photos. When we announced that we were kicking off a new vision training job in October, we planned to train a new model on 47,000 taxa and 25 million training images. However, in November, after training had started, we realized that we could make improvements to how we were picking our training data that would improve the quality of the vision models. After making those changes, we re-started training with over 55,000 taxa and over 27 million training images.
We changed a few things about how we generate training data
As I mentioned above, we made some changes to how we generate our data export for the training data. Patrick’s new code for making the data export has a few improvements worth mentioning:
- It includes more taxa right on the borderline of inclusion (taxa with at least 100 photos but fewer than 100 observations will now make it into the export, but didn’t previously),
- It limits the number of photos used per observation (max 5 photos from each observation), and
- It is more clever about picking distinct photos for each taxon, preferring to choose photos from as many different observations as possible. This increases the visual diversity of the dataset, which in turn helps the computer vision model learn.
Our testing suggests this new approach produced a much better dataset. Overall accuracy numbers went up with this model, even though the model’s job has gotten considerably harder (choosing between 55,000 options instead of between 38,000 options).
We excluded hybrids this time
We also chose to exclude hybrid taxa for this training run. The previous production model, released in July 2021, was the first to have significant amounts of training data for many hybrid taxa. Including those hybrid species in the model made it much less likely that the first suggestion would be correct for clades like Genus Anas which includes Mallard Ducks, the most-observed species on iNaturalist.
Our CV models are trained to recognize discrete, mutually exclusive, distinct taxa. Given a photo, there should be one right answer as to what discrete taxon it belongs to. Hybrid taxa, while being potentially useful taxonomic entities, make it hard for our CV models to visually distinguish hybrid taxa from their hybridized origins, and to confidently recommend any of these taxa in any scenario given their visual overlap. So we decided to remove hybrid taxa thinking it would make the classifier’s job easier and thus improve accuracy, and our testing showed this to be the case. We believe it’s better to accurately identify distinct species than inaccurately identify hybrids and their origins. This is a reminder that taxonomy is an abstraction trying to put hard edges on what is often a continuum. Hybrid taxa are good examples of where this abstraction is an oversimplification but our CV doesn’t do well with some of these edge cases like hybrids and we’ve found the benefits from simplifying outweigh the loss in accuracy from trying to accomodate hybrids.
Future work
In addition to training and deploying this new model, we’re working on a few areas to generally improve the quality of suggestions.
First, we are continuing to work on new approaches to improve suggestions by combining visual similarity and geographic nearness. We can’t share anything concrete right now, but we’re excited to be working with some Visipedia researchers on this.
Second, we are running more experiments to improve training speed. Updating our vision models once or twice a year creates a lot of pressure on each model. There are techniques for speeding up training (such as transfer learning and finetuning) that we haven’t been completely taking advantage of, that may someday allow us to release new models every month or two.
Third, we will continue to look at new approaches to including or excluding nodes in the taxonomy. For example, we may someday find a technique to include hybrid species without causing significant reductions to suggestion accuracy for neighboring species. Or we may be able to identify other ranks or nodes in the taxonomy that should not be represented directly in the vision model.
Fourth, we’re still working to compress these larger models for on-device use. The in-camera suggestions in Seek continue to use the older model from March 2020.
We couldn't do it without you
Thank you to everyone in the iNaturalist community! Sometimes the computer vision suggestions feel like magic, but it’s all people. None of this would work without the millions (!) of people who have shared their observations and the knowledgeable experts who have added identifications.
Beyond adding observations and identifications, here are other ways you can help:
- Share your Machine Learning knowledge: iNaturalist’s computer vision features wouldn’t be possible without learning from many colleagues in the machine learning community. If you have machine learning expertise, these are two great ways to help:
- Participate in the annual iNaturalist challenges: Our collaborators Grant Van Horn and Oisin Mac Aodha continue to run machine learning challenges with iNaturalist data as part of the annual Computer Vision and Pattern Recognition conference. By participating you can help us all learn new techniques for improving these models.
- Start building your own model with the iNaturalist data now: If you can’t wait for the next CVPR conference, thanks to the Amazon Open Data Program you can start downloading iNaturalist data to train your own models now. Please share with us what you’ve learned by contributing to iNaturalist on Github.
- Donate to iNaturalist: For the rest of us, you can help by donating! Your donations help offset the substantial staff and infrastructure costs associated with training, evaluating, and deploying model updates. Thank you for your support!
Comments
Yesss!!!
Nice!
Awesome! Thank you staff!
Awesome!!
Looking forward to seeing this in action. And a spur to push more taxa above the 100 observation mark. Lots of low-hanging fruit!
"55,000 taxa " - I presume that this is species-and-above taxa: or do we include subspecies and varieties with over 100 observations?
It seems to work great! Thank you!!
Excited to try this out, thanks everyone who worked on this
That's correct, Tony.
Wow, thanks for this important improvement!
Been waiting for this for a while. Thank you!
Awesome, I can't wait to try it out!
Amazing! Huge thanks to all the machine-learning people making something like this possible.
Sweet!
Very pleased to see this, thanks! I can't wait to try it out on some pteridophytes here in Australia.
Cheers :)
Exciting. Am I right in thinking the 100 photo requirement is not limited to research grade either?
(Just tested for a taxon where almost all obs are still Needs ID as all at genus level but appears to recognise just fine)
THIS IS INCREDIBLE!! Thank you iNat staff, for making this wonderful website a reality. This is going to be so useful for so many people. Good work :)
Thank you for your hard work.
I've been really amazed by the CV model since I have started. Can't wait to see what progress and new ideas will be created over time.
Any plans on creating a model for sound?
Magnificent!!! Love hearing about these updates.
Great and well done.
So exciting to see this progress. Can’t even imagine the state of the art in ten years!
Cool stuff! Thanks for making it happen!
Amazing news! Props to the programmers and all of the folks helping with identifications that have contributed to it's growth 👍
Incredible works by the team! Been taking photos and IDing a few taxons so they reach 100 photos and observations, so its great to seeing them finally getting CV! :)
Just in time for spring :)
Congratulations team iNaturalist for another amazing update, keep up the good work!!
Cool. Glad you guys could pull in some of the taxa with fewer observations.
Excelentes noticias!!!!
Felicidades!
Congratulations! Lots of work went into this.
Awsome!
Bravo! Well done. Thank you for the comprehensive update.
Great new updates!!
Great news!!
Thank you!!
Well done. Thank you
Hurrah!
Awesome!!!
Impressive
Yes! Nice! Worked for Paraselina brunneri (86 sightings, 70 are mine ;) ).
Well done :)
That's excellent news! Seven new species have been added in Brodiaeoideae taking the total recognized by CV from 20 to 27.
Androstephium breviflorum
Androstephium coeruleum
Bessera tenuiflora
Bloomeria clevelandii
Brodiaea filifolia
Triteleia hendersonii
Triteleia lilacina
That means there are only about 44 more to go! Encouragingly, several other species now have more than 100 photos, raising the prospect that CV can help distinguish Brodiaea minor from Brodiaea nana and more.
Hooray! I know it's hard work!!! But it's appreciated.
I did my own test!
Photos of Hesperomecon linearis (Narrowleaf Queen Poppy) were until now always identified as Platystemon californicus (Cream Cups)
But as of this update it correctly suggests Hesperomecon linearis
I'm glad I took an evening a half a year ago to go through all the "research grade" P. californicus ID's and fix the half dozen that were actually H. linearis
That's awesome, Henrik!
Our work to train the models pales in comparison to the collective labor that our community does. It's really the identifiers and observers who collectively produce the incredibly high quality dataset that makes our vision system possible.
Great news! Congratulations and thank you for the excellent work. It seems to be working great in some European spiders that were previously misidentified. Top! That 100 threshold makes me want to sit down and ID further!
¡Excellent!
At what stage will it be acceptable to use the AI to automatically assign identifications to observations posted without an ID?
The AI is already superior to 70%? 80%? 90%? of identifiers.
Given an acceptable protocol (e.g. backtrack up the taxonomy tree until the AI is 80% certain and the taxon is previously recorded within 50km), I think that we are almost there.
That's a good question @tonyrebelo. I think I'd like that logic to require a higher percentage confidence (95%? 98%?). That would lead to less precise automated IDs, but higher confidence from observers and human identifiers that the ID isn't a mistake. I've learned that anything that generates a small level of "plain wrong" IDs really irks people and can work against the goal of engaging people with nature.
If iNat did start to add automated IDs in this way, I wonder whether they should be excluded from the Research Grade calculation (and maybe from the Community ID logic). If not, we have the prospect of observations reaching research grade based on an incorrect automated ID and the observer clicking "Agree". Alternatively, maybe when the observer clicks "Agree" that removes the automated ID.
There's also a significant issue of how to deal with cultivated observations, which are probably overrepresented among Unknowns and where the idea of a 50 km radius may reduce the probability of a correct ID.
Thanks: interesting points.
I find it fascinating that we set such a high bar for the AI confidence, but there is no bar at all for user IDs. Many of whom are novices.
However, we dont exclude novice IDs from the Research Grade calculation, Community ID and so forth. More pertinently, we allow identifiers to use the AI and post the ID without any checks as to whether it is nothing more than an "automatic AI ID" versus a considered evaluation of the AI options. (and so at each City Nature Challenge the number of American animals that turn up in Cape Town is truly astounding).
My experience is that it is far, far easier to evaluate an ID as OK, iffy or downright wrong, than to actually make an ID in the first place. In this regard the AI is a brilliant first step. By all means, provide checks and balances, but please be fair to the AI - dont discriminate against it just because it is not human: rather use its consistency, experience and accuracy to penalize it (or not) versus humans.
I dont think cultivated and captivated observations are an issue, provided marked as such. Special protocols could deal with this, but the reality is that - exactly as for wild organisms - it is only an issue for the first record in an area: once recorded it is there. Gardens in an area are remarkably uniform, based on what has been supplied by local retailers and shared between clubs over the last few decades. Even street trees and borders between suburbs appear to share fashions and municipal nursery stocks. Moreover, garden plants are likely to be shared worldwide (quite apart from wild populations of these species) and thus be relatively well represented in AI training: in total contrast to wild organisms were local endemics and rare species may take decades before they make the cut for AI training.
Awesome work!
Congratulations! iNaturalist just keeps getting better and better over time. Very impressive!
Maybe we should think more about helping other to make better identifications, even if only to broad taxa.
Perhaps: but the AI is already being trained, and improved, and can make better IDs to far finer taxa much faster and more reliably.
I completely agree, and I would strongly support the addition of tools to help facilitate that. One might be that AI IDs are automatically added for Unknown observations in the way you have described. Another could be a reverse search tool for identifiers that finds the observations that best match a certain taxon but not yet given that ID (could be too computationally expensive, I guess).
My main reason to set a high bar for automated IDs is that observers and identifiers will give such IDs a high level of credibility (as they do currently with ID suggestions) and we ought to match that in terms of the confidence threshold required for a CV ID. In practice, a high bar means that automated IDs would sometimes choose a genus that is 98% certain in preference to a species that is 85% certain. I think that's fine. There's a big risk that a more aggressive approach would result in a noticeable level of mis-IDs (and an unpleasant feedback loop for the next round of AI training).
Also, even a small level of "plain wrong" automated IDs would quickly give the process a bad name. I think we can see how that might play out by looking at all the complaints in the forum about how inexperienced users blithely accept the current ID suggestions (reduced now that the IDs are better I feel).
It's not clear to me whether CV considers the locations of cultivated specimens (or just wild ones) as part of its nearby algorithm. If they're included, then the challenge of ID'ing cultivated observation is lower.
The bigger issue is whether iNat staff has any interest in adding automation along these lines. The impression I've got from forum responses is that there's a lot of hesitancy to go beyond "decision support" and add true automation features. If so, this discussion may be moot.
An easy way around the concerns of automated IDs for unidentified observations being blindly accepted to Research Grade is simply to limit them to the Family to Genus range. Those wishing to go further can then manually use the AI ID to go further.
I think that the AI is already precise enough to readily set a 90% bar, and to rank up if the discrepancy between ranks is greater than 5-10%.
I suspect that the pressure for automatic assistance will grow as unidentified (including those to rank of class and above) observations accumulate. Especially in countries and regions with many observers but very few identifiers. The increase in observations being posted is surely going to rapidly outpace the increase in competent identifiers joining iNaturalist. There are already 2M observations (5% of plants) identified as "Dicot"-"Monocot"-Pine"-"Fern"-"Moss" or higher. If the AI can take them to family or genus it would greatly increase the chances of a still finer identification, with little danger of "plain wrong" IDs.
To be honest, Tony, computer vision suggestions without a human in the loop isn't something I've been thinking about.
When I think about an identification, I think not just about a label applied to a piece of evidence. It's also communication between two people. Humans can explain and justify their identifications, quickly learn new criteria based on specific teaching, understand why someone might have made a particular ID mistake and offer specific guidance, can (sometimes poorly, admittedly) understand knowledge domains where they have more or less knowledge, recognize that they've gained new knowledge and circle back to an old identification with a new perspective, etc. Our vision model can't really do any of this.
However, I also have to acknowledge that the growth in observations on the site means that identifiers have an increasingly large backlog of work.
Curious what others think.
Thanks Alex. One of our most frequent complaints in southern Africa is when specialists dont explain why they disagree with an existing ID (which happens quite often here with our many really rare species that have never featured in field guides).
Specialists counter that they would love to, but dont have the time - every explanation is several observations not identified, or that they did so on their first ID, but see no point in doing so on each observation. The net result is that everyone who knows that the idenitifier is a specialist just slavishly changes their IDs.
I dont forsee the AI being able to explain an identification in biological terms (and terminology) on a phyla-wide scale - not because it cannot be programmed, but because useful databases of key diagnostic features dont exist [but if they did, I am certain that AI could be used to improve them], let alone in a form usable to AI image recognition. Also because rarer and localized species are unlikely to make the cut for AI training (unless a way of training for including rare species on a handful of observations becomes feasible), so that AI is by current programming only for relatively commonly recorded species.
I do foresee though that even specialists will come to rely on AI more and more to presort observations for ID, and even by "training"on a specific (rare on poorly known) taxon to find possible matches. After all, we already use other tools (books, printed keys), and advanced tools (electronic interactive keys) to assist with IDs. And image recognition AI is the best available tool for electronic images in large datasets. It is only a matter of time that vision models will equal specialists in precision, and way outperform them in speed and volume. A small price to pay for the loss of "people" communication?
Of course, we are not there yet ...
@alexshepard: How feasible would it be to develop a search tool that finds observations that CV reckons best match a certain taxon but that don't yet have that ID? That would greatly improve the efficiency of the many specialist identifiers. Rather than require those folks to manually trawl through vast numbers of observations ID'ed as unknown, or with a high-level taxon, or misidentified, they could focus on a smaller set of observations that all look plausibly similar to those taxa they know in detail.
Also, this would still remain in the realm of decision-assist technology rather than fully automated IDs.
YASSSS
@rupertclayton: Great idea!
Wonderful update, thank you!
nice
@rupertclayton - I certainly would regularly use such a tool, especially at genus and family level.
Works much better than previous version. Thanks a lot for this update.
Nothospecies are hybrids ...
@rupertclayton - what an interesting idea!
Regarding @tonyrebelo's comment above ("At what stage will it be acceptable to use the AI to automatically assign identifications to observations posted without an ID?
The AI is already superior to 70%? 80%? 90%? of identifiers."),
I don't agree that the AI is superior to most human identifiers (though it frequently exceeds my expectations of it in cases where the photos are clear), but I appreciate the idea of some form of failsafe to get Unknowns out of the Unknown pool. Then again, I think if this was implemented, the AI should stay at a safely vague ID like Kingdom, and having things that vague would only be slightly better than Unknown. Considering that many Unknown posts are low quality posts with blurry subjects from new users, it could prove difficult to get the AI to confidently identify such posts.
As of the present, I believe we have enough users actively and consistently working to identify observations out of the pool of Unknowns. I think it's a good idea to look at potential solutions (like the one you recommended) to cut down on the number of Unknown observations posted daily (especially since such solutions could free up the manpower currently focused on quality control for those Unknowns). I think this would make an interesting forum discussion, as this comment section isn't conducive to a clear and well-formatted discussion.
Are all hybrids still excluded from CV?
An ID at the level of Kingdom, or even Order for plants is essentially useless. It might work in the species-poor northern temperate regions, but is as good as "Unknown" in the tropics and species rich south.
The ideal level is family level: that is the level that experts and specialists can get involved.
But if the AI is confident, why not just go down to genus?
I suppose that if you have very many plants in "unknown," you're going to have to wade through the insects and birds and fungi of "unknown" to find them no matter what. So you might prefer plants left in a category you have to search anyway. However, one can start at "plants" to find unidentified plants. (One could even set the CV to do that level of classification, maybe.) Everyone has their own preferred work pattern, but I consider "plants" or "flowering plants" to be good places to start. Plenty of observations to ID there! Of course, family or lower identification is much better and can draw in specialists. But I can't really get my head around the idea that "unknown" is somehow a better bin for plant observations than "plants."
Don't worry, I can get my head around the idea that some of you in Africa prefer plants left as "unknown" and I do that. But it still seems strange to me.
@lotteryd has plans for an African dicot blitz.
@sedgequeen Let the figures speak:
ID to Order or above - for southern Africa: (i.e. https://www.inaturalist.org/observations?lrank=order&place_id=113055&subview=map&verifiable=any&iconic_taxa=Aves etc.)
Herps 628
Fish 835
Mammals 1,712
Birds 2,412
Molluscs 3,171
Arachnids 5,700
Fungi 23,066
Insects 41,265
Plants 67,830
versus:
Unknown 33,901 of which
page 1 has 2 reptiles, 1 insect, 1 mammal dung, 5 duds and over 90 plants (duds = no picture, multiple species, fossils, etc)
Page 10 is 4 duds and 3 others that are not plants
Page 20 is 1 dud and the rest plants
Page 30 is 1 scat, 4 duds, and the rest plants
I rest my case: the Unknowns are 99% plants, why bother moving them?: just leave them! Leave them for the people who can identify them to Family level where they are useful.
Especially seen that the unknowns are often easily identifiable, just not yet identified, whereas the "Plants" contains a significant amount of truly awful observations that probably cannot be identified any further.
Personally, my speciality groups of Proteaceae, Ericaceae, and the Fynbos endemic families have such a backlog that I will never ever have time to spend (waste?) on the "Unknowns" or "Plants". I rely on other people to bring them down to family level, and like our other specialists in the region, am overwhelmed by the volumes requiring identification.
The AI tool, and specifically @jeanphilippeb superb projects on identifying observations are indispensable to me (e.g. https://www.inaturalist.org/projects/unknown-proteaceae) and effectively do the drudge of extracting Proteaceae from the unknowns to a usable level. The error rate (false positives) is 22%, but to be fair, quite a few of these observations do have proteas in them, but are not the focal plant or are the host plant. I have no idea of the false negatives (Proteaceae missed), but have not encountered any in a few searches of the unknowns that I have done.
I agree that "Plants" tends to have a lot of photos that can't be ID'd to species, or even to family. What you're doing is clearly working for you.
And the dozen taxonomists and specialists I work with, who wont touch "Plants" because it is a waste of their time. ...
Dealing with the "Plant bin" is for diehards, who I would prefer to rather tackle unknowns and place them to family. and only ID to Plants when the case is hopeless.
Still clearing CNC and Pre-Mavericks. Then Great Southern Bioblitz.
Next year I will look at it's A Plant!
Add a Comment