A larger experiment to learn about the accuracy of iNaturalist observations

Happy I could help! Most of the ones I got weren't what I normally identify though. I think my pool was based more off of IDs I've left on my own observations rather than taxa I actually know in depth.

@lj_lamera, this is the query the experiment is using to find candidate validators https://www.inaturalist.org/identifications?user_id=tcriley&taxon_id=51903&category=improving

Feliz por participar!

Turns out your IDs on your own observations do count as "Improving" if the community agreed.

Happy to help out again and got a nice mix of easy and more challenging stuff, mostly close to my usual area with a few outliers. Some I won't be able to confidently put a species ID on, and proud to be a maverick on at least one. LOL Curious to see where that one is going.

Thanks for the invite, happy to help. One of mine completely out of my experience so left blank.

One broken image ... processing
https://www.inaturalist.org/observations/61678661

Cnidaria? I can jellyfish but.

Very curious to see the % response after the first 24 hours!

Always happy to help with these experiments!

Happy to be invited for the first time to participate in this experiment!

@reiner if you're happy with your previous IDs, you can leave them as they are, but if you have gained more knowledge since then and can improve them, I understand that you're encouraged to do so

My set was good for me, happy to see some rusts, mildews etc in the expanded set!

I don't know the best way to respond to the original 'invite' to participate. I emailed back, but this is what I wrote:
to the iNat experiment designer team-
I would very much have liked to contribute to the experiment - 3rd edition. BUT, you have given me observations in geographic areas that are beyond the area in which I am comfortable IDing species. I do not know the variation in appearance of species that I know in my home range, and I do not know the local look alikes elsewhere - so I don't normally ID beyond my home range. I think asking for IDs in areas not normally investigated (if that is the right word) changes the likely outcome AND makes the experiment unlike the standard operating procedure for me in IDing (and presumably me as a representative of IDers in general), so makes the results of the experiment not equivalent to standard results.
So - I can scan the list of locations and ID only ones in my normal ID geographic area (more or less New England, the more or less is very occasionally for some species in the northeast US and Canada).
Or - I can just not participate
Or - you can ask me to try to ID outside of my home range knowing that I think that negates part of the purpose of the experiment .
I really will wait to hear from someone involved before I move forwards with any experiment IDs.

Is something off in the image https://static.inaturalist.org/wiki_page_attachments/3785-original.png or am I just misinterpreting the correspondence between numbers and colors?

Happy I could help! and I’ve already added an identification to all of them.

@guerrichache for that reason I leave a copypasta comment

Not disagreeing with you
Observation Accuracy Experiment 3

Would it be possible to exclude taxa that users have only observed (rather than ID'd on on other people's observations), and/or CV IDs? I only got about 3-4 of these so it wasn't a big issue, but for example I was given an observation of Diplotaxis to ID; I have 0 IDs of that genus that are not my own (3) observations, and on my own observations those were CV suggestions that I'm unable to really verify. Excluding CV IDs might be a bit of a problem for people who actually select them when they are correct to avoid typing in a name rather than as a guess, but I don't know how common that is. For the former I would think that if someone is specialized in a taxon that they observe a lot, they'll also have a lot of IDs on other people's observations.

Yeah, that's why I think the former is probably better, but maybe there's a potential issue in that too.

comment 1 of 2

I commented on the previous posts about experiments 1 and 2, and want to update and clarify my overall view on these continuing experiments. I think the experiments are a good idea and that it will be ideal and necessary to continue to conduct additional ones, and to conduct ones where the many involved variables are modified in different ways. The only things I partly have a different view on are the following. It would be helpful to ask validators for feedback on experimental design before designing/continuing to conduct these experiments. Participants have given good feedback on the experiment journal post threads, although it can be difficult to bring up issues there. It would be relevant to know if there are plans are to publish the results of these experiments or future ones in any external sources, since in that event we'd want to be cautious that the results are presented accurately and that the experimental limitations, atypical circumstances, etc. are noted.

So, it would be best for staff to keep a record of all the limitations etc. people have mentioned in all the threads. For e.g., I just thought of another that I'm unsure if has been mentioned. Which is that these are obs. that are atypically receiving multiple identifier verifications even after already being RG, I estimate at least 3-5 identifier IDs. Also, because validators/all users are primed to know these specific obs. are in an experiment, that incentivizes them to expend more time or effort, and/or to collaborate more or consider the ID input of users perceived as having the highest ID expertise. Furthermore, large groups of identifiers have already previously coordinated and worked individually to nearly "complete" reviewing all pre-2024 obs. for several North America bee (e.g. Bombus, Apis) and aculate wasp (e.g. all of Vespidae and some Crabronidae and Sphecidae) groups (see here), which overall increased the number of prior validations (including for obs. that are still Needs ID) and increased the overall uncommon practice of checking already-RG obs.

And, at least some of these obs. (e.g. ones that are still Needs ID) are included in the experiments. Those obs. helpfully boost the accuracy results, although it could be mentioned as contextual note or limitation that the circumstances around how they've been identified prior to these experiments has been atypical, largely identifier-driven, but also in case further incentivized by by e.g. US state-wide pollinator surveys, research, or publications.

comment 2 of 2 from my previous

So, overall, the obs. pool being used in the experiments and their circumstances are somewhat atypical (also for additional reasons), and it's also clear that on average, any obs. receiving atypically larger numbers of validator/user IDs will have higher accuracy, especially for already-RG obs., which often otherwise only have 2 precise IDs. This could be tested and proven by comparing the results to the experiment accuracy results of RG obs. that only have 2 precise IDs (that made the obs. RG) randomly selected without taking into account whether validators reviewed them, or randomly selected after excluding all obs. used in past and present experiments. As a side note, the average accuracy of all existing RG obs. (before or w/o doing experiments) is also a very relevant stat to calculate in its own right.

As I and certain others have commented or implied, the accuracy results reported for experiments 1 and 2 would seem to need to be at least somewhat overestimated (it's hard to estimate the extent) for reasons we gave (another reason is that easy and overabundant species like honeybees may "inflate" the accuracy; also note that earlier inat accuracy experiments found lower accuracy, although methods differed somewhat), although are still a meaningful experiment and stat if caveated and described in an accurate way. Also, the previous accuracy results to me sounded like they were saying they were the accuracy results for the average RG obs., despite that that stat would be expected to be lower relative to whatever the former is, as I explained. I might instead describe the current experiments as something like the average accuracy of [atypically] extra-validated RG obs. Lastly, just to be clear, I agree with most everyone that the RG obs. accuracy, especially of extra-validated obs., is somewhat high, and that that's ideal and we're seeking to further improve that, and that it's ideal when possible for obs. to receive extra validations, i.e., for identifiers to check already-RG obs. All I'm "cautioning" is to take more consideration of the feedback validators have given when deciding how to describe or publish on the results of the experiments, and ideally when considering what future experimental designs will be.

E.g., I've suggested that multiple different kinds of experiments would not only be ideal but also necessary in order to more fully understand and represent what these current experiments are finding, e.g. to assess whether or to what extent the reported accuracy findings have been overestimated if reported verbatim without caveats. And, I'm continuing to participate in all of the experiments as a validator, even though that the organizers haven't indicated whether they're taking the feedback by myself and some of the other commenters into account. One final interesting thought is that overall, these experiments themselves must be increasing both the average all-obs. RG accuracy and the extra-validated obs. RG accuracy on the website.

An additional relevant and related version of the experiments would be to specifically exclude obs. of overabundant species.

@bdagley I don't understand your concern about the observations in the experiment getting additional IDs. The point is to verify whether the ID they had before the experiment (when they did not have excess IDs) is correct. It's a good point that once an observation has been used in one experiment, it should ideally be excluded from future experiments - I don't know if this is being done. It could be done by defining narrow date ranges for each experiment, perhaps.

That's an interesting problem. I would probably agree on the observation level but flag the taxon for its outdated taxonomy.

Received and finished with it

@bdagley Eliminating overabundant species from a test like this is not optimal, because it introduces a non-random bias so quantities like the top-line accuracy number cannot be recovered in an unbiased way. It also makes hypothesis testing harder (for example whether accuracy profile is a function of number of observations), and cross-checks of the accuracy rates observed across different versions of the test are not possible. It is better to modify the relative weights of species by a smooth monotonic invertible function of observation count (like the square root I suggested) instead, because that transformation can be statistically inverted when reporting results.

Also this experiment is a little different from an ID blitz in that for the test it is most ideal if people mainly answer in areas they are relatively expert in. It really becomes a challenge to figure that out though especially for genus-level observations where someone who mainly IDs in California might have literally no idea about an observation in Alabama.

@bdagley -- It seems to me that what is being tested is the overall accuracy of identifications on iNaturalist, the accuracy of ID's before the experiment, you might say. During the experiment, we observers do evaluate the selected observations carefully, maybe check references or consult experts if necessary. At the end of the experiment, these selected observations should be more accurate than they were to start. That is fine! The ID's tested being tested are the ones that existed before the experiment. (ID's before the experiment are typical of iNaturalist. After the experiment, the ID's are not. That can be dealt with by excluding tested observations from future tests.)

It's true that there is a lot of variation in ID accuracy in iNaturalist. Accuracy is probably close to 100% for RG observations of North American birds and quite low for Needs ID grasses. That is important. Every researcher using iNat data should consider it. However, this experiment is a very first step in evaluating iNaturalist accuracy. Like most experiments, it is limited. It's telling us interesting and useful information but there are VERY many other questions we would like answers to. We can't get everything at once. Unfortunately. I'm impatient.

@dhasdf iNat has well over 150,000,000 observations eligible for this experiment. a few thousand observations getting half a dozen fresh eyes is negligible, if its signal could even be distinguished from the constant flux of IDs and new observations that are made every day.

Maybe the algorithm for matching IDers and obs should be improved for the next round. I got 34 observations, and Im glad to work through. Hovever, some do not perfectly match my field of expertise. Though I educate myself to make the best of it, this might alter the result, because I might be unsure not because an ID is unsure, but I don't have much experience the species and/or region in question.
Here are characteristics for are some of my examples:
I have two Pinopsida cultivated in a garden, both in a region, where they would not apear in the wild, so in the end those could have been from any temperate region almost worldwide. Outside the experiment, I would have ignored those.
Two example did not include geolocation data. I think those should have been excluded because of this.
A major challange is the region in question. I do by far most of my IDying in Germany. For the experiment I got observation from Belarus to Spain. It is an extra challange to exclude species similar to what I am reviewing, that I might not be aware of because they are unkown in my region. While most of my identifications take seconds, some of those require additional hours of research to educate myself.
I’ve noticed comments in some saying "I am not an expert for this, but I do it for the experiment". With that, the experiment tests more the knowledge of the selected IDers and less so the quality of the original ID.
Just to emphasize: I love to do my 34 and I am done with 32 of them. I'll work hard for the remaining two ;-)

@deboas Believe it or not due to the Birthday Problem/Paradox, even though there is only 10,000 drawn observations out of almost 200,000,000, if you draw another 10,000 randomly there is roughly a 50% chance you will get at least one observation from the first set again in the second set. Sure that would only contribute a ~0.01% source of error but it would make sense to just avoid it.

I love this! I often leave my observations at order, because the suggestions are not reliable for PNG. I'm glad that you are working towards improving this feature. Also, a HUGE thank-you to everyone who takes the time to identify my observations, PNG is full of fun and interesting life!

@wildskyflower interesting, thanks!

''a sample of 10,000 observations'' I hope only Research Grade observations are sampled?

Two example did not include geolocation data

@annkatrinrose This is one of the reasons for the experiment. If you don’t think an observation can be identified as finely as it has been due to the lack of data, you can add a disagreement :)

Same thing applies to poor quality photos and cultivated obs

done with my 34 :-)

I don't really see that happening a lot. More than a third of the observations in my set just had an initial ID from the observer, and mostly based on CV suggestions. Even out of those at RG already, several just had the initial ID from the observer and one confirming one both based on CV suggestions. I think it's more about evaluating CV/initial IDs than checking community ID accuracy.

@annkatrinrose I completely agree, but in that case I wonder about an experiment specifically designed to root out exactly those types of observations. What percentage of observations with no community ID (essentially one ID) are accurate? What percentage of RG observations with only two IDs are accurate? It would be great if one could search for these observation sets with the Identify tool (in fact, I think this would be necessary to conduct an experiment).

Good luck getting those people in large enough numbers on iNat and keep the site running. In that case I would be able to help out with one spider genus, which is more or less an observational fringe case here with just a little more then 1000 observations in total. Also, I am not a fan of excluding all those amazing self-taught experts without an official degree. Taxonomists are probably a safe bet, but from my experience actually also not always.. in the spider world they are often over-focussed on genital traits completely ignoring prominent fieldmarks, enthusiastic amateurs easily recognize.. good luck finding a spider taxonomist that would be of any help in an experiment like this by using only the traits they actually have certified expertise in...and then they often only work on fringe cases as well..

@loarie I know, but my opinion is not relevant, and it's more like observations with species-level IDs, so my class-level ID would be even less relevant (and would look like a disagreement with species ID). More importantly, most people who received invitations will not place class-level IDs, but will be guessing and picking AI-suggestions.

"most people who received invitations will not place class-level IDs, but will be guessing and picking AI-suggestions."

That is also ungracious to all of us amateur identifiers who were invited. According to the Validator candidate criteria, "If an identifier had made at least 3 improving identifications on a taxon in a continent, we considered them qualified to validate that taxon within that continent." It is my understanding that those of us who make identifications for others are a small percentage of the overall iNaturalist community. Those of us who met these criteria must be a smaller percentage, yet. I'll admit that I can rarely read through a dichotomous key without my eyes glazing over, but I do the best I can with the resources I can understand. And, yes, that means often using the CV to start my identification process.

If this case, it often was obvious that several others had already offered IDs as part of this experiment (when there were a group of recent IDs on observations that were years old). I made it a point to not be swayed by these votes and to only make my ID--often at a higher level--according to my own ability.

Would it be possible in future experiments to stratify IDs for identifiers by regions they predominantly identify in, rather than by taxa?

@yerbasanta If regions were the only thing categorizing identifiers, that could be a problem because many identifiers (like me) identify certain taxa across the globe. There are many types of identifiers so I think a mix of both the location and taxa (which is which is already in place to some extent) would be better.

"ungracious to the scientists who were invited" and "also ungracious to all of us amateur identifiers"

"Number of candidate validators: 4866" and "If an identifier had made at least 3 improving identifications on a taxon in a continent, we considered them qualified to validate that taxon within that continent."

Surely "3 improving identifications" are proving that person is qualified to validate ID at top level, I'm glad that you feel privileged to be included into this group.

@sedgequeen experts are just someone who professionally working in the topic, for sure some amateurs have better ID rates than some experts, especially in IDs by photo where precise taxonomical characters are often not visible, so person more familiar with species can ID it better by the slight differences in the shape, colour or something else on the level of "intuition". Or someone can simply have better knowledge on a taxon without published works. The question is how should we know that IDs of such person are correct if we want to validate something and don't have our own knowledge on the group? With expert we can just check the list of publications and already be sure that person have skills in identifying some group. It's not like experts are always better identifiers than amateurs, it's just that if you need to validate identifications you need a person with a background proving that this person is capable of doing so.

Btw, while the results page differentiates quite well when it comes to animals, plants are only shown as "Plantae". Could you differentiate a little more, like e.g. mosses-ferns-gymnosperms-monocots-dicots-other plants?

@bernhard_hiller for observations at coarse ranks, all the experiment is looking for is confirmation (or not) of the coarse rank. The results will be shown at different levels of precision (family, species, subspecies), so only go as far as you are comfortable with.

Antarctica?

Wow - 9,400 observations validated by 2,750 validators in 1 week - thank you so much to everyone who made this experiment possible! We added a Results section to this post.

Since I mostly coarse ID unidentified regardless of geography, I was pleased and scared to be selected. I did my set as accurately as I could. I was comforted in seeing identifiers and forum regulars whose expertise I respect being as clueless as I was and IDing only to "animals" or confirming my opinion based on layman resources.
Reading the comments, I am always amazed by the people who feel compelled to only identify to species and are flustered if they are presented with something beyond their niche. Looking beyond your comfort zone is like searching for gold, to find a nugget you need to sift through tons of dirt (and who knows what else is hiding in this dirt ;) ).

@wildskyflower - yes, that was a casualty of scaling the experiment from 1k to 10k observations in that we had to truncate the bar-graph links at 500 obs to work with Explore. But the data is safe in the database and you're right that making that more accessible (probably as spreadsheet exports) is something else we need to add to the list of improvements - thanks for mentioning this.

For at least one of these experiments I was sent a large number of plants and fungi (which I never identify), mostly blurry photos, and few examples of any of the taxa that I know and routinely ID on a large scale.

Was this a different experiment then the one summarize here? The images I was asked to check seemed to be mostly low quality and neglected ones of groups I know little above.

Usually I ID my favorite taxa to species-group or at least genus-group (e.g., subgenus) rank, but for the experiment in question I was selecting "Fungi" and "flowering plants" for many of the submissions.

@johnascher These experiments aren't about "ID behavior" at all. They're checking how accurately the observations were identified before the experiment, by throwing some new identifiers at the sample of observations and then comparing the observation taxon before the experiment to the new identifications.

Maybe the observations you were sent were already at a high level like Fungi, Plants, or Flowering Plants?

It certainly would help out validators to restrict by region I think -- I have a certain passing familiarity with most of the macroscopic life forms in the area I've lived in all my life, and I know all the flowering plants (graminoids apart) to some degree even in genera I haven't studied in detail, so that I'd feel comfortable trying to figure out any of them from reference materials. But I'm at a loss in areas where I don't even know the range of plants that exists, and feel sure I might totally overlook some important factor needed for identification. That said, I don't want observations in areas where few validators live to be neglected, so maybe a mix of taxon-based and geography-based trials.

@kharkovbut of course experts are also making mistakes, I've already explained in the next comment above that the point here is that if you need to validate identification you need a person (or better a few) with a background proving their skills. And it's not "3 improving IDs". We can't know if person is actually good in IDs or only claiming to be so if there is no background. There are thousands of RG observations with wrong or speculating IDs, including some repeating mistakes by the "customs" of some active identifiers who are not experts. I'm more than sure that there is a "direct correlation between formal "level of expertise" and ability to successfully ID things at platforms like this one", despite that some amateurs probably have better ID rateres than some experts. What they're claiming to be a validation here is just a slightly increased accuracy, so the results of this experiment will be misleading.

What does one do when they come upon a completely wacky suggestion from iNaturalist AI?

Don't use it as an ID, and use a disagreeing ID if someone else did use it as an ID.

you need a person (or better a few) with a background proving their skills. And it's not "3 improving IDs"

The concern isn't necessarily credentials alone, since some external credentialed experts who've never identified from photos misidentify a percent of photos identifiers email to them. Identification of physical specimens and from photos are only partly overlapping skills, and the second must be learned "again" even by credentialed experts. The actual problem for validators is that "3 improving IDs" is far too low of a standard, regardless of whether they have credentials.

Someone merely guessing IDs can easily meet that standard. But, this isn't meant to exclude anyone from being a validator either, since with effort many users can reach an even higher standard. Vs., using this too low of a standard will at least somewhat mislead results, and increase concerns like that some validators merely agree to the previous IDs.

@bdagley It isn't entirely clear which IDs are CV-guided. In the past, when the CV pulled up the taxon that I wanted to identify the observation as, I clicked on it instead of typing to save keystrokes! Since I learned that that gets a special notation, I now type anyway -- but lots of other people must be doing the same thing.

Add a disclaimer - no-reply - so iNatters know in future.

I would like to see your whole chosen batch of obs. Then let us (chosen identifiers) pick out the taxon and / or location for ourselves. And see where the IDs end up. We can still add a confirming ID as you requested. (Or you could count the Mark as Reviewed without us adding an ID - which is either silent agreement or I can't even)

In this experiment most IDers chosen as validators for the same set of observations as me seemed to take it seriously and be more cautious than they usually would be when IDing random observations.

My experience on the site in general is that on occasions when an amateur and a professional who both have a lot of experience IDing that taxa on inat disagree, it is basically a coin flip who will turn out to be right. Most often it is the second to post their ID, because sometimes people go fast and miss things (or mistype) on initial IDing but disagreement has more intentionality. However, in most cases, someone without a lot of experience IDing on inat/from photos is relatively more likely to be wrong regardless of their qualifications. IDing from photos, field IDing, and IDing from pressed specimens are basically three different skill sets and it is possible to be good at any one without automatically being great at the other two.

If the incorrect IDs have moved the CID to wrong, you can either opt out of CID for those obs, or is there someone you can @mention for another ID?
(Overly general without hard disagreement is 'untidy' but has no effect)
And I do understand the untidy - I would like to return to the 'broad IDs added for these experiments' and delete mine!

I second @erikamitchell's interest in measuring cultivated/captive organisms that aren't yet marked, as well as other DQA dimensions. I wonder about that for the region which I pay attention to. I find observations that have been RG sometimes for years with incorrect dates (easier if you live where there's distinct seasons with clues like snow on the ground), locations and/or wild/captive status. Presumably it's a small % overall, but it's part of the picture of iNat accuracy, and I feel like most identifiers on iNat pay less attention to DQA accuracy compared to their effort in contributing a taxonomic ID.

I was not involved with this apparent experiment, but I have two observations to add to this discussion. I check any submissions under Oecanthinae everyday, and have offered IDs on thousands of tree crickets, so I feel I have a bit more expertise than the majority of users for this taxa. 1) Why doesn't iNaturalist include a location algorithm? Occasionally I encounter photos where users have chosen a suggestion by AI that are far out of range. i.e. European species guessed for a submission from the US, or an Americas species guessed for a photo submitted in Africa. 2) Is there any way to include a pop-up suggestion that strongly encourages a user to stick to genus if they are not certain of a species? Seems like there could be another category besides Research Grade (which seems like it should only be moved to by true experts). Thank you.

Yes, the word "expert" is being overused in this thread, even being abused by some. My mental "credibility alarm" goes off whenever I hear that word.

@nan-cee I know multiple very experienced people including researchers who have empty bios, or have left their bio empty for an extended period of time. Not everybody identifies/explains themself in their bio. Plainly, it's not a way to tell if someone is an "expert." Skill is subjective and based on shown experience, which can be in the form of consistently identifying things correctly.

I appreciate you sharing your expertise @nan-cee!

@igor117 I am well aware and know some people like this which are quite problematic. I never said to simply count the amount of IDs somebody has of a common species and compare it to other people. As I said, it’s completely subjective. Do they have explanations for their IDs/are they able to explain them when questioned? Do they consistently make the same misIDs, i.e. are they spamming agree on common species with the result of repeated mistakes that are easy to catch? It’s about consistently demonstrating knowledge, not consistently pressing A on the Identify tab.