A second experiment to learn about the accuracy of iNaturalist observations

Thank you for your work!

Great ! More of this please :)

About 1: Rare species are rare, so the impact on the data will be small. I'd be willing to bet there is not a single such case in the 1000 obeservations.

Done. What a mess of non-flowering, probably unidentifiable dicots. But I put a label (e.g., dicots) on everything so you'd know I participated.

I understand this is not required anymore.

So great to have some resolution on the "Single Subject" issue. Thanks.

But what do I do with "duplicates"? Observations that are obviously of the same organism at the same place and time, but spread over several observations, or even a straight duplicate observation using the same picture for the same organism.

For this data quality assessment I should post the finest ID I could, but that would count as an improvement, on an observation others have looked at previously and refused to ID (as I would have) because it was a duplicate, and for which a much finer ID already exists on the original.
So by posting my ID I will artificially improve the identification (from Tribe rank to species), which is already at the species level on the original observation. And in doing so I will support a practice that I am totally against, and state so on any observations I encounter (and for which I have been censured for identifying them as "Life" whenever I encounter them).
So, if you dont mind, I shall ignore this observation. [And not disclose it so as to limit other fanatics from trying to swing it one way or another. ]

sorry - staging link updated

I like the changes to the experiment design. But I do share the concern that "at least 3 improving identifications of the taxon" might not prove sufficient expertise to validate IDs. Narrowing things down by continent is definitely a huge improvement but many identifiers are much more specialized on a certain region. Asking someone who usually identifies observations from Alaska to identify an observation from Florida might not yield optimal results. But I am very glad that these experiments are done, and I find the results very encouraging!

Echoing the love for the new “Evidence related to a single subject” condition to the Data Quality Assessment!

In the 'Methods' tab of the Observation Accuracy Experiment page: the figure of 'Step 1' lacks an explanation about what the brown 'C' hexagons are.

I agree with @tonyrebelo that seeing other identifiers IDs and comments will mean the the results based on subsequent IDers are essentially uncontrolled/not independent. On one hand, having IDers be able to use others comments/IDs mimics the process of IDing on iNat and is realistic. However, because IDers have information other than that present in the evidence (ie, photo), they will use it when making their IDs, so a rigorous statistical analysis of any data generated by this protocol would be difficult/impossible.

I am really excited about the new DQA! I think this will give a quick fix to a lot of observations that aren't ideal and free up identifier time and effort for other things. It would be great to have a recommended workflow for multi-taxa observations with the new DQA as @dianastuder asked about. Based on my initial understanding, using the DQA (eg, a downvote) could be done with no other action and be legitimate. An IDer could also still ID to the lowest common taxon which might be helpful in some cases (eg, two species in the same genus), though this isn't necessary. The IDer could also leave a comment asking the observer to delete/split photos and leave a comment when they do so (since these wouldn't generate notifications), though again, this wouldn't be necessary. Observers often don't respond to requests like this left in comments, but sometimes they do, and, in these cases, the DQA vote could be withdrawn if an observation is corrected to focus on a single taxon. Any other thoughts about best practices for using this DQA?

I'm also very glad to see the new DQA flag. I'm looking forward to a blog post (or better still an addition to the help docs) explaining how identifiers should use it. In the meantime, I'm intending to add it to my standard process for "multi-subject" observations.

ID at the lowest level that covers all photos.
Add a comment explaining how to split observations and tag the user.
Downvote "Evidence related to a single subject"

Also... is it possible to query the new DQA field via a search URL or via the API?

Hi all - thanks for all the engagement and great feedback on these experiments. We updated the stats (we’re doing this manually once a day now but will automate this in future experiments) and are already up to 548 (39%) of candidate validators with 90% of the sample validated. So exciting to see the capacity of this awesome community to do these kinds of experiments. Thanks so much!

@sqfp thanks for catching that error, a fix is in process. We’ve tried to bin the feedback raised so far:

Validator criteria
Lots of great ideas on how to improve validator criteria:

__ @insectobserver123 proposed loosening constraints on taxonomic criteria: use ancestor taxa (e.g. family/genus) rather than species (e.g. >3 improving ID of the genus encompassing this species)
__ @insectobserver123 proposed further constraining number of improving IDs criteria (e.g. increase from >3 to >10+)
__ @matthias22 proposed further constraining geographic criteria: from continent to state

As was raised, the strictness of the validator criteria trades off with our capacity to get samples validated, so it’s very interesting to keep an eye on this as we learn more about what kind of response we’re getting from the candidate validators (which again so far is very heartening). Re: @richyfourtytwo’s point about rare species, the 1-100 bar of the Taxon Observations Count graph is a good proxy for rare species. If you toggle the show frequency button you can see that it represents 35/1000=3.5% of the random sample

Norms for responding to observations of multiple species
Echoing ideas raised by @thebeachcomber, @dianastuder, @richyfourtytwo, @matthewvosper, @cthawley, and @rupertclayton please prioritize (1) voting no to the new DQA condition. After that it’s also helpful to (2) add IDs of the nearest shared taxonomic node and (3) to politely ask the observer to split the observation. But since 1 will move these observations to Casual, 2 & 3 are not as high priority if you don’t feel like doing these additional steps. Please don’t vote no on the “Based on the evidence can the Community Taxon still be confirmed or improved?” question for multispecies observations since we can now disentangle those. It’s not possible at the moment to query observations by DQA metrics though DQA metrics are contained in the observation API response.

Duplicates
@tonyrebelo and @matthewvosper both raised concerns about Duplicates. We agree that duplicates can complicate assumptions about the sampling bias and the independence of observations necessary for certain statistical analyses- and they can also be frustrating for IDers by creating extra work. But we’ll have to think more about how they are related to this framework of improving Accuracy and Precision. They are certainly related if they contribute to IDer fatigue or frustration. To be continued, but in the meantime, please don’t change your ID behavior on duplicates, even if you justifiably find them frustrating.

Other things we could measure
@tonyrebelo and @cthawley brought up concerns about validators being able to see existing IDs (e.g. this not being a blind assessment). Under the assumption that our validator criteria is sufficient, the fact that validators are no blind won’t impact our estimates of Accuracy and Precision of the sample of observations. But we agree that it would be very important to do a blind experiment if we were trying to measure validator skill. And that this might be necessary to test our assumptions about validator criteria mentioned above, or to learn more about related topics such as identifier skill.

@tonyrebelo also raised the idea of measuring the related question “Are obs as finely identified as they can be?”. We agree this would be a very interesting thing to measure, but would require different validator criteria and a different experimental design from the experiment which is meant to answer how accurate and precise the observations are at the time the sample was generated. But we very much agree that it would be interesting to measure what precision is possible, because as is implied species/subspecies level precision isn’t possible for many observations.

Thanks again and keep the feedback coming, we'll update the stats again tomorrow.

@loarie one point I should probably make is that the stricter the other criteria are the smaller the ID number threshold should be, I was thinking of anywhere in the 50-2,000 range using family and continent, but using genus and state the number should be smaller to avoid making the criteria too strict and to avoid the rare species issue, though I still think 3 is too small for any scenario. I also think genus is likely too fine a taxon, some genera are like 1-2 species. I also would suggest country as a geographic measure to consider, in addition to state and continent

this type which we estimate to be ~350k observations in the entire iNaturalist dataset.

Which was apparently okay to happen for 5+ years prior. As the bee researcher LeBuhn once wrote (paraphrasing), species IDs are all that really matter (excepting few genus data studies). The previous inat requirement to use broad IDs for multi-species obs. (which many identifiers, including some admin, justifiably "disobeyed") rendered 350,000 would-be species-rank records nearly useless. My point isn't to criticize Scott or the idea of these experiments (we want to test accuracy), but to point out that he essentially took very little of the experienced identifier feedback on the last experiment into account, and that many problems remain. Turning now to reply to some of the commenter comments:

But what do I do with "duplicates"? Observations that are obviously of the same organism at the same place and time, but spread over several observations, or even a straight duplicate observation using the same picture for the same organism.

Exactly.

I agree ... that seeing other identifiers IDs and comments will mean the the results based on subsequent IDers are essentially uncontrolled/not independent

Yes, but not every aspect of any experiment can be perfect, and as the inat admin have always claimed, there are so many constraints to what they can and can't do. Instead, I'd suggest focusing mostly on the major methodological flaws (e.g., too small sample size, too low validator threshold, etc.).

I would support any improvements that happen to the system, but you have to do something about blurry photos. I see some users uploading blurry, unrecognizable species and receive identifications. Personally I wouldn't feel comfortable identifying a species from a blurry photo but it seems to be happening. I would really like to see inat doing something about this as well.

@marieta55, identifiers should not add identifications that they are not confident in. But I'm not sure there's much we can do to prevent identifiers from ignoring these guidelines. We can detect blurry photos, but they don't necessarily indicate that observations are unidentifiable. For example, compared to possible alternatives I'm confident this is Chinese Pond Heron based on the breeding plumage pattern and that this is Giant African Land Snail based on the shape despite both being extremely blurry photos.

If an observation is sitting at taxon X and you are confident other taxa can't be ruled out because the photo is too blurry, please add a disagreeing Identification of the nearest ancestor you're confident in and explain in a remark why you don't think the evidence supports ruling out the alternative(s).

Nice hearing you are always improving!

@tonyrebelo I didn't mean slightly out of focus photos. I am talking about shades and shapes like the following link: https://www.inaturalist.org/observations/191483453
This particular user seems to be quite knowledgeable about birds but I believe that he doesn't have the right gear to record his observations. The majority of his observations though seem to be of poor quality, zoomed in to pixel level which makes them unrecognizable. Personally I believe there should be a standard for acceptable photos. Of course I could be wrong but I do take a lot of time trying to improve the quality of my photos so that my contributions can be usable to people.

The observations I got were of species I wasn't super familiar with. One was a Sarracenia in a California botanical garden, but the only ID's of Sarracenia I have are of adding subspecies level ID's to Sarracenia purpurea in Ohio, since there's only one subspecies present here. The other observation I got was of an Echeveria flower (no leaves), but in the Crassulaceae family I usually only ID Kalanchoe so I'm not very good with identification of the rest of the family.

@truthseqr I think everyone knows that accuracy is dependant on taxon - that's why it was made possible to break it down at least a little bit. The sample size of these initial experiments is too small to dig down meaningfully too far though. It would be a mistake to do a huge experiment straight away. Most iNat observations are not of very rare and difficult taxa, so it makes sense that accuracy per observation will be greater than accuracy per taxon.

The similar species page measures something totally different - it records every disagreement with an ID of that taxon, not just disagreements after reaching research grade.

Question: Who can clear the "single subject" cross and how easy it is?

If it was your cross, you can click it off again.
If not yours, then you click a counter-vote.

@matthewvosper,
1) If the "Taxon Info --> Similar Species" page measures ALL disagreements with an ID of that taxon and this is not something the team wants to measure, why not create a different page that measures disagreements after reaching research grade? To do this programmatically is much more efficient than conducting many small experiments of this nature.
2) If an observation is misidentified, it could sit in the database for years before it's corrected and captured by the criterion that misidentifications are only counted when they're posted against a research grade observation.
3) I would suggest that the team should not proclaim 95% overall accuracy as was done after the first experiment, given the fact that misidentifications are species-specific.
4) I have personally viewed thousands of misidentifications of Geranium, Erodium, Fumaria, Malva, and other genera. That's a lot of misidentifications and it makes me skeptical of claims of 95% accuracy.
5) The reviewers are vetted for these experiments, which is not the usual case for iNat reviewers. I think this skews the data considerably.

@truthseqr I'm a huge fan of your misidentification journal posts. Thanks so much for continuing to do all those studies.

As matthewvosper mentioned, these Observation Accuracy Experiments are currently designed to estimate the overall accuracy and precision of all iNaturalist observations. In contrast, you're measuring the rate of misidentifications which is also very interesting and absolutely something we should learn more about as it relates to identifier skill. But they are different things. For example, for Fumaria densiflora (choosing Filter by Place = United States) I see 68 observations and 6 records of Fumaria officinalis under the Similar Species tab. But from an Observation Accuracy standpoint, none of those observations are currently not Accurate since they've all been rolled back to genus (we're measuring the difference between an obs sitting at Species and an obs sitting at genus as lower Precision but identical Accuracy).

As mentioned we're aware the current sample size of 1000 might be too small to provide robust estimates. But Observation Accuracy Experiment v0.1 estimated the accuracy of the Research Grade subset to be 95%. The accuracy of the Verifiable subset (which includes Needs ID as do the counts on the taxon pages) estimated the accuracy to be 90%

But the big difference is your studies are measuring the misID rate, and these studies are measuring the accuracy of observations which takes into account things like the observations rolling back to genus when there are disagreements. In terms of measuring misID rate, I think the percentages are actually worse than your studies estimate because the Similar Species tab only detects situations where alternate IDs were added, not all the observations with just 1 misID where alternate IDs have not yet been added.

@loarie, thank you for the explanation. Yet the issue if this value affects research grade or not is important - those controversial practices like habitat photos or bat sonograms are undesirable for AI learning but helpful in various biological respects, including those desiranle in GBIF, which adopts only RG observations. I hope these uses can be separated at the software level, like making somewhat different subsets of data passing to such general data using/accumulating systems like automated identification and GBIF.

Not sure if I am out of topic here, but I have noticed people are uploading images that are tempted. For instance, a person draw on the image, trying to be funny. I have reported the image, but the moderator said it was not an issue. Maybe, a box for separate those images, where the observation was modified, like any draw, from the pure images. It is useful for machine learning to use just pure image.

for duplicates @truthseqr
I put the other obs link in a comment
and vice versa
or - list all 3 or 4.
I've seen them, I've found them - I have done the work, just need to copypasta in a comment.

@richyfourtytwo I am saying to have a tag, as so we can decide what to feed our models. With a tag, we could use a filter to filter out those images.

@oleg_kosterin I would recommend and even insist to change the wording from “Evidence related to a single subject” to "The subject is concerned in all photos" (or something with better English)"

Exactly. Something like the “subject or focal organism is in every photo.”

@dianastuder And this needs updating please? https://help.inaturalist.org/en/support/solutions/articles/151000171680-what-do-i-do-if-the-observation-has-multiple-photos-depicting-different-species-

Definitely. Incidentally (but which is on-topic) also that that page that refers to the past policy, says "it's best to identify to the level that fits all photos," which means that was technically only a suggestion, not a mandate. Yet, I was flagged (and singled out among other users who made the same ID) by two curators a year ago who mistakenly asserted that I violated site policy, which was part of why my curator status was invalidly removed.

@loarie But Obs. Accuracy Experiment v0.1 estimated the accuracy of the RG subset to be 95%. The accuracy of the Verifiable subset (which includes Needs ID as do the counts on the taxon pages) estimated the accuracy to be 90%

But we already know, from our inat experience, and many reasons users gave here and elsewhere, that both of those must be (probably high) overestimates. It's misleading to refer to those results as "estimates," which could imply high confidence. Secondly, your older link now directs to a nonexistent page. Another relevant thing to do would be to calculate the average CV accuracy across all taxa when an observer uses it as the first ID and there are no subsequent IDs added. That might not even require a user-experiment (e.g., we know that photos of grass aren’t birds). I predict/infer “initial CV accuracy” to be low. From that, we could infer more about how accurate these recent accuracy estimates truly are. Plus, as you here granted, the sample sizes are too small in the new experiments. Validator eligibility also remains far too low, which you didn’t address.

@ any/all users, and which is on-topic here, I recently compiled a list of requested improvements for all aspects of inat (some of which were from other users), which is unique in including some requests that aren't on inat or the forum. Including e.g.'s like suggesting observers crop photos, an AI auto-photo cropping tool, an AI tool to ask "is an organism present?" for photos that seem to lack any, and a request to "recommend or require observers don't put watermarks or logos on top of the animals in their obs." The last one was actually censored from being posted in the past by staff, despite that they said it wasn't inappropriate. Many of the requests, if voted on or implemented, would significantly increase the accuracy of the CV/AI and of RG % accuracy.

Since the request list is very long, being a compilation (but probably shorter than the forum feature request list), I recommend viewing only a portion of requests per day. Some requests near the end are about website fairness, privacy, etc., but are written appropriately with constructive/cooperative intent re: the admin. Feel free to share with anyone who might be interested. The requests I wrote come from 4 years of extensively using inat and BG (which has higher accuracy) and often thinking about accuracy. Thanks for considering it. https://docs.google.com/spreadsheets/d/1b0VFdSO5iI8PbovZ4YNjyoIUGxP2KOy8iFZXz-ak43I/edit#gid=1754535465

In this discussion of why 95% accuracy must be an overestimation, I think something important is being ignored - the variation of error rates among taxa. Many species are virtually unknown except by people really interested in them, so they're rarely misidentified at the species level and any misidentifications rarely make it to RG. Also, some taxa are regularly checked and corrected by dedicated identifiers So although we're all well aware of (and frustrated by) taxa for which misidentification is common and error rates are high, we should remember that those get averaged with other taxa that have really low misID rates, especially for RG observations.

It would be great to move some of this discussion to the forum where more messaging and moderation tools are available.

Given that not everyone on iNat is on the forum I think it is good to be able to discuss things directly on iNat, though there is nothing wrong with talking about this on the forum as well

Given that the concept of the new DQA vote option of whether the ID subject is in every photo of the obs. has come up in this discussion, and that I (as in the past) suggested that it would be even better to allow the obs. to be unlinked (split) into two verifiable obs. (vs. rendering both photos unverifiable), that terminology that some identifiers like johnascher use originates from the similar website and early inat influence Bug Guide. This is an example of how it works there: https://bugguide.net/node/view/671812.

@gcwarbler lets use the 30 species of the duck genus Anas as an example.

If an observation was at species Mallard, its precision would be 100% because its precisely placed to a single species (Mallard). If the observation was at Genus Anas, because there are 30 species below the genus the Precision would be lower: 1/30=3.3%. If it was ID'd at family Anatidae, since there are 165 species below the family, the Precision would be 1/165=0.6% and so on.

Just like we report average Accuracy by taking the average of the Accuracy of each observation in the sample, we're reporting average Precision by taking the average of the Precision of each observation in the sample. For RG obs its close to 100% because aside from a few exceptions RG obs are all at species. But its lower (~77%) for Verifiable obs which aren't.

The reason we want to measure Precision is that it trades off with Accuracy. For example, if we rolled back all Duck observations to Family, the Accuracy would probably be very high, but the Precision would decrease. We want a dataset that is both Accurate and Precise.

The Precision statistic is nice in that it gives obs sitting at species a high value (100%) and obs sitting at coarser nodes lower values (approaching 0%), but its not ideal. For example some drawbacks are:
-- Its based on the number of species in the iNat taxonomy. This works well with ducks since all 165 are in the iNat taxonomy, but for groups like beetles where we probably only have a small percentage in the database and more are added manually every day, the denominator is shifting.
-- Like CV and the Species counts, we're ignoring subspecies and hybrids which isn't ideal because people do ID at these nodes. But if we included the 2 mallard subspecies which are rarely used by IDers, an ID of species Mallard would have a precision of 1/2=50%

@amanithor can you email help@inaturalist.org to try to troubleshoot your issue with the URL? I suspect it has something to do with your browser

Given the tangential mentions of CV accuracy above I might just mention a little experiment I did with the hoverfly genus Eristalis last year and the year before. First time I sampled 100 needsID whose current ID was at genus, second time only 50. I found the CV got the species right 75% and 76% of the time respectively. This included as many as 12 species on 5 continents. I thought it was pretty good. For context, Eristalis can be considered a genus that is difficult to ID from photos, but rarely impossible. It does generally have a lot of iNat observations to train from, but not for every species. Most of the errors were from poorer quality pictures. I thought that was pretty fantastic - certainly good enough to make identifiers lives easier. Maybe I should repeat it!

I agree with @matthias22 that constraining validator eligibility to continent is an improvement. However, instead of further constraining eligibility based on political boundaries, why not put a circular boundary around the observation and use that as the selection criterion?

2 of 2

“Anecdotally” (from experience), extensive identifiers, even ones who work in groups that may actually have 95%+ accuracy, can also intuit/infer/notice that 95% accuracy would be an overestimate as an average across all taxonomic groups. Because, as kueda said, accuracy and expertise vary significantly between groups, and we know of groups with much less review and fewer “expert” vetting. kueda and others also noted the inherent problem of groupthink, where all the “experts” think an ID is accurate but which they later learn isn’t, which would also contribute to overestimating % RG accuracy.

In conclusion, the above seems to suggest the following: the titles or descriptions of the recent and ongoing experiment results should be more significantly caveated and noted to likely be overestimates (or if not believed to be, provide compelling reason why not, and ask commenters what they think). Already, inat being estimated as 95% accurate is one of the top Google Chrome results to the keyword terms, without clarifying, so changing this post’s title would also be ideal. Also, it would be ideal to post each new experiment in three posts: the first asking for feedback on the design, the second stating how the experiment will work, and the third reporting on highly caveated results and asking the community if they interpret them to be accurate. Multiple similar iterations of the same experiments should also ideally be done: one using the current validator standard, one including all identifiers, one including the older much different “taxonomist expert” standard, etc. The sample size of identifiers, obs., and IDs should also be massively increased, and calculations on current CV/AI accuracy should be done, and possibly incorporated into the experiments, their results, and the interpretation of the accuracy of th results.

Versus, (the title itself of) “We estimate RG accuracy is 95%!,” make it sound like the estimation is highly accurate and not an overestimate. Presenting the results in a positive manner (e.g., the explanation mark) is also misleading, since 95% means 5% of all RG records sent to GBIF are wrong, and can make commenters feel like, or be perceived by others as being, “negative” to suggest reasons why the results would seem to be overestimates, incomplete, or otherwise misleading. By contrast, keuda’s earlier title and writing on loarie’s 2017 experiment was far more neutral and provided more caveats about the limitations, asked skeptical questions, etc. Overall, part of my main conclusion is that no methods yet used would even allow us to arrive at a good estimate of %RG accuracy. For this reason and the fact that many or most consider the stats overestimates, it wouldn’t be ideal to report these findings in other sources. Lastly, it also seems evident that since the staff are clearly trying to achieve and demonstrate a high % RG accuracy, that the website design should be tailored more toward ways that increase photo quality and RG accuracy.

So I did the Eristalis experiment again... 125 observations this time. I made it harder for the CV than I had done before by setting the taxon fliter to 'life' rather than 'by taxon', to make it more likely to get the genus wrong. (On the few occasions it tried to identify the plant or another thing in the picture, I just ignored those and picked the highest ID that seemed to have the correct target). Of the observations I could put a species on it got 86% correct this time! 16 different species on 3 continents. Those regular new releases must be making a difference. Of the 17 it got wrong, one was a species not known to the CV, 5 I noted had pictures in which the specimen was very small, some of the others were overexposed or otherwise difficult. Only 7 times I thought the CV had no excuse - including a very clear E arbustorum it bizarrely thought was Eristalinus taeniops, a decent, if slightly small picture of the densely hairy E flavipes that it thought was the very unhairy wasp Scolia dubia, and a fairly poor but still clear E tenax on a white wall that it interpreted as a cat flea! Overall, pretty brilliant.

2/4 So, as some commenters implied, these experiments are mostly actually estimating expert ID accuracy (including the accuracy of the other users who typically "follow" the experts), and mostly for RG or to-become-RG obs. Vs., the original description of the 2024 experiments made it seem like it was testing the avg. accuracy of all obs. and/or all identifiers. Part of the expert user effect is impossible to overcome, because the community does already tend to "follow" experts, unless an experiment was to be done where each participant can't see any IDs made by other users. Regardless, one takeaway is that even if a relatively high accuracy were to be estimated, it should be clarified that that's most likely at least somewhat overestimated and is a far higher than what the avg. ID accuracy of all users would be (in all circumstances, incl. where only they made an ID), and is much higher than avg. initial AI accuracy (which was only near 50% in 2017, excepting the subset of "we're pretty sure it's this taxon" suggestions that had less-low accuracy).

As I suggested, it would seem ideal not only to a conduct number of versions of these experiments, but to intentionally consider "the" experiment and "the" eventual results of it as including or considering a complex of different iterations of the experiments in presenting and interpreting them and asking if users/readers agree. For e.g., the following experiments would be useful to conduct (possibly multiple times), and it's also necessary to massively increase the obs. sample size for each (and we already have data from small obs. sample size experiments, which we can keep in consideration when later interpreting all results:

The current experiment version, the current experiment where all users (who participate) are incl., the current experiment with a more difficult validator eligibility, the current experiment with a much higher validator eligibility (and/or repeat the similar 2017 experiment), an experiment where validators can't see what IDs other users made or at least can't see who the users are, a current assessment of initial AI accuracy (when an observer uses AI for the initial obs. ID), an assessment of AI accuracy when an AI ID by a user after a preceding user’s AI-ID was incorrect, an assessment of AI accuracy when an AI ID by a user after the preceding user’s AI-ID was correct but was only at a broad taxon rank, and iterations of all of these experiments. Add’l variations or iterations should also try designs where most (or varying percents) of incl. obs. are expected to become RG or casual, or (in separate versions) Needs ID or casual by the experiment’s conclusion.

4/4 The following e.g.’s can be considered further/used. A small # of experts recently completed IDing all 6,000+ global Delta obs. (somewhat cryptic). But, the avg. identifier Delta IDs and initial-AI-IDs of Needs ID Delta obs. remain similarly inaccurate (no noticeable difference).

I and add’l experts also IDed approx. all European Eumeninae (very cryptic overall) Needs ID obs. Mostly reviewing Needs ID obs., I found/corrected 30 to 50% obs. that had been misidentified to genus by users or the AI. Yet, the current AI accuracy there still remains similarly low, even for genus suggestions. This also means that if the experts (and to clarify, anyone can in principle become experienced, it isn’t a label necessarily excluding anyone) were to suddenly stop IDing, the low AI accuracy and avg. user ID obs. ID accuracy would further lower overall inat % accuracy. This means the AI at least at least currently is at least partly counteracting wrt Needs ID and RG inat obs. accuracy.

Add’l AI problems contribute, e.g., that the Compare/Suggestions Visually Similar filter seems not to incorporate the Location filter. Furthermore, the fact that even locations with faunas currently completely accurately IDed by experts would regress to lower user and AI accuracy if the experts left, indicates that multiple add’l aspects of inat are currently at least partly counteractive wrt accuracy. It’s unfortunate that the hard-won complete ID of regional faunas could be entirely lost in such a scenario, esp. because that would mean the avg. remaining regular identifier would also be ignoring the available past obs. and comment history explaining how to ID the complete fauna experts “left behind for them.” And, no user including experts can ID forever, and even active high-volume experts eventually can’t sustain their volume (incl. me, in the future), so the scenario is possible.

I’m unable to put as much time writing/thinking about this in this post or future ones, so ask people to mostly refer back to my comments here and elsewhere, or they can ask me in the posts specific questions/comments. In future inat accuracy posts, I’ll also make briefer comments if any, but may still actually disagree with some design methods, results, and results interpretations/portrayals even if I don’t comment, if they contradict or overlook points I and others made. Last thoughts. Re: muir suggesting this post be put on the forum for moderation tools etc., I instead agree with the curator who noted why it should be here so everyone can comment (many experts don’t even use the forum). Ironically, a forum mod did flag one of my above comments, claiming I insulted Scott, but Scott resolved that flag as that I didn’t, so imo and demonstrably ime forum (and sometimes inat) moderation is sometimes unfair or inaccurate. Finally again, I consider the current Google search term top result reference to inat estimated to be 95% accurate too misleading, since many Google users won’t read the details past the title, so suggest editing the post title.

These estimates of avg. accuracy of the entire RG obs. dataset are in line with our expectations, largely because the iNat. dataset is skewed towards a relatively small number of common, easy to ID species (e.g. mallards, monarchs, etc.) that have an outsized impact on the average.

A. mellifera would be another easy species e.g., with verifiable RG obs. accounting for 26% of all verifiable Bees obs. This in combination w/ the fact that community and individual IDs are mostly by or influenced by experienced identifiers (“experts”), who can ID nearly 100% of the easy species in their groups, means that the results are mostly only about the accuracy of experts and those following them IDing the most common species that (increasingly over time as obs. are added) flood out all other species. Attesting to how "easy" [verifiable obs. only] such species are for experts to ID is overall, I found in Jan. that of the total of all verifiable obs. all global genus Apis species, 99.22% were RG and presumed correctly IDed, vs. only 58% of the total global verifiable Bees (all species) obs. that have been IDed so far were RG (species= 3,511).


Furthermore, global bee species overall range from easy, slightly, highly, to impossibly difficult to ID to species (1,444 verifiable RG global bee spp. have been IDed to date). I'm also unsure if broader-than-species taxa that nonetheless also flood out most other bee species that are easy to make RG via DQA (and likewise for other taxa groups) (e.g. Zadontomerus and Dialictus) factored into this experiment. If the caveats and explanations were made clearer in the result designs, results, and the results titles/reporting, I may agree that these specific experiments are somewhat accurate but possibly still insufficiently imprecise due to obs. and/or identifier sample sizes (which the designer and many others stated might be a limitation). So, these results can still be potentially misleading or mostly involving "atypical" circumstances that aren't made abundantly clear upfront to readers (incl. e.g. in top Google search results). The fact that the RG accuracy results are nearly 100% reinforces this, which is also nearly the same percent of the verifiable A. mellifera obs. that are RG at any given time in my calculations.

While experiments like this can be useful for showing what they can based on their specific design, to truly report on what many would consider a more accurate, common sense, or complete accounting of inat accuracy would require doing more experiments, including ones that exclude all the numerous easy species (note: this is actually done in some publications, although methods vary) or in some way to include equal numbers of obs. per species, and to use larger sample sizes (etc., see previous comments). Because, for e.g., we know most observers observe the most abundant and easy to ID species in urban areas (a bias that was also scarcely if at all mentioned), which flood out most of the other site-wide species data. It is good that many users participated, though, and I'm interested to see what additional experiments or ones that take participant feedback upfront and post-experiment interpretations into account may find. I still predict BG has hgiher accuray though, for e.g.

Great that these experiments are continuing!

I just want to echo a couple of people above in saying that

it would be great to have this conversation on the forum instead where we can respond more easily to other people's comments, reference in other forum threads, etc.
Also...to echo someone else's comment that I think it can be a bit misleading to use blanket statements such as the "research grade accuracy is 97%" subtitled on the main page. This is a complex topic and as touched on above, the exact meaning of accuracy in this context is slightly ambiguous... varies massively by taxon, etc. Clipping the number into a headline like that undermines things a little in my opinion, however interesting the data and experiment might be. Soundbite numbers tend to be endlessly repeated on the forum as well as off site as hard data, usually lacking context, even though at least with the last study which I looked into in more depth, it seemed the 95% number which was being touted lacked real substance. To me the numbers here seem again problematic to float around without context, however much insight they might provide to everyone. Not that it's not good to report the findings, but yes... good to not fall into the trappings of a sort of tabloid headline science approach.

Those thoughts aside, Great that there is a responsiveness from the staff to community feedback and that they are being developed further. I hope they continue to be a regular feature and I really hope that alternative methodologies are also explored.
It would be great to hear discussion about the approach more generally on the forum.

I agree with most of @sbushes comment, except that this discussion should be had instead on the forum. One additional reason the discussion should be had here on inaturalist is that the experiment is attempting to recruit users to participate, by using direct private messages. (Side note: private messages between users aren't actually currently private to the staff, only their private messages to each other are, which is wrong, since the staff should only need to see private user chats that get flagged.) So, that would be like if in a next experiment, inat sends private messages to inat users, but then tells them that the discussion will be on the forum, but only a portion of the users even have a forum account, and many experienced identifiers in particular don't use the forum.

It would also be confusing if the expertiments were discussed both on inat and on the inat forum, although I suppose non-staff forum users could choose to start additional forum discussions on these, but which would exclude some of the participants. As for following the various "threads" on this post, it actually isn't that confusing or hard to follow. This is exactly as the Bug Guide forums work and they have no moderation at all. As for me, I've given long detailed thoughts on the 2017 experiment most users seemed unaware of and on these two recent 2024 ones, and feedback for what I think would improve them, and things to avoid. So, I won't be repeating anymore long messages like that for subsequent experiments; if anything I'd only write briefer comments or otherwise recommend users refer back to my earlier comments.

One last slightly problematic issue comes to mind: A. mellifera (western honeybee) is one of the most observed and easiest to ID species, but in many locations are from managed hives/apiaries but not annotated as captive. There's a general lack of understanding among many users that if the bees are literally seen coming from a managed bee container, they should be categorized as captive. It's even come up in the past that evidently some staff think it's okay to keep honeybees known to be from managed colonies as wild, which disagrees with the DQA and guidelines.

The slight problems I'm pointing out here is that a large portion of the species included aren't actually wild, and therefore should've been excluded (which affects sample size, etc.). I suggest that the guidelines or other guidance add clearer guidance on Apis mellifera: if they're known to be managed they should be categorized as captive (making them casual grade) but if one is seen in the wild where both wild and managed honeybees can occur they can be left categorized as wild.

I'm not sure the numbers are meaningful without more context. An accuracy value for iNaturalist as a whole does not tell you what you want to know unless the sampling biases in your data usage match the sampling biases in the data used to create the accuracy value.

I think it's pretty obvious that there are subsets of the iNaturalist data where ID accuracy is, for all practical purposes, 100%. And subsets where ID accuracy is more like 25-30%. As a data user, I want to know where I am relative to that variation.

Thank you for your invitation. Pity, your time window was much too small to take part in this interesting event.

I second @matthias22 about narrowing down to state or other localized geographic region. As somebody who identifies things almost exclusively in one or two states (or even counties), I did my best on these experiments, but felt under-qualified due to the fact that many observations I was being asked to ID were hundreds or thousands of miles away from my knowledge-base and resources. Thanks for the opportunities and continuing to refine the platform and its use and accuracy.

Intriguing idea, @dianastuder! One could think of an experiment where only the observations are chosen but not the validators. The test observations would be made public and the whole community invited to weigh in according to each one's expertise. Each identifier could pick the taxa and regions they are familiar with. This might be a good system for validating rare species with relatively few observations that require fairly specialized expertise. Definitely not a good method for easily recognizable taxa with tens of thousands of observations such as mallards and honey bees. I don't know whether this is feasible. Perhaps it's just a crazy idea but I wanted to put it out there. It has a crowd aspect to it where the community could work together in order get an observation to the finest taxonomic level possible and to the greatest ID accuracy possible.
I would definitely like to see future experiments that focus on more circumscribed taxa or regions. Doing these experiments for every kingdom worldwide is very generalized. It would be helpful to our various communities such as botanists, entomologists. herpetologists, etc. to have more specific experiments that provide more applicable results to their fields of study.
Definitely love all this, and I am very motivated to contribute to the best of my abilities to future experiments!

@tonyrebelo and @cthawley concerns if seeing existing IDs is not a blind assessment are true, but that is the "living" situation in IDing observations as well as in both experiments at present. But this is a chance to prove your own knowledge whether you can "agree" to already present IDs or if you have to look up other sources before giving an ID. In case of your own uncertainty or doubts, not make any ID, leave the present page and move on to the next observation.

However it would be a nice and important experiment to make additionally a blind one, once. iNat would be able to measure validator skills and you as an IDer could learn as well a lot.

Good discussion. In a conventional research setting (e.g., taxonomists working at a natural history museum or a university), identifications are not blind in many cases, either, unless one deals with newly collected specimens. The determination history of specimens in natural history collections is carefully recorded and visible to any researcher who works on those specimens. In this regard, iNaturalist as a citizen platform is no different. Observations first come in as unidentified and then the determination history is carefully documented and visible to all the citizen scientists who look at them later. Therefore, I would not overemphasize this issue. Citizen science and institutional science work under very similar conditions in this regard. That does not mean, of course, that it wouldn't be interesting to find out how much ID choices are influenced by previous IDs. In fact, that's a fascinating question which I personally find intriguing. Based on my own observations here on iNaturalist, previous IDs can have a great influence on subsequent IDs, including when they are wrong. Initial incorrect IDs sometimes even influence experienced identifiers who know better to agree with them. The "Agree" button is deceptively easy to use :) Research on this is definitely to be encouraged. I would love to see a study where the same observation is presented to two groups of validators, one with a correct ID and one with an incorrect initial ID. And then quantify the different outcomes. That would obviously not be possible with existing sets of observations because one would have to create duplicates.

Well put, @tonyrebelo !

time-consuming & not very effective

Both true. But mine is visible to future identifiers. Your spreadsheet is only for your benefit.

I hope for more new DQA.
Duplicates (one, or more, to be deleted). Meanwhile Casual to take them away from identifier's backlog.
One subject scattered across many obs (to be combined). Ditto re Casual.

If the issue would only affect me - I would Mark as Reviewed - and keep moving along.

Thanks, @tonyrebelo. I'm aware of that field and I was using it for awhile. The part I was missing was how to find the duplicate observations after using that field. Thanks for the URLs. That helps a lot. I'll update the duplicates I've found.

Thanks @xris - that'a a great solution to the duplicates problem.

@truthseqr Your "Suggestion for selecting experts: each species has a list of top identifiers..."
True, but if iNat would only select the first ones, you may end up with a surprise and a very skewed result. I will now only reflect of top-identifiers of lepidoptera species, a field I am familiar with and having contributed a small share of IDs as a lay-identifier. If this list would reflect only those identifiers who contributed mostly "improving" IDs, or the first "supporting" or "leading" flagged ones (the latter in case of correcting previous wrong IDs), then this assumption would be very good.
Matter of fact that is a minority. While otherwise very knowledgeable experts often are found in the back ranks, pushed back by multi-clickers with "supporting" percentages above 85%. OK each of us can decide whom to contact to be helped with difficult IDs. An automated system that does only select the top-notch identifiers will not reflect the real expertise in iNat, hence a random selection is still the best.