May 09, 2017

Identification Quality Experiment Update

Hey folks, just checking in re: the Identification Quality Experiment. I really appreciate everybody's help. We now have over 3,000 expert IDs to start doing some analyses. Based on the data so far, 92% of the sample of observations were correctly ID'd. This is relative to the expert's IDs and assumes the expert is always right, which as @rcurtis, @charlie and others have mentioned isn't always the case, but lets assume its generally true.

One of my main questions is whether we can quantify how the accuracy of iNat observation's (that is the probability that they are properly identified) varies based on the contributions of the 'crowd' of IDers. My hunch is that 'earned reputation' of individual identifiers can be a pretty good indicator of whether an observation is properly ID'd or not. For example, if I've properly ID'd ladybirds a lot in the past (at least relative to the opinion of the community) does that mean that future observations that I identify as ladybirds are more likely to be properly identified? My hunch is yes, but in order to prove / quantify / model the relationship, we need a set of expert-identified observations (which is where your contributions to this experiment come in).

Let me first provide some background to my thinking about 'earned reputation'. I've grouped identifications on the site into four categories. Lets say an observation has a community ID of 'Ladybird Family'. The community ID is the consensus identification that emerges from everyone's IDs and dictates where an observation hangs on the tree of life. If I add an identification of 'Seven-spotted-ladybird'. This would be a 'Leading' identification since it is of a descendant of the community ID, but one that hasn't yet been corroborated by the community.

Lets imagine that enough people corroborate my 'Leading' identification that the community ID moves from 'Ladybird Family' to 'Seven-spotted-ladybird'. This would then become an 'Improving' identification because it is one that moved the community ID ball forward (ie it was the first suggestion of a taxon that the community later agreed with). I'm referring to the corroborating IDs as 'Supporting' identifications. They helped move the community ID from 'Ladybird Family' up to 'Seven-spotted-ladybird' but they didn't provide any novel suggestions.

Lastly, lets say I proposed something that the community disagrees with. For example, lets say I added an ID of 'Honey Bee' but there was still enough community weight for 'Ladybird Family' that the community ID remained there. Because my ID of 'Honey Bee' is a lateral sibling to the community ID (rather than a descendant or ancestor) lets call it a 'Maverick' identification. This doesn't mean I'm necessarily wrong, but it does mean I'm out of step with the community.

Now lets imagine that there was an observation of 'Ladybird Family' on iNaturalist and all we know about it is that it was identified by me. Lets call the number of times in the past that I made 'Improving' identifications of ladybird 'wins' (that is the times that I was the first to propose ladybird and the community later backed me up). Similarly, lets call the number of times in the past that I made 'Maverick' identifications of ladybirds 'fails'. Lets say my track-record for Ladybirds was 30 wins and 1 fail. Is there a correlation between this earned reputation and the probability that the observation ID'd by me as a ladybird is properly IDed?

The graph below shows all ~3,000 observations in the 'expert ID' sample as points colored by whether they were properly ID'd (gray) or improperly ID'd (black) relative to the expert-opinion. The x-axis shows the total number of 'wins' summed across all the identifiers contributing to that observation before it was assessed by the expert in this experiment. For example, if an observation was ID'd be me and charlie, the community ID was 'Carrot Family' and charlie and I together had 300 'wins' from past observations of carrots we'd ID'd, then the observation would sit around 300 on the x-axis. Similarly, the y-axis shows the number of 'fails' summed across all identifiers. The colors increasing from blue to red is the modeled probability that an observation is properly ID'd based on the number of win's and fails.

The model shows that there is a strong correlation between the number of wins and fails from the community of IDers and the probability that an observation is properly ID'd. For example, while the model suggests that an observation ID'd by a community with 0 wins and 0 fails (e.g. we know nothing about their earned reputation for that taxon) is 96% correct (lower left of the graph), an observation ID'd by a community with 200 wins and 0 fails for that taxon is 99% correct. Similarly, an observation ID'd by a community with 10 wins and 10 fails is 62% correct.

2-caveats: first, we know that the expert-IDs aren't always right so there's reasons the community is probably performing better than this model would indicate. But similarly, ID's aren't all independent (there's some group-think) so the community might be performing slightly worse than this model would suggest (which assumes IDs are independent). Second, we know there's a lot of other factors at play here. Other observation characteristics like location and type of taxa probably influence probability that obs are properly ID'd.

But as a simple, first-order approach. This study seems to indicate that there is a strong correlation between taxon-specific past earned-reputation among the identifiers of an observation and the probability that the observation is properly identified. This makes intuitive sense, but its cool we can quantify it. It would be pretty neat to use something like this to be a bit more rigorous and quantitative about the 'Research Grade' threshold which currently is pretty naive. We could also use something like this to try to speed up getting observations to 'Research Grade' status by putting them in front of the right identifiers (who we can target based on this passed earned-reputation). But there are also ways that this kind of system could bother people. Maybe it unintentionally gamifies things in a way that undermines the process. Or maybe its too black box and turns people off ('why is this observation Research Grade and this one not?').

Curious to hear you're thoughts. Also 3k is still a relatively small sample so would love to repeat this analysis with more IDs. So if you have the skills to help and haven't already joined the Identification Quality Experiment please do. And if you have joined, please add as many ID's through the interface as possible. Also curious to hear other thoughts on whats working / whats not working with the experiment. I know there's some concern about things like (a) ambiguous subjects of photos, and (b) accidentally stomping finely ID'd obs with coarser IDs. I'd like to find ways round these issues, but in the short term, skipping problematic obs should suffice.

Thanks again!

Scott

@charlie @fabiocianferoni @vermfly @jennformatics @d_kluza @arachnojoe @cesarpollo5 @ryanandrews @aztekium @lexgarcia1 @harison @juancarlosgarciamorales1 @garyallport @echinoblog @jrwatkin68 @bryanpwallace @wolfgang_wuster @bobdrewes

Posted on May 09, 2017 08:54 PM by loarie loarie | 11 comments | Leave a comment

April 05, 2017

Identification Quality Experiment Update

Thanks to everyone who signed up for the iNaturalist Identification Quality Experiment.

We're still getting the kinks of the study out before kicking this study into full gear, but we do have a small sample of 1,156 expert IDs to start playing with. Comparing the taxon suggested by the expert with the taxon associated with the observation just before the expert's suggestion, three things can happen:
1) match (88%): the expert's suggested taxon matches or is more precise than the taxon previously associated with the observation (e.g. observation is ID'd as Taricha or Taricha torosa and expert suggests Taricha torosa).
2) maverick (8%): the expert's suggested taxon is sister (maverick) to the taxon previously associated with the observation (e.g. observation is ID'd as Taricha torosa and expert suggests Ensatina eschscholtzii).
3) too precise (4%): the expert's suggestion is less precise than the taxon previously associated with the observation (e.g. observation is ID'd as Taricha torosa and expert suggests Taricha).

Two issues have became clear in these early days:
1) Ambiguous subjects: Because expert IDs are made blind, context like the description that observers often use to describe the subject of their observation may be missing. That can lead to situations like this where the expert thought the subject was the Laurel Sumac rather than the California Brittlebush. We're considering functionality to make the subject of an observation more explicit, but for now, skip observations if the subject is ambiguous:

2) Ambiguous disagreement: We are assuming that an expert's ID is the most precise ID that ANYONE can make based on the evidence provided. Thus, if an expert ID's an observation to genus that was previously identified to species, we're interpreting this as though the previous identification was 'wrong' by being too precise. Here's an example:

But since the expert IDs are made blind, the experts can't see how precisely observations are already ID'd. In some case, the expert's intent with a coarser ID was not to explicitly disagree with the more precise ID. For example, here's an observation was ID'd to the subspecies level and the expert added an ID at the species level. This species level ID was not intended to explicitly disagree with the subspecific ID, but thats how the system is currently counting it (ie the community's ID was 'too precise').

We're working on functionality to make it possible to distinguish explicit disagreements (e.g. 'no one can ID this to species from the evidence provided') from the alternative (e.g. 'I can only ID this to genus, but someone else might be able to ID it to species'). But for now, if you think someone else might be able to provide a more precise identification than you could, skip the observation.

Thanks again for your help. If you skip observations as described above (those with ambiguous subjects or when others might be able to make a more precise ID than yourself), then please proceed with making IDs for the experiment. I'll check back in when we have improvements to deal with these situations.

I'll also check in with more updates on the analysis as we have more data. Please contact me if you have any questions, concerns.

Thanks!

Scott

@charlie @fabiocianferoni @vermfly @jennformatics @d_kluza @arachnojoe @cesarpollo5 @ryanandrews @aztekium @lexgarcia1 @harison @juancarlosgarciamorales1 @garyallport @echinoblog @jrwatkin68 @bryanpwallace @wolfgang_wuster @bobdrewes

Posted on April 05, 2017 07:19 PM by loarie loarie | 19 comments | Leave a comment

December 23, 2016

The dangers of publishing partial taxonomic revisions...

iNat relies on the IUCN RedList as the taxonomic authority for mammals. The downside of taxonomic authorities are that they lag a bit from the primary literature - so maybe we're not quite as up-to-the-minute-informed as we could be about where species boundaries are and what our set of species should be as we could be. But the upside is we have a clear sense of the taxonomic concepts we should be following when we identify species (e.g. lets all agree to call fuzzy brown things from this are this name and fuzzy gray things from over here this other name). All being on the same page when it comes to taxonomic concepts is critical to organize conservation efforts, or inventory efforts like iNaturalist. And I'd argue that its more important that we are all on the same page regarding taxonomy even if our concepts don't represent up-to-the-minute bleeding edge taxonomy in the primary literature.

With that, IUCN during the 2004 Global Mammal Assessment used the current taxonomy that considered the 'Desert Woodrat group' as having five species: Neotoma lepida on the mainland, Neotoma bryanti known only from Cedros Island off Baja CA, and 3 other now extinct Baja Island endemics:

Figure 1.

Enter Patton et al.'s excellent revision in 2014. This study determined that the mainland woodrats, Neotoma lepida (sensu lato), actually should be split into 3 groups: a coastal species, an Arizona species, and the remaining species in between. Re: the former Baja island species, Patton et al. determined that they are all part of the coastal species except rats on Isla Ángel de la Guarda which are their own species. Patton re-used the name N. bryanti (senus lato) for this coastal species, and named the Arizona species N. devia, N. insularis for the Isla Ángel de la Guarda and reused Neotoma lepida (sensu stricto) for the remaining rats:

Figure 2.

What bothers me is that, in my opinion, IUCN should have held updating the RedList until they were ready to revise this entire group. Specifically by updating the existing assessments of N. bryanti and N. lepida, removing N. bunkeri, N. anthonyi, & N. martinensis from the list and adding N insularis and N devia. But what IUCN published in 2016 only included the addition of N. devia without addressing these other species (e.g. N. devia now overlaps with the range of N. lepida):

Figure 3.

Now the RedList gives the impression that N. lepida and N. devia co-occur in Arizona. Perhaps to address this, they yanked the map of N. lepida (sensu lato) but this just adds a lot of confusion making it impossible to understand what they mean by the names N. lepida with regard to the other taxa in the group.

If anyone from the RedList is listening, in my opinion its critical not to only partially incorporate taxonomic revisions into the published RedList. The N. lepida group should have been tackled in its entirety or not at all. In the meantime this puts us here at iNat in an awkward situation. Should we:

1. leave out species coming from our taxonomic authority (specifically N. devia rolling things back to Fig 1.)
2. or should we imply that they meant to incorporate all the revisions from Patton et al. and move things forward to Fig. 2 ahead of the published RedList?
3. or should we follow the Published RedList exactly (Fig. 3) even though it leaves us in an ambiguous state that partially follows Patton et al. in an unclear way?

Posted on December 23, 2016 05:38 PM by loarie loarie | 9 comments | Leave a comment

July 15, 2016

Walking around sapsucker woods (Trip)

walked around sapsucker woods

Posted on July 15, 2016 09:27 AM by loarie loarie | 3 observations | 0 comments | Leave a comment

March 06, 2016

Sister species everywhere!

Today, during the break in the rain, I got my wife to drop me and my daughter off at the back-side of Muir woods with a pick up scheduled at the main entrance while she ran some errands. My main target was Brackenridgia heroldi, my boogie woodlouse. Good news is I found it, bad news is... well I don't want to talk about it.. Was beautiful and moist with wonderful waterfalls and lots of Trichodezia californiata were flying everywhere.
Untitled

A funny thing about the hike was I saw a lot of pairs of plant species (e.g. two different species in the same genus) which was kind of fun. For example, I found the rarer largeflower fairybells (Prosartes smithii) touching the more common drops-of-gold (Prosartes hookeri)

largeflower fairybells (Prosartes smithii)

drops-of-gold (Prosartes hookeri) with the exerted flower parts

I also saw a Buckbrush (Ceanothus cuneatus) touching blueblossom (Ceanothus thyrsiflorus).

Not touching one another, but I did see a patch of California barberry (Berberis pinnata) at the beginning of the hike up in the mountains and a
Cascade Oregon-grape (Berberis nervosa) near the end deep in the moist redwoods.

Likewise, I saw blooming False Solomon's Seal (Maianthemum racemosum) near the beginning of the hike and Star-flowered Lily-of-the-valley (Maianthemum stellatum) near the end..

And lastly, Pacific trillium (Trillium ovatum) with their stalked flowers were everywhere, but at the end of the hike where it felt alot more coastal I saw one Giant Wakerobin (Trillium chloropetalum) with its sessile flowers.

Pacific trillium (Trillium ovatum) flowers are on a stalk

Giant Wakerobin (Trillium chloropetalum) with sessile flowers

Posted on March 06, 2016 10:50 PM by loarie loarie | 72 observations | 0 comments | Leave a comment

February 29, 2016

Search for the Snow Queen

I just became aware of a plant I'd never heard of before called the Snow Queen from one of @dgreenberger's observations. I thought I'd at least heard of most plants in Marin County if not seen them so I decided to try to track these down today. I followed Doreen Smith's advice on the Marin Native Plants FB page to head to Cataract Falls. Super beautiful, also super crowded with hikers which was kind of surprising given how far it is. Turns out Snow Queen is ridiculously common there. I saw it growing along the road just as I was parking before I even hit the trail and its definitely frequent along the Cataract Falls trail. Along with the Milkmaids, Redwood Sorrel, Anemone's (are we calling these Gray's or Oregons now?), and Pacific Trilliums, plenty of white wildflowers all along the trail.

I turned off the Cataract Trail onto the Helen Markt Trail to get away from all the waterfall traffic and was glad I did because I found the first California Pill Millipede on iNat (actually kind of disappointed though because at first I thought it was the woodlouse Venezillo microphthalmus but its actually a millipede that rolls up in a ball - how crazy is that?). On my way back to the car along the road I found a nice big Chanterelle that I was surprise hadn't been eaten.

Then on the way home I stopped by Azalea Hill to see if anything interesting was blooming on the serpentine. There was a beautiful bloom of Serpentine Spring Beauty and the worlds smallest rattlesnake. I was kind of hoping to see Brownies, which I've seen before but realized I've never iNatted. So I cheated and stopped one more time on the way back at the MMWD Ranger Station to tick them from a population Terry G. had posted before.

Posted on February 29, 2016 12:32 AM by loarie loarie | 61 observations | 3 comments | Leave a comment

February 24, 2016

Steelhead watching

High risk high reward chance of seeing a spawning steelhead in the Russian River paid off today!
GOPR1767
My basic approach is to put on a wetsuit and mask and snorkel, run 1/2 mile up the road paralleling Maacama Creek and float back down to where it meets the Russian River looking for Steelhead. I've done this many times over the years, but timing is critical, its always around this time of year (late Feb/early March) but difficult to tell exactly when there will be fish in the Creek/River. This is the first time I've done this with a camera (gopro). And I'm so stoked I was able to take a picture of one of these mysterious visitors of the ocean in time for Fish Week!

Posted on February 24, 2016 05:45 PM by loarie loarie | 1 observations | 11 comments | Leave a comment

February 07, 2016

Minus tide at Duxbury Reef

Some of my friends wanted to go tidepooling, so we had this nice minus tide on the calendar. Lots of kids in tow so hard to cover too much ground or do too much extreme tidepooling, but saw some cool stuff.

This was only my second trip to Duxbury, and the last time I went I didn't go over 'the ridge' to the south so I was determined to get there this time. It was definitely worth it. While there were tons of people near the entrance, almost no one was over the ridge. I was greeted by a nice Great Blue Heron for Heron Week foraging among the tides.
IMG_1464
It was weird because there was so much freshwater runoff from all the rain we're getting that the pools were a bit fuller than usual despite the low tide, and annoyingly cloudy. My main mission was to find an octopus for the kids. I succeeded in catching the second one we saw. There was a crazy abundance of Hopkin's Roses and we found a few Hilton's Aeolid. Other than that nothing to out of the ordinary, but it was such a beautiful clear day and was so much fun to splash around in the pools with my one year old, that this was definitely one of my favorite tidepool outings ever.

Posted on February 07, 2016 12:52 AM by loarie loarie | 18 observations | 1 comments | Leave a comment

Early Spring at Mt. Burdell

Went on a late morning hike to Mt. Burdell mainly because I was reminded by Doreen Smith's upcoming Marin CNPS trip that this was a good spot to hit in Early Spring.

My plan was to head straight to the wildflowery spot where I'd been a few times before but I somehow ended up at a different trailhead and got a bit turned around. I ended up first heading up the hill towards Mt. Burdell proper and snooped around in a tiny creek that was full of newts. I also saw some cool fruiting slime molds and a few interesting crickets. Was cool to see Margined Whites flying among the Milkmaids

Next I headed over to the wildflowery area. The Blennosperma were blooming in full force and the Fremont's star lillies were just starting but still a good display. I only saw one blue dick blooming. I didn't make it over to the place where I know Fragrant Fritillary blooms about now. Lots of Cows and horses and lots of Yellow dung flies on the cow paddy's they leave behind. I still think these are beautiful flies considering their name and habit...

Posted on February 07, 2016 12:38 AM by loarie loarie | 24 observations | 0 comments | Leave a comment

January 23, 2016

#wetnaturalist

Went out looking for hawks and newt, found hawks but no newts. Was nice to be reminded that the Bay Area is indeed in the Pacific Northwest (and not Nevada) with all this rain. Fetid Adder's Tongue are up and in Bloom!

Was pouring rain pretty much the whole time but luckily I brought an umbrella.

These steams were bone dry most of the last year - awesome to see them looking so lush

rain stopped for a bit an unsaddled the girl to catch a Ligidium

Posted on January 23, 2016 07:30 PM by loarie loarie | 19 observations | 7 comments | Leave a comment
Member of the iNaturalist Network   |   Powered by iNaturalist open source software