Upcoming Changes to Lists

The the new life list tool we released today is completely separate from the existing list functionality on iNaturalist and the original entity known as a life list (from here on “original life list” to avoid confusion). Here, we describe how these existing lists work and some changes we’re planning to improve scalability as iNaturalist grows.

Part of the reason why the existing list functionality on iNaturalist is so complicated is that lists are used for many different things. For example, there are normal user lists, original life lists, traditional project lists, default place check lists, and other place check lists. We’ll describe these different kinds of lists in turn and how they act differently. But let’s start with the most straightforward and simplest kind: the normal user list.

Normal User Lists
A great example of a normal user list is the favorite taxa list you can display on your profile. To do this, you create a list called “Favorites” and add some taxa to it.

The taxa added to a list are stored in records called “listed taxa.” Each listed taxon has a description and comments and keeps track of “observation stats” which include the number of times you’ve observed the species and links to the first and last observation. The latter allows you to do things like filter the listed taxa that haven’t been observed.

Normal user lists are in many ways analogous to Guides on iNaturalist. They serve as ways for a user to create and maintain a list of taxa that (aside from the observation stats) don’t interact with observations.

Original Life Lists

When you create a normal user list, you have the option to “Make it a life list.” Checking this box means that the list will create listed taxa for any observation you’ve made. If you restrict to a higher-level taxon and/or a place (e.g. “Brazil Frogs”), it will only create listed taxa from observations of that taxon in that place.

You can still manually create and destroy listed taxa on original life lists independently of listed taxa being automatically created from your observations. And issues with original life lists being out of sync with observations has been a persistent point of confusion. Let's call this automatic listed taxa creation from observations functionality “auto-listing functionality” for short.

Traditional Project Lists

Traditional projects have a “must be on list” rule which is really just a shortcut for many “must be in taxon” rules. In fact, in the newer collection projects, we no longer bother with these lists and just allow users to add many “include taxa” project rules. Nonetheless, we still support traditional projects and thus still support traditional project lists to facilitate this ‘must be on list’ rule. They behave exactly like normal user lists (i.e. no auto-listing functionality) except that the observation stats are filled from observations in the project rather than observations made by the user.

Default Place Check Lists

When you create a place, you have the option to mark “check lists allowed”. Doing so will create a default check list for the place. Place check lists behave very much like original life lists in that they include auto-listing functionality from research grade observations made within the place. Likewise, people can manually create and destroy listed taxa independent of the observations made in the place. The observation stats are filled by observations made within the place.

Listed taxa on place check lists also store “establishment means” (e.g. native/introduced) for species in that place which is used throughout the site. Likewise, the listed taxa on standard place default checklists also determine the "presence places" in an atlas. The existence of an atlas for a species disables auto-listing functionality for that species on default checklists.

Other Place Check lists

In addition to the default place check list, places can have other check lists. They can be restricted to a higher-level taxon and they can be marked as “comprehensive” from the list edit page to indicate that all species in that higher-level taxon for the place are included in the list. Like the default place checklist, they also have auto-listing functionality.

Problems with Lists and Proposed changes

The two major problems with lists are that functionality to track observation stats and auto-listing functionality aren’t scaling well, meaning that as iNaturalist continues to grow the server requests this functionality generates are bogging down the performance of the site.

We’ve reviewed lists on iNaturalist and have determined that they basically serve two separate use cases:

1. The ability to view a set of observations in species list form

2. The ability to maintain a reference list of species (independent from observations) to add context to and compare with observations (e.g. establishment means, atlases, etc.)

We think we can better serve use case 1 with new dynamic tools viewing observations in list form like the new Life List tool we’re unveiling today. We plan to build an analogous new Place Check List tool to view the species observed in a place in list form. This will allow us to remove the problematic auto-listing functionality from all lists so that they can be more focused on serving use case 2: acting as an independent reference list to add context to observations.

We also plan to remove observation stats from listed taxa. We realize that this will remove functionality to compare what species on a list have been observed and which haven’t been observed - this is, for example, what controls the color (green means observed and yellow means unobserved) of the check list places on taxon maps. But, if there is demand for this kind of functionality to compare a list with a set of observations, we think we can build more scalable functionality to do so rather than the existing observation stats.

To be clear, we're not planning to remove any existing listed taxa. We are only planning to disable the auto-listing functionality and the functionality that maintains observations stats on listed taxa.

Here’s our proposed roll out of these changes:

Phase 1 (today):
Launch new dynamic life list tool

Phase 2 (next few weeks):
Remove original life lists (they will become normal user lists)
Remove observation stats from normal user lists and traditional project lists

Phase 3 (sometime in the next few months):
Launch new dynamic place check list tool

Phase 4 (sometime in the next few months):
Remove auto-listing functionality and observation stats from place check lists
Remove other place check lists (they will become normal user lists, leaving only a single optional default place check list for each place)

Posted on October 27, 2020 09:17 PM by loarie loarie

Comments

Thanks Scott. Can you explain more about "We also plan to remove observation stats from listed taxa" or provide examples of what it looks like now to have observation stats from listed taxa, and what will look different after the change?

Posted by muir over 3 years ago

Hi muir, in the second image above the orange arrow is pointing to the display of 'observation stats' on a listed taxon page. e.g. for American Pika on the California Checklist the listed taxon would be https://www.inaturalist.org/listed_taxa/5443 and the observation stats store the number of RG obs (441), the First iNat observation ("August 15, 2007 in Evolution Lake, Kings Canyon National Park, CA by gdurkee") and the Last observation ("October 17, 2020 in Twin Bridges, CA 95735, USA by paul31").

As described in the post, the scale of processing to keep these observation stats updated is a growing problem, but they support functionality to:
1) search for listed taxa on a list that haven't been observed, e.g. Mammals on the California Checklist that haven't been observed https://www.inaturalist.org/check_lists/312-California-Check-List?view=photo&taxon=40151&observed=f
2) color the listed taxa that display on taxon maps green (RG obs exists) or orange (no RG obs exists)

So if we removed observation stats they would no longer be displayed on listed taxa, we'd remove the 'observed=yes,no' filter from list searches, and the listed taxa would all display one color on taxon maps (e.g. just green rather than green and orange)

Posted by loarie over 3 years ago

Got it (I think). I like the orange-shaded places on the map, but it won't be a big loss in my opinion.

Posted by muir over 3 years ago

@loarie Considering how often (or not) taxon maps like the above are accessed, I wonder if it would be an unacceptable performance hit to simply have taxon maps start accessing two place checklists (the new dynamic and old original ones) to determine map coloration, instead of the one being accessed now, for each map place? Color everything in the original checklists orange, then change places to green that are in the new dynamic checklist? (Or something equivalent but more computationally efficient). Or put another way, would doubling the "draw" time of a map be unacceptable?

Posted by jdmore over 3 years ago

Traditional Project Lists :

This is a much cooler way of adding a list than the current collections project where one has to add (or remove) species one by one. For long lists and many lists (e.g. https://www.inaturalist.org/projects/nemba-alien-species-south-africa and https://www.inaturalist.org/projects/south-african-red-list-plants-and-animals) managing them is a pain.
Every time the Red List is updated or the legislation on alien plants is promulgated (once or twice annually) the long lists have to be edited one by one and they are in a meaningless order wasting a lot of time searching for them. Adding the functionality of Traditional Project Lists to current Collections Projects would be a great help. The alternative for Collections projects is a batch add and batch remove functionality, but why not just use the Traditional Project Lists for this?

Traditional Project Lists surely do not need auto-listing functionality - they are just lists for use as filters in projects (traditional and ?why not? collections).
Can they not serve the same role for places?
And could they not also be an option within Filters in ID and Explore tools?

Posted by tonyrebelo over 3 years ago

How does the "plan to remove observation stats from listed taxa" affect the Taxon (Species) Page.
e.g. https://www.inaturalist.org/taxa/132848-Protea-cynaroides
It has a "Last Observation" box, which is most useful.

I am going to miss the Last Observed entry in the Place Lists: .e.g.
https://www.inaturalist.org/check_lists/63740-City-of-Cape-Town-Check-List
and especially the missing taxa (incl. subspecies)
https://www.inaturalist.org/check_lists/63740-City-of-Cape-Town-Check-List?q=&view=photo&observed=f&commit=Filter

We used that extensively for planning our City Nature Challenge priorities.

But so long as there is an equivalent to the Unobserved Species in the Life List - for the Place List that uses the various checklists as the filter (not the global All-Life filter) - then that will be fine.

Posted by tonyrebelo over 3 years ago
Posted by tonyrebelo over 3 years ago

@tonyrebelo Thanks for pointing that out. I have moved it.

Posted by andrewgillespie over 3 years ago

Accessing Lists.

Can I please request that ALL Lists are accessioned in the Lists menu (https://www.inaturalist.org/lists/). Currently, it unnecessarily involved (impossible) to find to which reserves (places) one has added a “tree” or “aliens” or "grass" list to.
This should include not only:

Life Lists
Normal User Lists (== user lists)

But also:

Traditional Project Lists (== filter lists),
Other Place Check Lists (== checklists)

And what about a tab for "Lists" in the "Search" results

Posted by tonyrebelo over 3 years ago

Hi all - just a note that we finally deployed Phase 2 today. As a reminder that's
Remove original life lists (they will become normal user lists)
e.g. n https://www.inaturalist.org/lists/loarie my original life list remains but is now just a normal user list (we might rename them all down the road to something like 'loarie's static life list' reduce confusion or do a campaign to encourage people to delete them if they've never been tailored)
Remove observation stats from normal user lists and traditional project lists
this means on all lists except checklists, listed taxa are no longer caching/updating information about the first and last observation etc. This means you can no longer do things like filter listed taxa by observed? yes:no - e.g https://www.inaturalist.org/lists/10038-My-Nudibranchs-menagerie
We're excited to no longer be processing all of this in time for the upcoming northern hemisphere spring bump as it should help with scaling alot. We'll update on phase 3 when we've made some progress

Posted by loarie about 3 years ago

Hi @loarie,
Thanks for the update on the lists.
I'm a bit confused by the recent changes. Previously, it would allow me to have a list of my observed taxa, with an illustration corresponding to the latest picture I uploaded for this taxon. I found it cool to have a "photo album" of my observations, arranged by taxa.
Now it is just a list of the taxa I saw, with a placeholder picture that sometimes seems a bit random (e.g. a picture of a hatchling for some birds, instead of the adult). Is there a way to revert back to the previous version?
Alternatively, I quite like the new Dynamic Life List, with its different filters. However, some things seem to be sorely missing. For example, why is there no option to organise my observations in the right-hand panel by taxonomy, instead of more recent/older observation? This is possible for the species panel.
Many thanks!

Posted by donalddavesne about 3 years ago

The list has changed just now and I have spent hours aligning my own images to my observations which now appear to have been lost. How do I get them back? I am a creature of habit and am not computer savvy. This is a nightmare!

Posted by philipmarkosso about 3 years ago

Hi @donalddavesne - thats correct that we've lost the functionality on traditional lists to have your listings display a photo from one of your observations as opposed to the default taxon photo. Now traditional lists can only show the default taxon photo (since we removed the functionality that was maintaining the most recent observations associated with each listing).

Also when we built the new dynamic life list (linked to from here), as you mention, we didn't include that functionality so even if you set the right panel to the 'species' tab and add the 'sort: taxonomic' setting, the list of species you've seen arranged by taxa that you see shows the default taxon photos (as opposed to photos from your observations). But those steps should allow you to organize the species you've seen by taxonomy.

If you select the 'observations' tab (as opposed to the 'species' tab) on the right side you will see your observations (rather than the species they represent) but there's no way to sort these taxonomically.

Hi @philipmarkosso - as mentioned above, there used to be an option on traditional lists to have your 'listings' show the most recent photo from one of your observations as opposed to the default taxon photo. This change removed that functionality which essentially had listings listening for and storing of the most recent observation which unfortunately wasn't scaling well. Unfortunately that meant we could no longer support the functionality that had a listing show the photo from this most recent observation if one existed rather than the default taxon photo.

Apologies for those of you who miss this feature on traditional lists to show a photo from your most recent observation of that taxon rather than the default taxon photo for that taxon along side the listing. Traditional lists were not scaling well in this 'lifelist' role where listings were constantly checking/updating in response to observations - and we had to switch to a different system (the new dynamic life lists) to offer 'lifelist' functionality at the current scale iNaturalist is operating at. Traditional lists (except checklists which we'll address in Phase iII) are now not checking/updating in response to observations which meant we had to stop supporting the 'show the photo from your most recent observation' functionality. Its possible we could try to build 'show the photo from your most recent observation' functionality into the new dynamic life lists (species tab of right panel) but it would incur a pretty big performance hit

Posted by loarie about 3 years ago

Hi @loarie thanks a lot for your detailed answer!
I understand that replacing the default taxon images with our own pictures would end up being quite unscalable.
In my mind (but I have no idea if it's doable), a nice solution would be to have, in the Dynamic Life List, the possibility to sort by Taxonomy in the "observations" tab. Since it's already doable with the "species", it sounds like something achievable to me, but obviously I don't know how limited you are by the sofware. I don't think including our pictures in the "species" tab would make much sense: I think it's nice as is, e.g. a list of the species we saw (illustrated by a nice, curated picture), rather than our "raw" observations.

Posted by donalddavesne about 3 years ago

Hi @loarie, is there still a way to view which species in a list you have observed? The lists are still showing how many I have observed, but there is no way that I can see to tell which species. If not, is this a feature that you guys are considering adding back in? My lists are mostly just groups of taxa that I hope to see, and I always enjoy 'ticking off' a new species. Perhaps it could be added into the dynamic life list (I'm thinking perhaps something similar to the current 'compare lists' function?).

Posted by matthew_connors about 3 years ago

Thanks @bouteloua, that's exactly what I was after!

Posted by matthew_connors about 3 years ago

@loarie @tiwane I followed a link from this feature request about adding what the map colors mean to the legend:
https://forum.inaturalist.org/t/add-map-legend-for-all-overlays-on-taxon-pages/7584

It's about a year later and the lack of a legend for the orange, green, and pink like on the map above is still a cause of confusion. I updated the range maps for Malacothamnus and, even though I explained what the colors meant in my post about it, I've still had people confused about what they mean. Are those going to go away soon? This post seems to maybe indicate that. If not, seems like it would be good to have everything displayed on the map included in the legend.

Posted by keirmorse over 2 years ago

are you referring to Phase 4 described in this post? To date we've completed Phase 1 and Phase 2 but haven't even started on Phase 3. That might give you a better sense for timelines

Posted by loarie over 2 years ago

Sounds like Phase 4, I think. Would it maybe make sense to include these layers in a legend until Phase 4 can happen then? Should I redo the feature request? Seems like everything on a map should be included in the legend.

Posted by keirmorse over 2 years ago

Yes if you're requesting something that doesn't need to be entangled with replacing place checklists than please make a feature request on the forum.

I see it does say 'few months' next to Phase 3, that might have been a poor choice of words - 'several months' or 'few years' might have been better. We've had to scale back on new web features a bit to focus on scaling issues. Also, one obstacle to implementing the dynamic place checklists is that we currently can't performatively compute the dynamic list from millions of observations - that isn't a problem for peoples life lists (which involve less than 100s of thousands of obs) but would for large places like the CA when millions of observations are involved.

Posted by loarie over 2 years ago

Done.

Posted by keirmorse over 2 years ago

Some of the bee and wasp identifiers are currently trying to add species checklists for Places. We tried this using two methods: by creating separate lists also in Places (e.g. "Bees of Guam"), or instead by simply adding species to the Place default list (Guam Check List), which is currently our preferred method (more pros than cons). We've run into technical issues/bugs for both methods, if anyone could address, thanks.

Bug report: https://forum.inaturalist.org/t/creating-multiple-checklists-causes-duplicate-identify-suggestions/28011.
Project organization page and discussion: https://forum.inaturalist.org/t/bee-and-wasp-checklists-wiki/27909

Posted by bdagley over 2 years ago

At the moment, checklists in Identify seem very buggy - sometimes the same species will be listed 2 or 3 times. There's also a lot of county checklists marked "comprehensive" that are nowhere near comprehensive, or include species that aren't actually listed for the region, or have incorrect establishment means (native plants listed as invasives, and vice versa). These things don't appear to be editable, at least by non-curators.

Will upcoming work address these issues, or should I be creating bug reports / flags about them?

Posted by graysquirrel almost 2 years ago

Our plan is still to make these changes to checklists but we need to increase our engineering capacity first so these are delayed. Apologies for the expectations we set above

Posted by loarie almost 2 years ago
I see it does say 'few months' next to Phase 3, that might have been a poor choice of words - 'several months' or 'few years' might have been better. We've had to scale back on new web features a bit to focus on scaling issues

Any sign of Phase 3 on the horizon? I have some nature reserve checklists that I promised conservation agencies, but I cannot deliver checklists.
Am having to download all data, and then compile these myself (quite easy in excel with pivot tables): but it does mean downloading lots of data, rather than just a species list as required.

E.g. https://www.inaturalist.org/places/bracken-ridge-ecoestate has 10 species, but https://www.inaturalist.org/observations?place_id=192890 has 96 species.

Is there a way in the interim to manually refresh a checklist? Something like: https://www.inaturalist.org/listed_taxa/refresh_observationcounts?listed_taxon_id=106907127 but for a place rather than a species?

Posted by tonyrebelo 8 months ago

I have a few notes and tips for how checklists currently function, using the example of bees and wasps. Bees and wasps are cryptically similar groups, and currently still have certain valid subspecies, most of which correspond to different color forms, such as having redder color forms in Florida compared to northeast US. Although, when identifiers use Compare or Suggestions the default search filter is Observations, which doesn't show the subspecies. Selecting Checklist will show them, which is helpful. Although, as mentioned, when users create new checklists vs. adding taxa to existing place checklists each species will be shown two or more times in Compare/Suggestions. And, when the location filter is turned off to correspond to worldwide, numerous copies of the same taxa are shown which is essentially unusable.

When I added checklists in the past, I did find that using the method of adding taxa to existing lists was ideal, preventing the duplicates from showing. Furthermore, a filtered search within the checklist page can be used so that the user who created the checklist can save a link for only the taxa they added, like bees. One minor downside is that the other approach of creating new lists has a box to add a citation, although many published checklists are compilations of various sources anyway.

When identifying I often use Compare/Suggestions set to the checklist filter, although sometimes switch between it and Observations. In doing so, I've noticed that Observations often displays more taxa vs. the Checklist filter. I've assumed this may be due to Observations showing taxa identified in the location before there are RG observations of them, yet have wondered what the threshold is for taxa to appear in the Checklist filter. For example, one RG observation or multiple? One other related tip is to go to a species page, select see all photos, and then add a place like Florida, and then you'll see the variation/color forms of the species in that way. In conclusion, I continue to recommend the approach of adding taxa to existing place checklists, because otherwise the duplicates limit the usage of the checklist filter for identification, although updating/fixing checklists in the future would also be ideal.

Posted by bdagley 8 months ago

We haven't started work on Phase 3. The major sticking point is that our Elastic Search infrastructure for filtering and grouping obs that is behind the dynamic lifelist doesn't scale to millions of observations. Currently our top observer has 239,982 obs and 7,794 species which the dynamic lifelist can handle. But we even have counties with many more obs than that. There are almost 10M US birds. We decided to first invest sometime on the Elastic Search infrastructure behind this. But probably regardless of gains there we'll probably have to make some tough choices on constraints for places and their corresponding dynamic lists

Posted by loarie 8 months ago

Are you not designing a Lamborghini where a Chevrolet will suffice? It is great to have one, but can we afford to use it?
The Elastic Search works great on personal life lists, but it is not just premature for place lists?
The current filters generate species lists for places already, even at continental scale. Perhaps the tough choice should be to go for a simpler system for the time being?

Posted by tonyrebelo 8 months ago

the explore species tab only shows the top 500 most observose species sorted by observations. dynamic life lists show all species and ancestors and enable sorting by taxonomy. Thats the place checklist feature we don't want to loose without a replacement.

Posted by loarie 8 months ago

Agreed!!

But the explore selects all 3.5M southern African observations in under 10 seconds, and counts observations of the 40,549 species (even if it displays only 500). That counted list, then linked to the taxonomy, should not break the bank?
The current checklist loads 30,703 of 47,106 species observed in 3 seconds in taxonomic order. That is a fraction of the time it takes Loarie's Species List (7,710 species) to load.
[We dont need unobserved species, unless compared to a master place list for that place (and node counts are a luxury too)]
Is there not another way of maintaining place checklists without incurring the heavy indexing load?

Posted by tonyrebelo 8 months ago

Add a Comment

Sign In or Sign Up to add comments