Real-time Computer Vision predictions in Seek by iNaturalist version 2.0

On April 5, in conjunction with the release of the Our Planet series by Netflix, iNaturalist released a new version of Seek by iNaturalist. To heed David Attenborough’s call to action in Our Planet to protect biodiversity, we need to understand what’s here and what we might lose. Seek by iNaturalist unlocks curiosity about the natural world by giving speedy identifications with computer vision and challenging you to earn badges for finding species new-to-you.

How it works

When you open the Seek camera and point it at a living thing, the app immediately tells you what you’re looking at, even before you take a picture. This on-screen identification is tied to the tree of life, and guides you towards taking a more identifiable photo by getting more specific as you fill the frame and get the right angle or features. When the app narrows it down to species, it prompts you to take a picture, which earns you a badge and unlocks more information about the species. This “augmented reality” view of the world makes it easy to explore and interpret the natural world around you all while guiding you to take more identifiable photos.  Seek can’t always identify things to species (it’s still learning...), but it aims to provide the most precise correct name it can. Here’s some footage of it in action!

In Seek v1, which launched in March 2018 with support from HHMI Tangled Bank Studios, you needed to take a photo before you had a sense of whether or not it was even identifiable by Seek. This caused frustration when people repeatedly experienced the lack of a computer vision match. Now, since you can see predictions overlayed on the camera you get immediate feedback about what you see and the specificity with which Seek can identify it.

The species included in Seek are based entirely on photos and identifications made by the global iNaturalist community, so the Seek camera will work best in places where there is already an active community of iNaturalist users, and for species that are easily identified from photos. Seek also uses data submitted to iNaturalist to show suggestions for “species nearby,” but unlike iNaturalist, findings made with Seek will not be shared publicly, making it safe for children to use.

Seek is geared to encourage outdoor exploration of wild biodiversity (rather than pets, zoo animals, or garden plants). We hope kids, families, educators, and anyone into games will start exploring their natural surroundings with Seek, and we want this to inspire the next generation of biodiversity stewards by encouraging exploration and unlocking the names of species as a way to learn more about them. We want to make it easier for curious people who may not consider themselves naturalists to learn more about nature.

Innovations in computer vision came from research collaborations with Grant Van Horn, an Adjunct Scientist with iNaturalist. Van Horn’s dissertation research at CalTech, advised by Pietro Perona, made it possible to produce a refined dataset of iNaturalist observations, train the classification model, and export it for efficient inference on mobile devices. The computer vision model includes 15,798 species and 12,524 broader taxonomic groups (such as kingdom, phylum, class, order, family, and genus). The accuracy of predictions and breadth of species included will continue to improve as the iNaturalist community and dataset grows.

How to download it

Seek is freely available on both iOS and Android. Unlike the iNaturalist mobile apps which have separate code bases for iOS and Android, we used React Native for Seek, which is a relatively new technology that allowed us to build Seek for both Android and iOS with a single codebase. It’s still nascent — there’s not even a stable 1.0.0. release yet — but we liked the benefits:

  • Saves us time in development, since we’re only using one coding language.
  • Brings Seek to a wider, more varied audience, since we’re able to support 7,000+ devices on Android alone — this also allows for a more international audience, since Android tends to be more popular anywhere outside of North America, Australia, and western parts of Europe.

Seek is currently translated into 7 languages: English, Spanish, Hindi, Chinese, Portuguese, German, and French.

This major update to Seek was made possible with support from WWF and the Our Planet series on Netflix. Seek is created by iNaturalist, which is a joint initiative of the California Academy of Sciences and the National Geographic Society.

Posted by tiwane tiwane, April 05, 2019 07:03

Comments

Thumb

Great!

Posted by carlacorazza about 1 year ago (Flag)
Thumb

Great! But I would prefer to see suggestions in English or scientific names, and not in Spanish (local names are very different in each country or even region). I don't find a configuration button for changing language.

Posted by luchogu about 1 year ago (Flag)
Thumb

Bravo! I've only been aware of/using iNat for a short time, but even in that short time, I've been shocked/impressed at how many changes your team has moved forward with, as well as involvement/responsiveness of folks in the forums, etc.

Posted by colincroft about 1 year ago (Flag)
Thumb

This sounds really great for identifications. Excited to give it a go!

Posted by magichin about 1 year ago (Flag)
Thumb

Fantastic! What a great app for children of school age to use and learn the biodiversity of our planet. This may also prove to be a boost for iNat users to get preliminary I.D.'s or confirm what they have seen but not uploaded.

Posted by lonnyholmes about 1 year ago (Flag)
Thumb

Can I link my Seek account to my iNat account so that all of my identified species are on one database?

Posted by johngcramer about 1 year ago (Flag)
Thumb

Very good point on linking the Seek and iNat accounts. Can it be done?

Posted by lonnyholmes about 1 year ago (Flag)
Thumb

@johngcramer and @lonnyholmes Linking the two apps completely cannot be done, however we are looking into the possibility of allowing Seek users to contribute some observations to iNaturalist, but there are some hurdles. Keep in mind that one of Seek's main reasons for being is that iNaturalist cannot be used by children under 13 due to privacy issues (every observation is a public record of where and when the observer happened to be) so it's been designed for a different more casual audience.

I'm curious, do dedicated iNat users see themselves using Seek to record observations? I think they serve fairly different use cases and audiences but I could be way off here.

Posted by tiwane about 1 year ago (Flag)
Thumb

@luchogu thanks for the feedback, I'll pass it along.

Posted by tiwane about 1 year ago (Flag)
Thumb

No from my point. I will continue to use iNat and probably not spend much time with Seek. Its only advantage to me is the possibility of getting an I.D. without actually taking and uploading the observation. Agreed, useful for two entirely different user groups. Thank you for the quick response.

Posted by lonnyholmes about 1 year ago (Flag)
Thumb

I prefer iNat, the risk I see in Seek is that, once received the answer about what they have seen, people pass away without posting on iNat.

Posted by carlacorazza about 1 year ago (Flag)
Thumb

@tiwane as an amateur nature enthusiast and photog, Seek is a good way for initial id or at least then when I post to iNat I can direct to specific group. So far my posts have received good responses on possible ids , but there are unknown plants and blossoms I'd like to find out more,

Posted by magichin about 1 year ago (Flag)
Thumb

As an iNaturalist noob and nature ID lightweight, I would prefer to use Seek to give me a clue what I'm looking at before I take the photo, then publish it to the public observations on iNaturalist. As a professional programmer, I can think of at least 5 different ways you could protect children's location privacy and still allow this.

1) Seek user logs into Seek with an iNaturalist account => photo locations = public. Otherwise location is obscured.
2) Seek user identifies them self as child or adult (with explanation that adults' photo locations will become public). If they identify as child, then location is Obscured in iNaturalist.
3) Seek user self-selects to allow photo submitted to iNaturalist.
4) Each time photo is IDed, Seek asks if they want to submit image with location to iNaturalist.
5) Seek could show red border around screen if photo location would be public as a warning
etc.

Posted by scoutinvermont about 1 year ago (Flag)
Thumb

When I tried to download it I got "Your device is not compatible with this version" Samsung S4.

Posted by memcinto about 1 year ago (Flag)
Thumb

Great for plants, but the wild birds generally don't let me get close enough for the app to recognize them - a zoom similar to the phone feature would be helpful.

Posted by aspiedextra about 1 year ago (Flag)
Thumb

I might consider using it as a way to reach friends more easily. iNaturalist is complex enough to use that I think it would be a lot easier to get people into exploring nature with Seek, and using it myself would help me introduce them to it. If I did use it, I would appreciate being able to have the option of logging in and adding the observations to iNaturalist to save time (not having to open and operate both apps at once).

Posted by upupa-epops about 1 year ago (Flag)
Thumb

Seek struggles to identify trees just from their bark (but then so do I in winter!).

Posted by scoutinvermont about 1 year ago (Flag)
Thumb

@scoutinvermont my understanding is that the computer vision only uses the first image from each observation, so that issue could be improved by posting observations of trees, having the bark as the first image, and getting them to Research Grade. At least that's how I understand it to work.

Posted by upupa-epops about 1 year ago (Flag)
Thumb

Wow! I am telling my Mom about this back home. She sends me photos once in a while of a plant or insect, asking me what I think it could be. I truly think this would be fun for her. I think kids would like this and it is a fun way to start getting them interested in nature and protecting it. Heck! Adults as well!. I do think this would be fun for my Autistic nephew too. What a fun way for more people to be involved in what surrounds them in there every day existence. 👍🏻

Posted by walkingstick2 about 1 year ago (Flag)
Thumb

I'm finding a small problem with Seek in that once it identifies a particular species, the slightest movement can cause it to change to a less specific level of identification such as genus level or etc. It would be really nice if it would lock on for a second once it gets to a species level ID, to give us some time to snap the photo. Several times in a row it would lose the species level ID before I could record the identification in the app. I think it could use some improvement there.

Posted by johngcramer about 1 year ago (Flag)
Thumb

I want to congratulate the team for their great effort for such a valuable application.
I was thinking about this for Himalayan region since 2 years but could not find any funding.
Please let me know if there is any funding options to contribute.

Posted by dinesh16 about 1 year ago (Flag)
Thumb

I've played with this for a few days now, and it's really amazing. Major kudos to the development team.

It's different than iNaturalist, and I think that's the point. I use it differently too -- when I'm outside and want a game with some badges, Seek's ideal. When I'm outside and want to document and share with the community, I use iNat. Also, it makes me feel great that the more that I use iNat, the better that the visual algorithm and Seek gets. :)

I think this is a PERFECT app for kids as they're out exploring nature -- and hey, I have some fun with it too!

Posted by sambiology about 1 year ago (Flag)
Thumb

I also like the idea of the older folks that are not computer savvy and don t want the trouble of learning it, how this would be easy for them to use. I have talked to a few people while out walking and they ask me what I am doing and what do I do with the photos I take? When I tell them about iNaturalsit and how easy it is to use, especially the Application on the phone. A few take on a blank look and they say I am not a tech user. I say do you email or use Facebook or other? They tell me no! Many don t think they can learn it or want the headache of it. They are set in there ways, sort of speak. (lol). I think this"Seek" would be much simpler for those people and something they could do with there grandkids or there grandkids could teach them and they could do it together. It is a different type of user friendly program that should stay simple. Thus, getting more users that are intimidated by what they consider technical and possibly finding if they can do this, maybe they can join iNaturalsit.
Either way, it is making them aware. Though this is geared for under age, I truly think it would benefit the elders as well.

I agree with @sambiology, Though it deals with nature, it is a different form of use.
Just my thoughts. ✌🏻

Posted by walkingstick2 about 1 year ago (Flag)
Thumb

I also wish there was a setting I could toggle to show latin names rather than common ones. Otherwise the app is incredible!

Posted by mykophile about 1 year ago (Flag)
Thumb

@johngcramer Out of curiosity, are you on iOS or Android?

Posted by alexshepard about 1 year ago (Flag)
Thumb

@alexshepard: I'm using Android.

Posted by johngcramer about 1 year ago (Flag)
Thumb

Why not allow to link / transfer the findings in Seek with (if present) an account in iNaturalist by the same user?

Posted by kienberg about 1 year ago (Flag)
Thumb

Sorry, I see above that this issue has been addressed above already, I see the main convenience of Seek in the simplicity if use for targets that don‘t move and are close enough to the phone, so it would save the extra import of these pics into iNaturalist, if there‘d be a link; this is not possible, of course, for more complex image situations (Tele, Macro, Löw light) taken with appropriate cameras or pics from older collections that may need editing and tagging before uploading to iNaturalist;

Posted by kienberg about 1 year ago (Flag)
Thumb

I'm interested in using this app for some programming possibilities in our natural history museum. I want to make sure that any observations made in the dioramas aren't going in to a database as we'd just be using it for identifications and information purposes. I see it being good for use with students as well as possible bioblitz training purposes. Thanks!

Posted by djsheffield about 1 year ago (Flag)
Thumb

@alexshepard - I'm seeing the same thing as johngcramer and I'm using an iPhone.

> I'm finding a small problem with Seek in that once it identifies a particular species, the slightest movement can cause it to change to a less specific level of identification such as genus level or etc.

Posted by scoutinvermont about 1 year ago (Flag)
Thumb

@djsheffield if you're asking for confirmation that using Seek on taxidermied animals won't distort anything on our end—yes, that's fine. Much better to use Seek in that circumstance than iNaturalist!

@scoutinvermont, if you get a fleeting species level ID in Seek, go ahead and quickly press the camera button and it should still get it.

Posted by carrieseltzer about 1 year ago (Flag)
Thumb

@carrieseltzer That is exactly what I was wondering, thanks! I was hoping that would be a benefit of this app.

Posted by djsheffield about 1 year ago (Flag)
Thumb

Why are only 7 languagues supported?

Posted by optilete about 1 year ago (Flag)
Thumb

@optilete we had it translated into 7 languages before releasing the update, but we plan to open it up for crowdsourced translation into other languages (just like the iNaturalist apps and website) soon.

Posted by carrieseltzer about 1 year ago (Flag)
Thumb

How is the third picture of earth created? Can it be found on iNaturalist?

Posted by optilete about 1 year ago (Flag)
Thumb

@optilete if you're up for translating Seek into other languages, you can do so here: https://crowdin.com/project/seek

Posted by alexshepard about 1 year ago (Flag)
Thumb

@johngcramer @scoutinvermont we are hoping in a future release to do "frame averaging" to create a composite prediction out of the most recent n video frames, which should help this out.

In contrast, right now what's basically happening is that each frame of the video feed (about 1 per second) is classified as if it was a solitary photograph, and Seek updates the UI to display that prediction. if you keep playing with it, you'll learn that some plants might need to be zoomed out to get a prediction, while for other plants you might need to get quite close to the flower to get a stable prediction.

Someday we may be able to train a model that is significantly more sophisticated, with spatial analysis, persistence of vision/memory, and aggregating predictions.

Posted by alexshepard about 1 year ago (Flag)
Thumb

@alexshepard https://play.google.com/store/apps/details?id=org.inaturalist.seek Maybe not all text is included in your link...

--
The description of the app org.inaturalist.seek in the Google Play Story and probably also in the iOS app store stuff is missing in crowdin. So open the playstore. Look for the app Seek and see the description of the app in the playstore is only available in English :-(

Posted by optilete about 1 year ago (Flag)
Thumb

@optilete the link I posted should send you to crowdin, a platform for translating the content of apps. We use it to translate Seek (and iNaturalist for iOS and Android) into a few languages that we support, and crowdin allows volunteers to provide additional translations.. Let me know if you can’t find the Seek project on crowdin.

Posted by alexshepard about 1 year ago (Flag)
Thumb

Fantastic creation. iNat continues to lead the way and the computer scientists are top notch. Thanks to all!

Posted by lonnyholmes about 1 year ago (Flag)
Thumb

@alexshepard I found the project and you might have noticed that from all languages in the world only Dutch hits the 100% procent translation. But if I take a look at the description text in the app store: https://play.google.com/store/apps/details?id=org.inaturalist.seek I did not find this description in Crowdin. Can this text https://play.google.com/store/apps/details?id=org.inaturalist.seek be added to Crowdin https://crowdin.com/project/seek

Posted by optilete about 1 year ago (Flag)
Thumb

@optilete thanks for your edited comment, now I understand what you meant. We will get the Play and App Store metadata added to crowdin soon, the Seek developer has been away for the past week.

Posted by alexshepard about 1 year ago (Flag)
Thumb

the Play and App Store metadata is updated, thanks!

Posted by optilete about 1 year ago (Flag)
Thumb

It would be helpful to have a table/matrix contrasting features of Seek with those of iNaturalist. As a new user of both I can only hint at some possible features to include. Here are a few starters:

Feature Seek iNaturalist
-----------------------------------------------------------
ages______________all__________ over 13
location__________blanked_______geolocated?
storage size________?________________?
collaboration_______?__________ID often offered
connectivity______ offline/online???

Posted by taaronson 11 months ago (Flag)
Thumb

I only wish I could connect Seek to the iNaturalist app....

Posted by seekinge 10 months ago (Flag)
Thumb

@seekinge can you be more specific about how you would like to connect Seek to iNaturalist?

Posted by alexshepard 9 months ago (Flag)
Thumb

Hi all! I noticed this thread is active and wanted to drop in with some feedback about Seek. I use Seek constantly and absolutely love it. There are a few things I’d change if I could, in order of what seem easy-to-hard to implement:

1. Having observations exportable as a csv file. This would make the app easier to incorporate into field botany or other science classes.

2. Offering an option to delete individual observations. Sometimes I want to get an ID from a photo on the internet without it being logged as an observation, but in order to get the ID from a photo, we have to log it as an observation.

3. Being able to look at all of our observations of a single species. I’ve used Seek to log Great Mullein in Virginia, China, and Hawaii, and it would be cool to have a record of each of these individual observations.

4. Offering the same taxonomic / Wikipedia information when the ID can’t get down to species that would be provided for a species observation. Often I find that my IDs can get down to genus, but then all the information I get is that “your observation is a member of the daylilies!” I often find myself still wanting the taxonomic context or additional information about the genus or family, like what we get for species IDs. I can even imagine a way to list some possible species in the group, like those that have been observed nearby.

5. Being able to record offline observations. I’m often using Seek where I have no cell signal, and I’ve noticed that the real-time identification still works great, but then when I push the button to record the observation it won’t work. I know that internet is required for pulling up Wikipedia and occurrence data, but I still really want to be able to take the identification I’m seeing on the real-time screen and save it with the gps point, and then be able to get the additional information once I have cell signal.

6. Introducing a social feature. Obviously tough and I scanned some of the discussion about linking Seek to iNat above. But it would be really cool to see what my friends and family have been Seeking! And having the badges and stuff just beg for some between-user competition :)

I wonder if these are things y’all have talked about or have heard other concerns with, and how hard they'd be to implement. Like I said, I use the app constantly and am really excited about its potential as a learning (and teaching!) tool.

Posted by patrick60 9 months ago (Flag)
Thumb

I'm loving this app, but am distraught to see that my observations only include the first plant, arachnid and insect I collected. Since I was in an area with no cell phone connection, I couldn't look up the species information in the field and looked forward to reviewing them when I returned to the land of the connected. Where are the rest of my observations?

And, BTW, I am thrilled that the app can still get me to species level even if I am not connected.

Posted by joellenlampman 9 months ago (Flag)
Thumb

Its a great app! Only downside is - all the stats in the app are device specific. I always carry 2 devices while on field and both are connected to my same iNaturelist account. Ideally, on both my devices I expect to see same stats in Seek app. But since the stats are device specific; I cannot see a consolidated view. Please note that; in case someone resets the devices; there is a possibility that your entire list of observations, badges might get wiped out!

I would strongly suggest developers to sync back the data from iNaturelist to prepare the stats.. that would be a great addition!

Posted by paragkokane 8 months ago (Flag)
Thumb

This is an incredible app! I've described it to friends as a real life Pokedex, and it's gotten people who otherwise wouldn't consider themselves naturalists interested in taxonomy and biodiversity. What a time to be alive.

Posted by kleu 8 months ago (Flag)
Thumb

Forgot to add--people would absolutely love a social feature. A feed that updates what my friends and family are Seeking across the country? What a fun way to connect!

Posted by kleu 8 months ago (Flag)
Thumb

What if the students have iPads and the app but do not have internet or live data - can they store it and then discover things in the classroom later?

Posted by pauldsteury 2 months ago (Flag)
Thumb

@pauldsteury Seek's in-camera identifications work without internet or live data. However, this depends on a new feature in modern iPhones and iPads, so it may not work on older iPads. Give it a try, it's free!

Posted by alexshepard 2 months ago (Flag)
Thumb

Here in the UK in the time of coronavirus, current restrictions require families to practice social distancing but they are encouraged to take daily exercise.
I'm hoping to be able to set up a challenge for local kids in our area so that they can take interest in nature on their daily walk. The idea would be that they would get the highest number of species and perhaps get a prize at the end of a set period.
I have set up local project areas in iNaturalist.
Any ideas on how I could make it work for kids through Seek?

Posted by mick13 2 months ago (Flag)
Thumb

@mick13 since Seek is a much better option for kids, you might have them submit screenshots of their observations, although if they have a lot that might not work well. Rather than just numbers of observations, I would focus on something like "can you observe a plant, an insect, and a bird", or just have them pick their favorite species they found that day and share a few facts about it.

I led nature photography camps for middle schoolers about five years ago and my colleague and I came up with scavenger hunts which the kids liked. We tried to make them a bit open-ended so that the kids could use some creativity or look for things that weren't *just* organisms. For example, photograph a plant/animal interaction, or photograph an animal's home, photograph a dead thing (even leaf could count for that), photograph something of a certain color. It was a lot of fun to see the gears in their heads turning and they really took their time looking around and noticing things.

Posted by tiwane 2 months ago (Flag)
Thumb

@mick13 I have the same question and will be working through this - do I set up a project in iNat then import it into Seek? I would like to administer a project for my kids's elementary school.

Posted by sabrinad about 2 months ago (Flag)
Thumb

I loved the app, it is very useful and interesting

Posted by teresasans about 2 months ago (Flag)
Thumb

My 6-year-old granddaughter and I had a blast using this app on our adventure walk!

Posted by truthseqr about 1 month ago (Flag)
Thumb

This looks fabulous! I need to make my students aware of this new app. Can Seek submit observations to the City Nature Challenge? I hope so! If not, could you work on project compatibility for next year's Earth day? Thanks for enabling our youngest citizen scientists. -Mr. Wes

Posted by mrwesscience about 1 month ago (Flag)
Thumb

Yes @mrwesscience! Please refer to the new Seek User Guide.

Posted by carrieseltzer about 1 month ago (Flag)

Add a Comment

Sign In or Sign Up to add comments

Is this inappropriate, spam, or offensive? Add a Flag