Real-time Computer Vision predictions in Seek by iNaturalist version 2.0

On April 5, in conjunction with the release of the Our Planet series by Netflix, iNaturalist released a new version of Seek by iNaturalist. To heed David Attenborough’s call to action in Our Planet to protect biodiversity, we need to understand what’s here and what we might lose. Seek by iNaturalist unlocks curiosity about the natural world by giving speedy identifications with computer vision and challenging you to earn badges for finding species new-to-you.

How it works

When you open the Seek camera and point it at a living thing, the app immediately tells you what you’re looking at, even before you take a picture. This on-screen identification is tied to the tree of life, and guides you towards taking a more identifiable photo by getting more specific as you fill the frame and get the right angle or features. When the app narrows it down to species, it prompts you to take a picture, which earns you a badge and unlocks more information about the species. This “augmented reality” view of the world makes it easy to explore and interpret the natural world around you all while guiding you to take more identifiable photos.  Seek can’t always identify things to species (it’s still learning...), but it aims to provide the most precise correct name it can. Here’s some footage of it in action!

In Seek v1, which launched in March 2018 with support from HHMI Tangled Bank Studios, you needed to take a photo before you had a sense of whether or not it was even identifiable by Seek. This caused frustration when people repeatedly experienced the lack of a computer vision match. Now, since you can see predictions overlayed on the camera you get immediate feedback about what you see and the specificity with which Seek can identify it.

The species included in Seek are based entirely on photos and identifications made by the global iNaturalist community, so the Seek camera will work best in places where there is already an active community of iNaturalist users, and for species that are easily identified from photos. Seek also uses data submitted to iNaturalist to show suggestions for “species nearby,” but unlike iNaturalist, findings made with Seek will not be shared publicly, making it safe for children to use.

Seek is geared to encourage outdoor exploration of wild biodiversity (rather than pets, zoo animals, or garden plants). We hope kids, families, educators, and anyone into games will start exploring their natural surroundings with Seek, and we want this to inspire the next generation of biodiversity stewards by encouraging exploration and unlocking the names of species as a way to learn more about them. We want to make it easier for curious people who may not consider themselves naturalists to learn more about nature.

Innovations in computer vision came from research collaborations with Grant Van Horn, an Adjunct Scientist with iNaturalist. Van Horn’s dissertation research at CalTech, advised by Pietro Perona, made it possible to produce a refined dataset of iNaturalist observations, train the classification model, and export it for efficient inference on mobile devices. The computer vision model includes 15,798 species and 12,524 broader taxonomic groups (such as kingdom, phylum, class, order, family, and genus). The accuracy of predictions and breadth of species included will continue to improve as the iNaturalist community and dataset grows.

How to download it

Seek is freely available on both iOS and Android. Unlike the iNaturalist mobile apps which have separate code bases for iOS and Android, we used React Native for Seek, which is a relatively new technology that allowed us to build Seek for both Android and iOS with a single codebase. It’s still nascent — there’s not even a stable 1.0.0. release yet — but we liked the benefits:

  • Saves us time in development, since we’re only using one coding language.
  • Brings Seek to a wider, more varied audience, since we’re able to support 7,000+ devices on Android alone — this also allows for a more international audience, since Android tends to be more popular anywhere outside of North America, Australia, and western parts of Europe.

Seek is currently translated into 7 languages: English, Spanish, Hindi, Chinese, Portuguese, German, and French.

This major update to Seek was made possible with support from WWF and the Our Planet series on Netflix. Seek is created by iNaturalist, which is a joint initiative of the California Academy of Sciences and the National Geographic Society.

Posted by tiwane tiwane, April 05, 2019 07:03

Comments

Thumb

Great!

Posted by carlacorazza 16 days ago (Flag)
Thumb

Great! But I would prefer to see suggestions in English or scientific names, and not in Spanish (local names are very different in each country or even region). I don't find a configuration button for changing language.

Posted by luchogu 16 days ago (Flag)
Thumb

Bravo! I've only been aware of/using iNat for a short time, but even in that short time, I've been shocked/impressed at how many changes your team has moved forward with, as well as involvement/responsiveness of folks in the forums, etc.

Posted by colincroft 16 days ago (Flag)
Thumb

This sounds really great for identifications. Excited to give it a go!

Posted by magichin 16 days ago (Flag)
Thumb

Fantastic! What a great app for children of school age to use and learn the biodiversity of our planet. This may also prove to be a boost for iNat users to get preliminary I.D.'s or confirm what they have seen but not uploaded.

Posted by lonnyholmes 16 days ago (Flag)
Thumb

Can I link my Seek account to my iNat account so that all of my identified species are on one database?

Posted by johngcramer 16 days ago (Flag)
Thumb

Very good point on linking the Seek and iNat accounts. Can it be done?

Posted by lonnyholmes 16 days ago (Flag)
Thumb

@johngcramer and @lonnyholmes Linking the two apps completely cannot be done, however we are looking into the possibility of allowing Seek users to contribute some observations to iNaturalist, but there are some hurdles. Keep in mind that one of Seek's main reasons for being is that iNaturalist cannot be used by children under 13 due to privacy issues (every observation is a public record of where and when the observer happened to be) so it's been designed for a different more casual audience.

I'm curious, do dedicated iNat users see themselves using Seek to record observations? I think they serve fairly different use cases and audiences but I could be way off here.

Posted by tiwane 16 days ago (Flag)
Thumb

@luchogu thanks for the feedback, I'll pass it along.

Posted by tiwane 16 days ago (Flag)
Thumb

No from my point. I will continue to use iNat and probably not spend much time with Seek. Its only advantage to me is the possibility of getting an I.D. without actually taking and uploading the observation. Agreed, useful for two entirely different user groups. Thank you for the quick response.

Posted by lonnyholmes 15 days ago (Flag)
Thumb

I prefer iNat, the risk I see in Seek is that, once received the answer about what they have seen, people pass away without posting on iNat.

Posted by carlacorazza 15 days ago (Flag)
Thumb

@tiwane as an amateur nature enthusiast and photog, Seek is a good way for initial id or at least then when I post to iNat I can direct to specific group. So far my posts have received good responses on possible ids , but there are unknown plants and blossoms I'd like to find out more,

Posted by magichin 15 days ago (Flag)
Thumb

As an iNaturalist noob and nature ID lightweight, I would prefer to use Seek to give me a clue what I'm looking at before I take the photo, then publish it to the public observations on iNaturalist. As a professional programmer, I can think of at least 5 different ways you could protect children's location privacy and still allow this.

1) Seek user logs into Seek with an iNaturalist account => photo locations = public. Otherwise location is obscured.
2) Seek user identifies them self as child or adult (with explanation that adults' photo locations will become public). If they identify as child, then location is Obscured in iNaturalist.
3) Seek user self-selects to allow photo submitted to iNaturalist.
4) Each time photo is IDed, Seek asks if they want to submit image with location to iNaturalist.
5) Seek could show red border around screen if photo location would be public as a warning
etc.

Posted by scoutinvermont 15 days ago (Flag)
Thumb

When I tried to download it I got "Your device is not compatible with this version" Samsung S4.

Posted by memcinto 15 days ago (Flag)
Thumb

Great for plants, but the wild birds generally don't let me get close enough for the app to recognize them - a zoom similar to the phone feature would be helpful.

Posted by aspiedextra 15 days ago (Flag)
Thumb

I might consider using it as a way to reach friends more easily. iNaturalist is complex enough to use that I think it would be a lot easier to get people into exploring nature with Seek, and using it myself would help me introduce them to it. If I did use it, I would appreciate being able to have the option of logging in and adding the observations to iNaturalist to save time (not having to open and operate both apps at once).

Posted by upupa-epops 14 days ago (Flag)
Thumb

Seek struggles to identify trees just from their bark (but then so do I in winter!).

Posted by scoutinvermont 14 days ago (Flag)
Thumb

@scoutinvermont my understanding is that the computer vision only uses the first image from each observation, so that issue could be improved by posting observations of trees, having the bark as the first image, and getting them to Research Grade. At least that's how I understand it to work.

Posted by upupa-epops 14 days ago (Flag)
Thumb

Wow! I am telling my Mom about this back home. She sends me photos once in a while of a plant or insect, asking me what I think it could be. I truly think this would be fun for her. I think kids would like this and it is a fun way to start getting them interested in nature and protecting it. Heck! Adults as well!. I do think this would be fun for my Autistic nephew too. What a fun way for more people to be involved in what surrounds them in there every day existence. 👍🏻

Posted by walkingstick2 14 days ago (Flag)
Thumb

I'm finding a small problem with Seek in that once it identifies a particular species, the slightest movement can cause it to change to a less specific level of identification such as genus level or etc. It would be really nice if it would lock on for a second once it gets to a species level ID, to give us some time to snap the photo. Several times in a row it would lose the species level ID before I could record the identification in the app. I think it could use some improvement there.

Posted by johngcramer 14 days ago (Flag)
Thumb

I want to congratulate the team for their great effort for such a valuable application.
I was thinking about this for Himalayan region since 2 years but could not find any funding.
Please let me know if there is any funding options to contribute.

Posted by dinesh16 14 days ago (Flag)
Thumb

I've played with this for a few days now, and it's really amazing. Major kudos to the development team.

It's different than iNaturalist, and I think that's the point. I use it differently too -- when I'm outside and want a game with some badges, Seek's ideal. When I'm outside and want to document and share with the community, I use iNat. Also, it makes me feel great that the more that I use iNat, the better that the visual algorithm and Seek gets. :)

I think this is a PERFECT app for kids as they're out exploring nature -- and hey, I have some fun with it too!

Posted by sambiology 14 days ago (Flag)
Thumb

I also like the idea of the older folks that are not computer savvy and don t want the trouble of learning it, how this would be easy for them to use. I have talked to a few people while out walking and they ask me what I am doing and what do I do with the photos I take? When I tell them about iNaturalsit and how easy it is to use, especially the Application on the phone. A few take on a blank look and they say I am not a tech user. I say do you email or use Facebook or other? They tell me no! Many don t think they can learn it or want the headache of it. They are set in there ways, sort of speak. (lol). I think this"Seek" would be much simpler for those people and something they could do with there grandkids or there grandkids could teach them and they could do it together. It is a different type of user friendly program that should stay simple. Thus, getting more users that are intimidated by what they consider technical and possibly finding if they can do this, maybe they can join iNaturalsit.
Either way, it is making them aware. Though this is geared for under age, I truly think it would benefit the elders as well.

I agree with @sambiology, Though it deals with nature, it is a different form of use.
Just my thoughts. ✌🏻

Posted by walkingstick2 14 days ago (Flag)
Thumb

I also wish there was a setting I could toggle to show latin names rather than common ones. Otherwise the app is incredible!

Posted by mykophile 13 days ago (Flag)
Thumb

@johngcramer Out of curiosity, are you on iOS or Android?

Posted by alexshepard 8 days ago (Flag)
Thumb

@alexshepard: I'm using Android.

Posted by johngcramer 8 days ago (Flag)
Thumb

Why not allow to link / transfer the findings in Seek with (if present) an account in iNaturalist by the same user?

Posted by kienberg 4 days ago (Flag)
Thumb

Sorry, I see above that this issue has been addressed above already, I see the main convenience of Seek in the simplicity if use for targets that don‘t move and are close enough to the phone, so it would save the extra import of these pics into iNaturalist, if there‘d be a link; this is not possible, of course, for more complex image situations (Tele, Macro, Löw light) taken with appropriate cameras or pics from older collections that may need editing and tagging before uploading to iNaturalist;

Posted by kienberg 4 days ago (Flag)
Thumb

I'm interested in using this app for some programming possibilities in our natural history museum. I want to make sure that any observations made in the dioramas aren't going in to a database as we'd just be using it for identifications and information purposes. I see it being good for use with students as well as possible bioblitz training purposes. Thanks!

Posted by djsheffield 3 days ago (Flag)
Thumb

@alexshepard - I'm seeing the same thing as johngcramer and I'm using an iPhone.

> I'm finding a small problem with Seek in that once it identifies a particular species, the slightest movement can cause it to change to a less specific level of identification such as genus level or etc.

Posted by scoutinvermont 3 days ago (Flag)
Thumb

@djsheffield if you're asking for confirmation that using Seek on taxidermied animals won't distort anything on our end—yes, that's fine. Much better to use Seek in that circumstance than iNaturalist!

@scoutinvermont, if you get a fleeting species level ID in Seek, go ahead and quickly press the camera button and it should still get it.

Posted by carrieseltzer 3 days ago (Flag)
Thumb

@carrieseltzer That is exactly what I was wondering, thanks! I was hoping that would be a benefit of this app.

Posted by djsheffield 3 days ago (Flag)
Thumb

Why are only 7 languagues supported?

Posted by optilete 2 days ago (Flag)
Thumb

@optilete we had it translated into 7 languages before releasing the update, but we plan to open it up for crowdsourced translation into other languages (just like the iNaturalist apps and website) soon.

Posted by carrieseltzer 1 day ago (Flag)

Add a Comment

Sign In or Sign Up to add comments

Is this inappropriate, spam, or offensive? Add a Flag