Real-time Computer Vision predictions in Seek by iNaturalist version 2.0

On April 5, in conjunction with the release of the Our Planet series by Netflix, iNaturalist released a new version of Seek by iNaturalist. To heed David Attenborough’s call to action in Our Planet to protect biodiversity, we need to understand what’s here and what we might lose. Seek by iNaturalist unlocks curiosity about the natural world by giving speedy identifications with computer vision and challenging you to earn badges for finding species new-to-you.

How it works

When you open the Seek camera and point it at a living thing, the app immediately tells you what you’re looking at, even before you take a picture. This on-screen identification is tied to the tree of life, and guides you towards taking a more identifiable photo by getting more specific as you fill the frame and get the right angle or features. When the app narrows it down to species, it prompts you to take a picture, which earns you a badge and unlocks more information about the species. This “augmented reality” view of the world makes it easy to explore and interpret the natural world around you all while guiding you to take more identifiable photos.  Seek can’t always identify things to species (it’s still learning...), but it aims to provide the most precise correct name it can. Here’s some footage of it in action!

In Seek v1, which launched in March 2018 with support from HHMI Tangled Bank Studios, you needed to take a photo before you had a sense of whether or not it was even identifiable by Seek. This caused frustration when people repeatedly experienced the lack of a computer vision match. Now, since you can see predictions overlayed on the camera you get immediate feedback about what you see and the specificity with which Seek can identify it.

The species included in Seek are based entirely on photos and identifications made by the global iNaturalist community, so the Seek camera will work best in places where there is already an active community of iNaturalist users, and for species that are easily identified from photos. Seek also uses data submitted to iNaturalist to show suggestions for “species nearby,” but unlike iNaturalist, findings made with Seek will not be shared publicly, making it safe for children to use.

Seek is geared to encourage outdoor exploration of wild biodiversity (rather than pets, zoo animals, or garden plants). We hope kids, families, educators, and anyone into games will start exploring their natural surroundings with Seek, and we want this to inspire the next generation of biodiversity stewards by encouraging exploration and unlocking the names of species as a way to learn more about them. We want to make it easier for curious people who may not consider themselves naturalists to learn more about nature.

Innovations in computer vision came from research collaborations with Grant Van Horn, an Adjunct Scientist with iNaturalist. Van Horn’s dissertation research at CalTech, advised by Pietro Perona, made it possible to produce a refined dataset of iNaturalist observations, train the classification model, and export it for efficient inference on mobile devices. The computer vision model includes 15,798 species and 12,524 broader taxonomic groups (such as kingdom, phylum, class, order, family, and genus). The accuracy of predictions and breadth of species included will continue to improve as the iNaturalist community and dataset grows.

How to download it

Seek is freely available on both iOS and Android. Unlike the iNaturalist mobile apps which have separate code bases for iOS and Android, we used React Native for Seek, which is a relatively new technology that allowed us to build Seek for both Android and iOS with a single codebase. It’s still nascent — there’s not even a stable 1.0.0. release yet — but we liked the benefits:

  • Saves us time in development, since we’re only using one coding language.
  • Brings Seek to a wider, more varied audience, since we’re able to support 7,000+ devices on Android alone — this also allows for a more international audience, since Android tends to be more popular anywhere outside of North America, Australia, and western parts of Europe.

Seek is currently translated into 7 languages: English, Spanish, Hindi, Chinese, Portuguese, German, and French.

This major update to Seek was made possible with support from WWF and the Our Planet series on Netflix. Seek is created by iNaturalist, which is a joint initiative of the California Academy of Sciences and the National Geographic Society.

Posted by tiwane tiwane, April 05, 2019 07:03

Comments

Great!

Posted by carlacorazza over 2 years ago (Flag)

Great! But I would prefer to see suggestions in English or scientific names, and not in Spanish (local names are very different in each country or even region). I don't find a configuration button for changing language.

Posted by luchogu over 2 years ago (Flag)

Bravo! I've only been aware of/using iNat for a short time, but even in that short time, I've been shocked/impressed at how many changes your team has moved forward with, as well as involvement/responsiveness of folks in the forums, etc.

Posted by colincroft over 2 years ago (Flag)

This sounds really great for identifications. Excited to give it a go!

Posted by magichin over 2 years ago (Flag)

Fantastic! What a great app for children of school age to use and learn the biodiversity of our planet. This may also prove to be a boost for iNat users to get preliminary I.D.'s or confirm what they have seen but not uploaded.

Posted by lonnyholmes over 2 years ago (Flag)

Can I link my Seek account to my iNat account so that all of my identified species are on one database?

Posted by johngcramer over 2 years ago (Flag)

Very good point on linking the Seek and iNat accounts. Can it be done?

Posted by lonnyholmes over 2 years ago (Flag)

@johngcramer and @lonnyholmes Linking the two apps completely cannot be done, however we are looking into the possibility of allowing Seek users to contribute some observations to iNaturalist, but there are some hurdles. Keep in mind that one of Seek's main reasons for being is that iNaturalist cannot be used by children under 13 due to privacy issues (every observation is a public record of where and when the observer happened to be) so it's been designed for a different more casual audience.

I'm curious, do dedicated iNat users see themselves using Seek to record observations? I think they serve fairly different use cases and audiences but I could be way off here.

Posted by tiwane over 2 years ago (Flag)

@luchogu thanks for the feedback, I'll pass it along.

Posted by tiwane over 2 years ago (Flag)

No from my point. I will continue to use iNat and probably not spend much time with Seek. Its only advantage to me is the possibility of getting an I.D. without actually taking and uploading the observation. Agreed, useful for two entirely different user groups. Thank you for the quick response.

Posted by lonnyholmes over 2 years ago (Flag)

I prefer iNat, the risk I see in Seek is that, once received the answer about what they have seen, people pass away without posting on iNat.

Posted by carlacorazza over 2 years ago (Flag)

@tiwane as an amateur nature enthusiast and photog, Seek is a good way for initial id or at least then when I post to iNat I can direct to specific group. So far my posts have received good responses on possible ids , but there are unknown plants and blossoms I'd like to find out more,

Posted by magichin over 2 years ago (Flag)

As an iNaturalist noob and nature ID lightweight, I would prefer to use Seek to give me a clue what I'm looking at before I take the photo, then publish it to the public observations on iNaturalist. As a professional programmer, I can think of at least 5 different ways you could protect children's location privacy and still allow this.

1) Seek user logs into Seek with an iNaturalist account => photo locations = public. Otherwise location is obscured.
2) Seek user identifies them self as child or adult (with explanation that adults' photo locations will become public). If they identify as child, then location is Obscured in iNaturalist.
3) Seek user self-selects to allow photo submitted to iNaturalist.
4) Each time photo is IDed, Seek asks if they want to submit image with location to iNaturalist.
5) Seek could show red border around screen if photo location would be public as a warning
etc.

Posted by scoutinvermont over 2 years ago (Flag)

When I tried to download it I got "Your device is not compatible with this version" Samsung S4.

Posted by memcinto over 2 years ago (Flag)

Great for plants, but the wild birds generally don't let me get close enough for the app to recognize them - a zoom similar to the phone feature would be helpful.

Posted by aspiedextra over 2 years ago (Flag)

I might consider using it as a way to reach friends more easily. iNaturalist is complex enough to use that I think it would be a lot easier to get people into exploring nature with Seek, and using it myself would help me introduce them to it. If I did use it, I would appreciate being able to have the option of logging in and adding the observations to iNaturalist to save time (not having to open and operate both apps at once).

Posted by upupa-epops over 2 years ago (Flag)

Seek struggles to identify trees just from their bark (but then so do I in winter!).

Posted by scoutinvermont over 2 years ago (Flag)

@scoutinvermont my understanding is that the computer vision only uses the first image from each observation, so that issue could be improved by posting observations of trees, having the bark as the first image, and getting them to Research Grade. At least that's how I understand it to work.

Posted by upupa-epops over 2 years ago (Flag)

Wow! I am telling my Mom about this back home. She sends me photos once in a while of a plant or insect, asking me what I think it could be. I truly think this would be fun for her. I think kids would like this and it is a fun way to start getting them interested in nature and protecting it. Heck! Adults as well!. I do think this would be fun for my Autistic nephew too. What a fun way for more people to be involved in what surrounds them in there every day existence. 👍🏻

Posted by walkingstick2 over 2 years ago (Flag)

I'm finding a small problem with Seek in that once it identifies a particular species, the slightest movement can cause it to change to a less specific level of identification such as genus level or etc. It would be really nice if it would lock on for a second once it gets to a species level ID, to give us some time to snap the photo. Several times in a row it would lose the species level ID before I could record the identification in the app. I think it could use some improvement there.

Posted by johngcramer over 2 years ago (Flag)

I want to congratulate the team for their great effort for such a valuable application.
I was thinking about this for Himalayan region since 2 years but could not find any funding.
Please let me know if there is any funding options to contribute.

Posted by dinesh16 over 2 years ago (Flag)

I've played with this for a few days now, and it's really amazing. Major kudos to the development team.

It's different than iNaturalist, and I think that's the point. I use it differently too -- when I'm outside and want a game with some badges, Seek's ideal. When I'm outside and want to document and share with the community, I use iNat. Also, it makes me feel great that the more that I use iNat, the better that the visual algorithm and Seek gets. :)

I think this is a PERFECT app for kids as they're out exploring nature -- and hey, I have some fun with it too!

Posted by sambiology over 2 years ago (Flag)

I also like the idea of the older folks that are not computer savvy and don t want the trouble of learning it, how this would be easy for them to use. I have talked to a few people while out walking and they ask me what I am doing and what do I do with the photos I take? When I tell them about iNaturalsit and how easy it is to use, especially the Application on the phone. A few take on a blank look and they say I am not a tech user. I say do you email or use Facebook or other? They tell me no! Many don t think they can learn it or want the headache of it. They are set in there ways, sort of speak. (lol). I think this"Seek" would be much simpler for those people and something they could do with there grandkids or there grandkids could teach them and they could do it together. It is a different type of user friendly program that should stay simple. Thus, getting more users that are intimidated by what they consider technical and possibly finding if they can do this, maybe they can join iNaturalsit.

Either way, it is making them aware. Though this is geared for under age, I truly think it would benefit the elders as well.

I agree with @sambiology, Though it deals with nature, it is a different form of use.
Just my thoughts. ✌🏻

Posted by walkingstick2 over 2 years ago (Flag)

I also wish there was a setting I could toggle to show latin names rather than common ones. Otherwise the app is incredible!

Posted by mykophile over 2 years ago (Flag)

@johngcramer Out of curiosity, are you on iOS or Android?

Posted by alexshepard over 2 years ago (Flag)

@alexshepard: I'm using Android.

Posted by johngcramer over 2 years ago (Flag)

Why not allow to link / transfer the findings in Seek with (if present) an account in iNaturalist by the same user?

Posted by kienberg over 2 years ago (Flag)

Sorry, I see above that this issue has been addressed above already, I see the main convenience of Seek in the simplicity if use for targets that don‘t move and are close enough to the phone, so it would save the extra import of these pics into iNaturalist, if there‘d be a link; this is not possible, of course, for more complex image situations (Tele, Macro, Löw light) taken with appropriate cameras or pics from older collections that may need editing and tagging before uploading to iNaturalist;

Posted by kienberg over 2 years ago (Flag)

I'm interested in using this app for some programming possibilities in our natural history museum. I want to make sure that any observations made in the dioramas aren't going in to a database as we'd just be using it for identifications and information purposes. I see it being good for use with students as well as possible bioblitz training purposes. Thanks!

Posted by djsheffield over 2 years ago (Flag)

@alexshepard - I'm seeing the same thing as johngcramer and I'm using an iPhone.

I'm finding a small problem with Seek in that once it identifies a particular species, the slightest movement can cause it to change to a less specific level of identification such as genus level or etc.

Posted by scoutinvermont over 2 years ago (Flag)

@djsheffield if you're asking for confirmation that using Seek on taxidermied animals won't distort anything on our end—yes, that's fine. Much better to use Seek in that circumstance than iNaturalist!

@scoutinvermont, if you get a fleeting species level ID in Seek, go ahead and quickly press the camera button and it should still get it.

Posted by carrieseltzer over 2 years ago (Flag)

@carrieseltzer That is exactly what I was wondering, thanks! I was hoping that would be a benefit of this app.

Posted by djsheffield over 2 years ago (Flag)

Why are only 7 languagues supported?

Posted by optilete over 2 years ago (Flag)

@optilete we had it translated into 7 languages before releasing the update, but we plan to open it up for crowdsourced translation into other languages (just like the iNaturalist apps and website) soon.

Posted by carrieseltzer over 2 years ago (Flag)

How is the third picture of earth created? Can it be found on iNaturalist?

Posted by optilete over 2 years ago (Flag)

@optilete if you're up for translating Seek into other languages, you can do so here: https://crowdin.com/project/seek

Posted by alexshepard over 2 years ago (Flag)

@johngcramer @scoutinvermont we are hoping in a future release to do "frame averaging" to create a composite prediction out of the most recent n video frames, which should help this out.

In contrast, right now what's basically happening is that each frame of the video feed (about 1 per second) is classified as if it was a solitary photograph, and Seek updates the UI to display that prediction. if you keep playing with it, you'll learn that some plants might need to be zoomed out to get a prediction, while for other plants you might need to get quite close to the flower to get a stable prediction.

Someday we may be able to train a model that is significantly more sophisticated, with spatial analysis, persistence of vision/memory, and aggregating predictions.

Posted by alexshepard over 2 years ago (Flag)

@alexshepard https://play.google.com/store/apps/details?id=org.inaturalist.seek Maybe not all text is included in your link...

--
The description of the app org.inaturalist.seek in the Google Play Story and probably also in the iOS app store stuff is missing in crowdin. So open the playstore. Look for the app Seek and see the description of the app in the playstore is only available in English :-(

Posted by optilete over 2 years ago (Flag)

@optilete the link I posted should send you to crowdin, a platform for translating the content of apps. We use it to translate Seek (and iNaturalist for iOS and Android) into a few languages that we support, and crowdin allows volunteers to provide additional translations.. Let me know if you can’t find the Seek project on crowdin.

Posted by alexshepard over 2 years ago (Flag)

Fantastic creation. iNat continues to lead the way and the computer scientists are top notch. Thanks to all!

Posted by lonnyholmes over 2 years ago (Flag)

@alexshepard I found the project and you might have noticed that from all languages in the world only Dutch hits the 100% procent translation. But if I take a look at the description text in the app store: https://play.google.com/store/apps/details?id=org.inaturalist.seek I did not find this description in Crowdin. Can this text https://play.google.com/store/apps/details?id=org.inaturalist.seek be added to Crowdin https://crowdin.com/project/seek

Posted by optilete over 2 years ago (Flag)

@optilete thanks for your edited comment, now I understand what you meant. We will get the Play and App Store metadata added to crowdin soon, the Seek developer has been away for the past week.

Posted by alexshepard over 2 years ago (Flag)

the Play and App Store metadata is updated, thanks!

Posted by optilete over 2 years ago (Flag)

It would be helpful to have a table/matrix contrasting features of Seek with those of iNaturalist. As a new user of both I can only hint at some possible features to include. Here are a few starters:

Feature Seek iNaturalist

ages___________all_______ over 13
location_______blanked_geolocated?
storage size
__?__________?
collaboration
_?____ID often offered
connectivity
___ offline/online???

Posted by taaronson about 2 years ago (Flag)

I only wish I could connect Seek to the iNaturalist app....

Posted by seekinge about 2 years ago (Flag)

@seekinge can you be more specific about how you would like to connect Seek to iNaturalist?

Posted by alexshepard about 2 years ago (Flag)

Hi all! I noticed this thread is active and wanted to drop in with some feedback about Seek. I use Seek constantly and absolutely love it. There are a few things I’d change if I could, in order of what seem easy-to-hard to implement:

Having observations exportable as a csv file. This would make the app easier to incorporate into field botany or other science classes.

Offering an option to delete individual observations. Sometimes I want to get an ID from a photo on the internet without it being logged as an observation, but in order to get the ID from a photo, we have to log it as an observation.

Being able to look at all of our observations of a single species. I’ve used Seek to log Great Mullein in Virginia, China, and Hawaii, and it would be cool to have a record of each of these individual observations.

Offering the same taxonomic / Wikipedia information when the ID can’t get down to species that would be provided for a species observation. Often I find that my IDs can get down to genus, but then all the information I get is that “your observation is a member of the daylilies!” I often find myself still wanting the taxonomic context or additional information about the genus or family, like what we get for species IDs. I can even imagine a way to list some possible species in the group, like those that have been observed nearby.

Being able to record offline observations. I’m often using Seek where I have no cell signal, and I’ve noticed that the real-time identification still works great, but then when I push the button to record the observation it won’t work. I know that internet is required for pulling up Wikipedia and occurrence data, but I still really want to be able to take the identification I’m seeing on the real-time screen and save it with the gps point, and then be able to get the additional information once I have cell signal.

Introducing a social feature. Obviously tough and I scanned some of the discussion about linking Seek to iNat above. But it would be really cool to see what my friends and family have been Seeking! And having the badges and stuff just beg for some between-user competition :)

I wonder if these are things y’all have talked about or have heard other concerns with, and how hard they'd be to implement. Like I said, I use the app constantly and am really excited about its potential as a learning (and teaching!) tool.

Posted by patrick60 about 2 years ago (Flag)

I'm loving this app, but am distraught to see that my observations only include the first plant, arachnid and insect I collected. Since I was in an area with no cell phone connection, I couldn't look up the species information in the field and looked forward to reviewing them when I returned to the land of the connected. Where are the rest of my observations?

And, BTW, I am thrilled that the app can still get me to species level even if I am not connected.

Posted by joellenlampman about 2 years ago (Flag)

Its a great app! Only downside is - all the stats in the app are device specific. I always carry 2 devices while on field and both are connected to my same iNaturelist account. Ideally, on both my devices I expect to see same stats in Seek app. But since the stats are device specific; I cannot see a consolidated view. Please note that; in case someone resets the devices; there is a possibility that your entire list of observations, badges might get wiped out!

I would strongly suggest developers to sync back the data from iNaturelist to prepare the stats.. that would be a great addition!

Posted by paragkokane almost 2 years ago (Flag)

This is an incredible app! I've described it to friends as a real life Pokedex, and it's gotten people who otherwise wouldn't consider themselves naturalists interested in taxonomy and biodiversity. What a time to be alive.

Posted by kleu almost 2 years ago (Flag)

Forgot to add--people would absolutely love a social feature. A feed that updates what my friends and family are Seeking across the country? What a fun way to connect!

Posted by kleu almost 2 years ago (Flag)

What if the students have iPads and the app but do not have internet or live data - can they store it and then discover things in the classroom later?

Posted by pauldsteury over 1 year ago (Flag)

@pauldsteury Seek's in-camera identifications work without internet or live data. However, this depends on a new feature in modern iPhones and iPads, so it may not work on older iPads. Give it a try, it's free!

Posted by alexshepard over 1 year ago (Flag)

Here in the UK in the time of coronavirus, current restrictions require families to practice social distancing but they are encouraged to take daily exercise.
I'm hoping to be able to set up a challenge for local kids in our area so that they can take interest in nature on their daily walk. The idea would be that they would get the highest number of species and perhaps get a prize at the end of a set period.
I have set up local project areas in iNaturalist.
Any ideas on how I could make it work for kids through Seek?

Posted by mick13 over 1 year ago (Flag)

@mick13 since Seek is a much better option for kids, you might have them submit screenshots of their observations, although if they have a lot that might not work well. Rather than just numbers of observations, I would focus on something like "can you observe a plant, an insect, and a bird", or just have them pick their favorite species they found that day and share a few facts about it.

I led nature photography camps for middle schoolers about five years ago and my colleague and I came up with scavenger hunts which the kids liked. We tried to make them a bit open-ended so that the kids could use some creativity or look for things that weren't just organisms. For example, photograph a plant/animal interaction, or photograph an animal's home, photograph a dead thing (even leaf could count for that), photograph something of a certain color. It was a lot of fun to see the gears in their heads turning and they really took their time looking around and noticing things.

Posted by tiwane over 1 year ago (Flag)

@mick13 I have the same question and will be working through this - do I set up a project in iNat then import it into Seek? I would like to administer a project for my kids's elementary school.

Posted by sabrinad over 1 year ago (Flag)

I loved the app, it is very useful and interesting

Posted by teresasans over 1 year ago (Flag)

My 6-year-old granddaughter and I had a blast using this app on our adventure walk!

Posted by truthseqr over 1 year ago (Flag)

This looks fabulous! I need to make my students aware of this new app. Can Seek submit observations to the City Nature Challenge? I hope so! If not, could you work on project compatibility for next year's Earth day? Thanks for enabling our youngest citizen scientists. -Mr. Wes

Posted by mrwesscience over 1 year ago (Flag)

Yes @mrwesscience! Please refer to the new Seek User Guide.

Posted by carrieseltzer over 1 year ago (Flag)

I've been using Seek for a month and a half (and I think this time of usage shall be long enough for me to write a brief review and critiques).

So far, Seek has helped me identify lots of species I often overlooked at, like grasses, palms, even amaranths (which I often refer as 'weeds'). It's versatile when it comes to identify plants, but here's what I found when I started observing animals and fungi:

It's unable to identify several kinds of fungi (save for a couple of stinking dapperlings). Wherever I aimed my phone camera to the fungus, Seek always stuck to the fungus' genus (I forgot which one). The fungus was in a good condition, and it may decay in a couple of days, so I thought the app should be able to identify it. But, unfortunately, no.
It seems Seek is still have issues identifying freshwater fish species. It misidentified a lemon cichlid as a yellowhead wrasse, a male guppy as a Siamese fighting fish (although from a distance, the guppy does look like one), a peacock bass as a Mayan cichlid, and mollies as garibaldi. Furthermore, Seek can't identify catfishes (esp. plecos or sharptooth catfish), angelfishes, neon and cardinal tetras, clown featherbacks, black ghost knifefish, and arowanas.
Identifying birds is easier said than done; photographing a bird with a phone is a disaster, and one needs a huge amount of luck to get one crystal clear. So far, I've identified 15 out of 18 birds I've encountered with Seek. The app struggles to identify blue-winged leafbirds (and eventually misidentified it as a green jay), and it's unable to identify grey-cheeked bulbul and Fischer's lovebird. And mind you, some birds will start jumping and flying around when you attempted to approach them (if caged) and just flutter away when you're looking at them intensely.
While I appreciate Seek's capability to identify jumping spiders (which I truly love), it somehow struggles to identify ants, esp. that one trap-jaw ant. Overall, it's also a disaster to photographing insects with a phone, since they'll just fly away when you approach them too closely, but looking at Seek's identification algorithm on insects (it only got 1 wrong identification out of 9)
Identifying geckos - genus Hemidactylus, in particular - is surprisingly hard. Seek only got a flat-tailed house gecko after about 4 failed attempts to identify the reptile. And several other types of lizards, like the changeable lizard, is a hit-or-miss; you'll ended up identifying the lizard with a photo that looks like you accidentally took a selfie.
I found it funny Seek took an effort to identify a (domestic) dog. This cute little dog is a Pomeranian, by the way. But somehow, it's easily identify a Turkish Angora of mine.

While I found out that you can just simply identify organisms from your photo library, I don't go that way; that would be less thrilling than snapping pictures with your phone while risking your trousers or shirts getting dirty or straight-out cheating (download pictures from the Internet and identify it, eh?), but it may be useful for identifying animals that otherwise would be a pain to photograph by phones, i.e. birds, spiders, or butterflies. My critique is just one: update the database so freshwater fishies can get more justice (kind of). Save for Seek unable to identify a 'South American amaryllis' and 'Bougainvillea,' it's a nice app.

8,5/10. I'm looking more for updates on the species' database and smarter identification algorithm. Also, add more challenges that might force me to take a picture of a saola :)

Posted by vagabond46 over 1 year ago (Flag)

If you want a smarter algorithm for identification,use the iNaturalist app or the web.

Posted by optilete over 1 year ago (Flag)

J'aime tellement

Posted by jeremie10 11 months ago (Flag)

"Seek also uses data submitted to iNaturalist to show suggestions for “species nearby,” but unlike iNaturalist, findings made with Seek will not be shared publicly, making it safe for children to use."

Finally figured out the main difference between the two apps!

Posted by spicetruck 3 months ago (Flag)

I think your information is outdated..
Hope this post and its link are up2date
https://forum.inaturalist.org/t/seek-app-vs-inaturalist/16606/5?u=optilete

Posted by optilete 3 months ago (Flag)

Add a Comment

Sign In or Sign Up to add comments