Adding Observations State-by-State
Purpose of This Post:
Several of us have been working on populating this new project, working through all the Lepidoptera observations for each state in the project area & adding those that show caterpillars. I will keep track of our progress in this post. As you complete a state, add a comment & I will update the list accordingly. It would also be helpful if you would post which states you are working on (or intend to work on), so that we don't duplicate efforts.
Completed:
Alabama - jtuttle (checked by kylejones)
Arkansas - jtuttle (checked by kylejones)
Connecticut - berkshirenaturalist (checked by kylejones)
Delaware - eraskin (checked by kylejones)
Florida - jtuttle (checked by kylejones)
Georgia - jtuttle (checked by kylejones)
Illinois - jtuttle (checked by kylejones)
Indiana - eraskin (checked by kylejones)
Iowa - eraskin (checked by kylejones)
Kansas - eraskin (checked by kylejones)
Kentucky - eraskin (checked by kylejones)
Louisiana - jtuttle (checked by kylejones)
Maine - berkshirenaturalist (checked by kylejones)
Maryland - kylejones
Massachusetts - berkshirenaturalist (checked by kylejones)
Michigan - kylejones (checked by eraskin)
Minnesota - jtuttle (checked by kylejones)
Mississippi - jtuttle (checked by kylejones)
Missouri - jtuttle (checked by kylejones)
Nebraska - eraskin (checked by kylejones)
New Hampshire - berkshirenaturalist (checked by kylejones)
New Jersey - jtuttle (checked by kylejones)
New York - berkshirenaturalist (checked by kylejones)
North Carolina - eraskin (checked by kylejones)
North Dakota - eraskin (checked by kylejones)
Ohio - kylejones
Oklahoma - kylejones (checked by eraskin)
Pennsylvania - berkeshirenaturalist & kylejones
Rhode Island - berkshirenaturalist (checked by kylejones)
South Carolina - jtuttle (checked by kylejones)
South Dakota - eraskin (checked by kylejones)
Tennessee - jtuttle (checked by kylejones)
Vermont - berkshirenaturalist (checked by kylejones)
Virginia - kylejones
Washington, D.C. - kylejones (checked by eraskin)
West Virginia - jtuttle (checked by kylejones)
Wisconsin - jtuttle (checked by kylejones)
Underway:
Texas (see separate post here)
Methods:
I have been using the Identify tool to work through states - this way, I can mark each page of observations as reviewed, which will make it easier to review only the new observations for those states (won't have to look through all the observations of mature butterflies & moths over & over). If you would like to use that process, you can click this link & then search for a state in the "Place" box. This will pull up all Lepidoptera observations for that state that have not yet been added to the project, in ascending order by observation date (this is my personal preference, but you can do them in whatever order you please, of course). For more discussion of methods, see this post & related comments.
Last update: 1/19/2017, 2131
Comments
FYI, before I switched to state-by-state, I was doing all of eastern U.S., oldest to newest, so I did a ton on VT, NH, and other New England state (moot now, since those states have been completed), but also on OH, PA, VA, FL, NY, MI, and WV -- so whoever works on those may (unfortunately) have to click through some number of pages without caterpillars...
I had started working on TN, but eraskin, are you still working on that? If not, I'll tackle that next. If so, I'll go to WI and MN (probably easy to knock those off).
Good information. You can go ahead & take on TN, thanks.
I'm still running crossways and entering species groups for eastern U.S. (tussocks, silks, a few others). Also cleaning up each state -- there seems to be a few that get through the cracks each time. From the top of the list above I have worked down through Alabama on clean-up and probably picked up close to 100 that way. Legitimate Leps I think, no sawflies, pupae, or cobwebs. Well, maybe a few.
That brings up something I meant to ask you all about: how you are handling tent caterpillars and webworms, questionable photos, and bagworms. Here's my process on these things, more or less:
--I have been including tent caterpillar or webworm nets when there is clearly a mass of caterpillars in them with some hope of ID or already with an ID. There are a few tents identified to genus (Malacosoma) or only to Lepidoptera that I left out because there was so little info on the actual caterpillars in the photo; however, perhaps those should be included (and perhaps you have picked them up, kylejones, in your clean-up). Leaving them out most likely under-represents them, and as a scientist, I worry about that. :-)
--I've pulled in anything that clearly seems to be a caterpillar, even if there's little hope of identifying it beyond genus. I've excluded a few extremely blurry or tiny photos that I'm not confident can be called Lepidoptera based on the info in the photo. Again, I've probably missed a few that should have been included or included a few that shouldn't have been...
--I've included a couple of bagworm observations where I can see the caterpillar protruding from the pupa, but otherwise I have excluded all bagworm observations and all pupa observations. Should bagworms be included? Are they any different from other pupae?
I'm glad to modify how I'm doing this -- let me know what you think!
I'm pretty consistent with your suggestions, but should probably recheck tent cats--it shouldn't take long. Towards the end I included them if there were clearly identifiable cats on the web. Not sure about the beginning. Since this is a caterpillar project maybe there needs to be a clear sign that there are caterpillars present. Bagworm can be cat or pupae so not so straightforward I guess. I don't think I've run into either during this exercise.
I excluded low quality photos similar to your suggestions. Some sawfly larvae slipped in and I re-identified, but they may still be in the project. I couldn't actually remove them.
I don't know the range of all the Malacosoma, but in the northeast if it is a Malacosoma in a tent it is an M. americana since disstria doesn't build tents.
And of course, please check my work!
@kylejones, thanks for doing that quality control. I'm sure I have missed a few here & there. And thanks for re-identifying sawfly larvae when you find them - I don't have the experience to distinguish, but have used admin tools to throw out the ones that were easy to find (ID rounded up to subclass because of your IDs).
I have been thinking that we would need to decide what to do about tent caterpillars, bagworms, etc. Thanks for bringing it up, @jtuttle. I don't think that I have been consistent in whether I have included them or not, but it seems like a general policy of only including observations in which at least on caterpillar is in clear evidence (rather than simply a caterpillar artifact) best aligns with my original intent for the project. Does that make sense?
I have also added a few that were tiny or blurry but clearly caterpillars. Not sure these observations really have any value, but not really hurting us. At some point I may go through & mark them as not verifiable.
Thank you all for your interest & attention to detail!
I realized Washington D.C. is treated as a state, so I just pulled those in.
@kylejones, I just sorted the states alphabetically - hope that doesn't mess up your order for re-checking. I added notes for the ones you've checked, so it's still clear which ones haven't been checked. Seems like, as we get more states, it will be a lot easier to find things if we keep them in order.
checked iowa, kansas, kentucky, louisiana, new hampshire
Tennessee, Wisconsin, and Minnesota completed.
Arkansas completed.
Awesome! I have added a list of the states still to be completed (see above), and have claimed a couple more for myself. Need some low-hanging fruit to keep me motivated (still only about 1/4 of the way through NC).
West Virginia, and New Jersey completed. That's it for me for a while. :-)
Delaware, Mississippi checked, I'm slowing down too!
Arkansas checked, and I'm picking up far fewer missed cats than before.
We are getting better at it as we go. :-)
More checked: Arkansas, Indiana, Minnesota
Completed: Oklahoma
South Carolina checked
Ohio underway
I can make a start on Maryland.
Just finished Ohio. Feel free to check my work!
Checked: Delaware, New Jersey, Tennessee, Wisconsin.
I did about half of Missouri and will work on finishing it up.
Missouri completed.
Missouri, New York, West Virginia checked.
Missouri is already done--you can take it off the unclaimed list. I will work on Michigan. Have you seen the size of Texas? Over 49K.
Texas is intimidating!
Thanks - I have corrected that. Closing in on halfway through NC's 16k, and I'll work on VA as well. Maybe there's some way we can divide up the Texas observations to make them more manageable.
I'm game if we can divide them up.
There were iNaturalist Places for East Texas & West Texas, but they were points rather than polygons. I added polygons (divided at the 100th meridian), so we should now be able to easily target the eastern part of the state (place_id=65181). However, it will take some time for iNaturalist to re-index the observations that fall into those new polygons, so probably not ready to go yet (at the moment, only 3 observations pop up for East Texas). We may need to divide & conquer further still, but hopefully this will help once the system catches up.
Just hit the halfway mark for NC (8,000). Slow & steady... And the project is closing in on 9,000 observations! Hoping for at least 10k by the time we get through all the states, as an impressive (if arbitrary) benchmark.
It appears that the East Texas place has been populated with observations. It also appears that most of the Leps observed in Texas are from this half of the state (still more than 40,000 observations). Maybe we can go county-by-county, or family by family?
...on a less ambitious note, Michigan is done!
For Texas, we could perhaps start with the urban counties, since that's likely where most of the observations are. If more than one person is needed for a particular county (i.e., if tons of observations), one person could start from the "beginning" (obs sorted in ascending order) and the other could start from the "end" (obs sorted in descending order)...or something alone those lines.
I'm working on Virginia.
That's a good idea. I need to look at a Texas county map.
Making much better progress on NC since I quit letting myself get distracted by more manageable states. I'll start looking at how best to split up Texas counties once I finish with that.
I'm working on Illinois.
Just added our 10,000th observation! More here. Would never have gotten this far without such great help from you all.
NC complete! I'm going to take a little break before figuring out how to address Texas. Looks like they have a heap of counties.
I'll check Michigan, DC, & Oklahoma. Also working on pulling the PlaceIDs for all the states in our project area, so that I can concatenate the ones for the states I have reviewed & thereby easily review them all at once as new observations are added.
Here's the search string that I will use to monitor new observations for the states that I have added caterpillars for: http://www.inaturalist.org/observations/identify?quality_grade=any&order=asc&place_id=3,4,13,20,24,25,26,30,44&taxon_id=47157¬_in_project=caterpillars-of-the-eastern-us
Here are the PlaceID strings for the states that the rest of you dedicated caterpillar-observation-adders have completed. As you complete additional states (or places - guess we may get into individual counties in Texas), just add on their PlaceIDs. If you want to continue monitoring the states that you have worked on, which of course you should only do if you want to/it seems like fun.
@berkshirenaturalist: 2,8,17,41,48,49
@jtuttle: 19,23,27,28,32,33,36,37,38,43,45,51
@kylejones: 5,12,29,31
For reference, here are all the PlaceIDs for states in our project area:
Alabama: 19
Arkansas: 36
Connecticut: 49
Delaware: 4
Florida: 21
Georgia: 23
Illinois: 35
Indiana: 20
Iowa: 24
Kansas: 25
Kentucky: 26
Louisiana: 27
Maine: 17
Maryland: 39
Massachusetts: 2
Michigan: 29
Minnesota: 38
Mississippi: 37
Missouri: 28
Nebraska: 3
New Hampshire: 41
New Jersey: 51
New York: 48
North Carolina: 30
North Dakota: 13
Ohio: 31
Oklahoma: 12
Pennsylvania: 42
Rhode Island: 8
South Carolina: 43
South Dakota: 44
Tennessee: 45
Texas: 18
Texas (Eastern): 65181
Vermont: 47
Virginia: 7
Washington, D.C.: 5
West Virginia: 33
Wisconsin: 32
That's great, thanks! I've been occasionally adding new observations by searching in the "US Eastern States, US" place, but I think that doesn't catch all of our states, and going state by state won't be feasible over the long term -- glad to know how to concatenate state IDs.
Its to bad the filter option doesn't have regional options such as this http://thomaslegion.net/uscensusbureauregionsthewestthemidwestthesouthandthenortheast.html . You can filter by date observation added and it would be easier to maintain, but unfortunately I don't think there's a way to filter out west coast observations on a larger scale.
Virginia is done.
I didn't really enter much for Vermont other than my own sightings, so I'm doing the check for Vermont. It's pretty clean, but taking a while because of the number of observations.
I did Vermont but you can double check it if you want.
There is an "American Southeast" place that I also use sometimes. I don't know if there is one for the Northeast and Midwest -- worth checking, perhaps...
Just checked. There is a "US Mid-Atlantic, US" place and a "Northeast & Mid Atlantic States, US" place.
Illinois is complete.
Finished checking Vermont, Illinois.
Any ideas how to mark as reviewed a single observer (~6,000 observations in one state) since they do not allow their observations to be added to this project anyway?
I've been wondering if we should simply invite some of those observers to join the project and see what they say?
Thanks for bringing this up - I ran into a few observers early on with this setting, but neglected to make a note of their usernames. In the states that I reviewed more recently, I started keeping a list, so that I can invite them to join. Is anyone else keeping track of users whom we should invite for this reason?
@kylejones, I don't know of any way to do that. However, I wonder if it might be possible to add a bit to your URL search string that will instead simply not show observations by that user. I'll look into it.
I played around with that search string but didn't get very far (anywhere). I'll send you a few usernames to consider inviting. I haven't kept a list, but remember a few.
I've gotten about 65 or so pages into Florida, working from oldest to newest. I'm not necessarily claiming Florida, though, because I'm not sure how quickly I would end up getting it done. Just an FYI if anyone does claim it, and I'll keep you posted if I make more progress on it.
Well, I did end up getting Florida done quickly -- it's complete.
Impressive!
I may have recruited a HS computer science teacher and students to help us with batch editing functions, etc. They may not get to it for a couple of weeks, and it looks like we are cranking away on East Texas, so maybe it will be to little avail, but it might be fun to see what they come up with!
Cool, thanks! Are people working on Texas? I was planning to break it up further into blocks of several counties, but won't bother if it's being managed otherwise.
Looks like @jtuttle has been busy. I've also been secretly adding Maryland.
Michigan, DC, & Oklahoma checked.
:-D I can't help myself...it's iNat therapy when I have (or can make) a few moments to spare...
I did a little on East Texas, and then a little on Dallas County, just testing, really -- I'll hold off for the more orderly approach! Please feel free to assign me a block when you've gotten a chance to do that.
Okay, I'll work on delineating some chunks of East Texas - hope to post those by sometime next week at the latest. In the meantime, here's a look at the distribution of Lep observations in East Texas, which I'm using to inform my (highly technical) chunk delineation.
Divisions of East Texas ready to be claimed!
@whodamanhd, are you still working on Maryland? kylejones went through it as well, so if you complete it we'll consider that a first pass & a check.
florida checked
North Carolina checked (whew!)
Started working on PA. Hope that's okay w @berkshirenaturalist !
Yes. Sorry. Had to take a break. Eyes were bugging out. I think I got to page 20 or so.
I totally understand. That's why I don't mess with Texas.
pennsylvania done.
We did it! A summary post is in the works, but I've been very busy with other things lately. In the meantime, if you would like to quickly review new observations in states & TX sections that you have already checked for potential caterpillars, I have built custom URLs for everyone who added from multiple areas, concatenating together all the PlaceIDs for states/sections that each person has either added or reviewed. Here they are:
berkshirenaturalist
eraskin
jtuttle
kylejones
psweet
If you would like to see the total number of observations that you have reviewed, just remove the "¬_in_project=caterpillars-of-the-eastern-us" section from the URL. Thanks again for all your help!
I went over 50,000 observations reviewed this morning, for Lepidoptera in the geographic area linked in my previous post. I'm sure some other project contributors have way more than this, particularly @kylejones. I'm still hoping to do some analyses & put up another journal post, now that we are done reviewing past observations. but time keeps getting away from me.
@kylejones, @jtuttle, have either of you noticed a sudden jump today in the number of observations that show up in your custom searches? I usually have anywhere from 3-10 pages of new Lepidoptera observations per day in my review area, but somehow between the morning and now more than 4,000 additional un-reviewed observations have appeared.
Wow, that's a leap! I'm not sure I'd notice (I use a different approach, because I don't want to mark as reviewed all the adult Lepidoptera observations as I page through looking for caterpillars -- I do like to browse and ID adult moths at other times. :-) ).
I've also changed my "procedures"--pretty ad hoc at this point, so I wouldn't notice changes like that.
Okay, good to know.
@eraskin , did you figure out what's going on? I think I am also noticing something weird. Over the past few days, when I've worked on adding observations to the project, I've noticed that there are some observations not already in the project where I've already annotated as "Larva". However, I do not do those annotations unless I am also adding observations to the project (or annotating observations already in the project). I tested Mississippi, which has relatively few pages of Lepidoptera observations, by going through all pages (a day or two ago) and adding/annotating any caterpillars that have accumulated since I last checked. Just now I rechecked Mississippi, and there are indeed some older caterpillar observations, which I know I added/annotated, showing up as not in the project. A couple of them show the same pattern of a Larva annotation by me, even though the observation is not in the project. I know I just went through all of these a day or two ago. In short, it seems like some observations are getting dumped out of the project or aren't really being added to the project. I'm not sure, but I don't think this is happening for all of the observations that I add/annotate. Weird, and I definitely don't want our recent efforts to be wasted/repeated many times because of some glitch...
I've noticed some observations will not let me use annotations to select a life stage. Here are a few examples:
https://www.inaturalist.org/observations/8239570
https://www.inaturalist.org/observations/7962918
https://www.inaturalist.org/observations/7784803
If I am using the full page edit, it just will not let me select "Larva". If I am using the multi-page edit (sorry I don't know the correct terminology), I can select larva but it just spins away and doesn't change it. Ironically, while I was typing this out it looks like @jtuttle was able to select larva for at least one of these.
Definitely something weird going on. I was hoping that it would get corrected after a few days, but it hasn't, and I've stalled out on adding new caterpillar observations. I'll see what I can find out.
Okay, a little bit hard to know what to ask about, given the weird, vague symptoms, but I put a post on the forum & we'll see what happens.
I'm on furlough for two weeks, so I'm going to brute-force my way through all the observations that suddenly appeared. Part of that big bubble seems to be made up of observations that don't have a photo, so eliminating those will simplify matters. If I get through that, maybe I'll go back to working on Canada, and maybe start working on Mexico.
FYI, I've already added observations for these states in Mexico: Yucatan, Quintana Roo, Campeche, and Tabasco.
Cool, thanks! I saw on the map that Mexico was starting to be populated.
I've continued to "test" adding observations from various places to the project, and it still appears that many of them do not "stick" -- they show up again within a couple of days as if they have not been added (and often, they have "larva" designated by one of us). @eraskin , if you're planning to spend time adding lots of observations to the project, I'm not sure that's recommended unless you are sure that they are going to really stay added to the project. It doesn't seem to be happening to all places or all observations that I add, but it's difficult to tell what the scope of the problem is.
Can you send me links to a few of the ones you find that seem not to have "stuck?"
Yes, good idea. I will try to get a couple of examples to you in the next day or two.
Here are three from New Brunswick, CA, one of the areas I "tested" recently:
https://www.inaturalist.org/observations/7625092
https://www.inaturalist.org/observations/8227505
https://www.inaturalist.org/observations/8227132
There are others, but these are obvious because they show that I added the "Larva" annotation. I also added the observations to the project, but they didn't "stick" (either didn't actually add, even though the observation page showed that they did, or were dumped out of the project after adding).
One possibility that occurs to me (though I'm not convinced it explains all of the observations that aren't adding properly): Perhaps iNat changed the way it indicates that an observer doesn't allow her/his observations to be added to projects s/he hasn't joined. I don't see the notification of this that I used to see, so maybe iNat removed the notification and gives the impression that an observation has been added, but it really hasn't been.
Hmm, I hadn't noticed, but I don't think I've gotten that message recently (can't add to project because of user settings). That's a good theory - I'll see if I can pin it down using some observations that I know I've been unable to add in the past. I just tried to add those three that you linked to, and again they seem not to have "stuck."
I think you figured it out. This seems like an oversight in the programming - I'll start a thread on the forum.
Okay, thanks! Fixing that might fix the entire problem, or it would at least fix that part of it so that we can tell whether or not there are other observations that won't add.
@eraskin and @kylejones , I noticed the observations that wouldn't accept the Larva attribute seem to have been cleared up, but now there are new ones. Did either of you figure out what was going on with those, or what causes this? Just curious...
Also, I get the impression when I use the annotation link you created that the bulk of the annotations got done, and now it's just a matter of keeping up with the new ones? If so, WOW!!! That was a huge task, and kudos to all who helped complete that (and sorry this comment is on the wrong post...).
@eraskin , very belated ;-) response to your custom URLs comment way above -- I haven't been able to get this to work. It pulls a single location (not necessarily one in the list, I think) rather than all locations. I think I have had this problem before, though I did manage to successfully make a multi-place_id URL for myself (for a different project) using the observations? code rather than the identify? code. Thoughts? I ask because I am only able to contribute to this sporadically, so I think a couple of subset links would work better for me.
Scratch that last post about custom URLs -- it works! I just had a brain glitch...
@jtuttle, glad you figured out your custom URL question, although I'd be happy to help with that if you run into issues in the future. I have not figured out anything about the ones that won't accept the Larva annotation - not sure I've run into that issue, although I remember that it came up once before. If you post a few problem observations as examples, I'll see what I can figure out (although last time it seems we were able to annotate observations that wouldn't take annotations for @kylejones).
Yes, the whole backlog got annotated, thanks in large part to a big push by @coatlicue & @kimberlietx. As you said, now it's just a matter of checking periodically to keep up with the new observations.
Just noticed that we have exceeded 30,000 observations!
I've run across several that were annotated but not in the project. I've been meaning to get back here to find out what was going on. Is that resolved?
@kimberlietx, that most often happens when someone has their user options set so that others can't add their observations to projects. See this message board post (which we probably need to bump again).
Ahhhh! And you don't realize it's not adding to the project, so you don't leave a msg. Gotcha!
Got a little behind in the last couple of weeks, but I'm back in the saddle & hope to catch up soon. Just reviewed my 100,000th Lepidoptera observation in the geographic area I'm reviewing for the project. Anyone else still steadily reviewing observations in search of caterpillars? Hopefully we will soon be able to use annotation-based rules, which should reduce the workload somewhat.
Hi, I haven't been reviewing caterpillar observations lately (except casually)...too swamped with work and other tasks! I'll be interested to see how well the annotation-bsed rules work.
Add a Comment