About the August 2022 Unplanned iNaturalist Outage
On the night of August 26th (approximately 8 pm Pacific Daylight Time), a power outage impacted almost all of iNaturalist’s servers for several hours. iNaturalist rents servers from Microsoft Azure and the outage happened at Azure’s US West 2 data center.
As far as we can tell, no critical data was lost - that means photos, sound recordings, observations, identifications, projects, comments, and any other content uploaded to iNaturalist should all be there. You can read updates and discussion from the event on the iNaturalist Forum.
However, for anyone to be able to find all of the data, about a week’s worth of search indices needed to be rebuilt before we could bring iNaturalist back online, and that process - done almost entirely by @pleary - took about a day. Everything should be back as it was with the exception of notifications - some notifications generated in the past week may be lost, while others that you already viewed may appear again (we’re trying to err on the side of people getting them if they didn’t use iNat last week). In the next week we’ll be looking into how to prevent this situation from happening in the future.
We want to thank the iNaturalist community for the support it's shown us; on the iNaturalist Forum, on Twitter, and elsewhere during the past two days (and every day, really). We’re humbled that iNaturalist is an important part of so many lives, and we’re deeply sorry it was down for this long, especially if it was during a crucial event you had planned. We’re looking forward to seeing what you observed during the downtime!
Comments
Thanks to the staff for working so hard through the weekend to get the site up and running again! So glad to see iNat back.
And now we know we are addicted!
Thanks for all of the hard work and additional time spent in getting the site back up and running. It's very much appreciated.
Thank you to those who were oncall for this outage and helped bring iNaturalist back online! You're the SREs of iNaturalist and we appreciate you!! \o/
https://sre.google/sre-book/being-on-call/
Thanks to everyone who helped make it possible for us to use inaturalist again!
best regards
khd
Thanks all who worked so hard. Don't know where I would be without iNat and it seems so for many of us.
What a great community and an amazing bunch of researchers.
Back to work for me, a lifeline to the world of learning, sharing and searching out our natural world.
Thanks again.
Larney
Relieved to hear no data was lost due to the power outage. It serves to remind us all how valuable iNat is!
Not having access to iNat made me realise just how important it is to me. Thank you to everyone who helps deliver and maintain this magnificent service!
Fantastic to see iNat back online so soon! Thanks so much for all your hard work <3
muchas gracias to the guys & gals at iNat for your dedication to the cause. Very much appreciated indeed.
Muchas gracias por vuestro esfuerzo!!!
Sounds like a lot of hard work - well done everyone!
I sincerely appreciate the commitment and hard work that all the iNat team have provided over the last 30 or so hours. Thank you all so much - I bet you're all looking forward to some time off!
BRAVO à tous et un grand merci
Thank you so much for getting iNaturalist back up in all its glory again!
Making my annual donation early in appreciation!
Thank you
Thank you to our dedicated iNat team it means a lot to be able to use the platform again. It means a lot to so many of us. @pleary thanks for your hard work processing a week's worth of data and observations etc, we appreciate your skills and expertise being able to help get iNat back online again after the outage, and that you are part of out iNat community. Well done everyone, keep up the great work. Much appreciated from Karenweaving62
THANKYOU for providing such a wonderful and informative website, and for maintaining it. No doubt a LOT of work goes into this endeavor, we appreciate it
Many thanks for your wonderfull service and thankfully now ok.
Is there anything that can be learned from this experiance.
Is it possible this could happen again and if so what measures can be taken to prevent INat going down?
All the best
Rob
Thank you inat team - you are awesome!
Thanks so much to everyone who worked tirelessly this Friday night and Saturday
to bring iNaturalist back to life! I am glad I was able to use the iNat app in offline
mode during the outage - the only thing that was missing was the AI integration
which provides the suggestions. Keep up the great work y'all!
So glad you are back up. iNaturalist is such an important resource for science as well as a great tool for education. I really appreciated the efforts made to let people know what was happening. I saw it on Facebook myself. Thank you for providing this amazing platform and keeping it running.
Thank you for the update and for your continued work to make iNaturalist the place that it is!
Thank you so much everyone who worked so hard to bring iNat back to us so quickly and thoroughly. A reminder for me that I couldn't do what I do without you behind the scenes!
I just made an extra donation to show my appreciation. - Jean
Thank you iNat team! 👏
What a beautiful community! 😀
Wow, thanks so much for your fast work putting this back together! In the meantime, the experience offered some much needed perspective, both about how much I value this community and that it's ok for one flower to go unrecorded. Maybe.
Going to record those now... :)
THANK YOU so much for this great community and service. Donate, everyone! It's the best way of all to say thank you.
We are so glad that INat is back! I was so afraid that some data could be damaged or lost. Thank you very much! This effort was and is very important for us. :)
Kudos to the brilliant work of the staffs
Thank you!!
Many thanks to the very dedicated staff!
Thank you everyone! So nice to have iNat back again!
Wonderful news! Thank you so much Patrick and staff. Your effort is very much appreciated, we have a bioblitz going on today! Enjoy your sunday (and rest)
Thanks for all your work to restore iNaturalist! I really like the app and website, and I always learn something new.
At the same time, I have a few questions:
Is it possible to diversify server locations and/or providers? The Azure West US 2 location is in Washington State (see https://azure.microsoft.com/en-us/global-infrastructure/data-residency/#overview). The US Northwest is subject to hazards like earthquakes, wildfires, and even extreme heat (alas, the latter two are becoming more likely due to human-caused climate change). It would be good to have a backup facility in a different part of the world (or at least a different part of the US).
Why are search indices so ephemeral that a power outage requires completely rebuilding the indices? Do they only exist in server RAM or temporary memory? Can they be saved every few minutes or hours to a file on hard disk or SSD, and backed up regularly, so that only the last few records would need to be indexed upon system restart?
I was surprized to learn that iNaturalist uses Microsoft Azure at all. I had thought that Amazon AWS is the hosting platform, due to the iNaturalist campaign in 2021 to align user observation copyright licenses with Amazon's requirement for free hosting - see https://www.inaturalist.org/blog/48165-we-want-you-to-license-your-inaturalist-photos-before-april-15th
Sorry if I missed something, but has iNaturalist ever listed all the services and companies on which it depends? If so, then where is the list? If not, then can you please publish this list and keep it updated to improve transparency and align community expectations?
Speaking of hazards and data centres, does iNaruralist require or at least encourage its providers to use renewable energy for the servers and their cooling? As iNaturalist accumulates and transmits ever more high resolution images, minimizing server carbon footprint would help reduce emissions from fossil fuel used to generate electricity, helping to slow down climate change and its devastating effects on humans and biodiversity.
"Large technology companies such as Google, Microsoft, and Amazon have all been investing in renewable energy projects. But experts say it is hard to monitor how effective this transition is because the leading technology companies are large, opaque, and largely unregulated." Source: https://yaleclimateconnections.org/2021/02/the-internets-big-carbon-footprint-need-not-doom-the-climate/
Transparency and accountability would reduce opaqueness and encourage a real shift to more sustainable computing. Is there anything further that iNaturalist can do to help?
Thanks in advance for answering these four questions, and keep up the good work on iNaturalist!
At a mushroom foray yesterday without iNat, we had to go back to the olden days and hit the books to ID our fungi! Oh boy, did we miss you, iNat! Thank you so much for your hard work to recover!
Thanks everyone, for getting iNaturalist back up. Much appreciated.
Just to note: I processed 880 comments/IDs on Friday (GMT+1), and so was ready to ditch them when after the indexing today it showed 674 comments/IDs - but it turns out that 70% of these (ignoring today's few dozen - note in Edit: 70% of half: the second page had all been processed) were not in the 880 that I processed on Friday,
It seems that at least I have benefited by the system going down?
Thank you to @pleary for doing an extreme amount of unexpected work over the weekend. I do not really understand what happened (nor do I need to), but my husband is a UNIX Sys Admin / architect of 25 years and he kept twisting his face up and making horrible little gasping sounds as I read him the updates so I knew it was bad.
Thank you again to @pleary and all involved. I so appreciate you.
After reading the forum post late last night, I was surprised to see that iNat was already back up this morning. I was expecting another day of checking the site every two hours to see if it was back. What a tremendous present from the staff, particularly @pleary, for all the effort.
We don't know how much we use Inaturalist until it's gone! Thanks for all the work.
Adding my voice to the chorus of appreciation for the hard work put in this weekend. Thank you!!!
Great response by everyone involved.
Woo Hoo! Thanks for all the extra work to get this up and running! Most appreciated!
Thanks so much Inat staff for making this wonderful website come back online. I was also very nervous that my photos would be lost, especially since I can never recover them once they are deleted.
I’m grateful to everyone who has been working for Inat, as they go above and beyond expectations.
Dear iNat Team,
Thank you , Thank you very much.
I sure it must have been nerve wracking ,
I have used the site a bit and it seems all good.
Thank you from a very grateful inatter
ram
I don't know what the iNat team schedule is -- but am sure you will indulge in some much needed R & R. (rest and recreation).
Thank you so much for creating and maintaining this platform. Your dedication is amazing.
Thank you for everything you do!
So happy to see that it's back! Thank you for all your hard work to get everything up and running again! Cheers, Andrea
Thank you.
Greatly appreciate the hard work put in by everyone involved to get it up and running again.
Every time I log in to INat you say welcome back so now it is my turn: Welcome back INaturalist!.....and thanks for all the hard work you all did to be back, Cheers Bart
I was suffering from iNaturalist Withdrawal Syndrome, and thankfully, @pleary knew the cure. THANK YOU, Patrick!
Great to have you all back, and thanks for all the work and expertise that got it up and running again.
Thank you iNat team for getting the site up and running again and creating such a meaningful platform!! I’ll be increasing my monthly donation and hope others will too. 🍄🌲🐸🐥🪱🐛🪲🕷🦋🐳🦎
I am working with a team of volunteers in New Mexico helping NMSU update their Master Gardeners online curriculum. My focus is in plant Identification. I am recommending formally adding INaturalist to what is currently a traditional methodology of learning plant families, using keys, and recognizing physical characteristics to make IDs. Your temporary outage reminded me of how much this world has changed and how much iNaturalist contributes.
thank you. We appreciate even more what we missed not having ....
Hmmmm - sounds similar to the whole point of nature itself - perhaps a lesson not take it for granted either.
A huge thanks to all the staff that worked to resolve this, keep us up to date etc. We are really grateful. And it is nice to see how supportive and patient all the iNat users are! Lovely to be part of this community.
MicrosoftFail
SwitchToApple
:-D
Thank you Staff for all you do with iNat!
28 Aug 2022 - To the iNaturalist tech team, huge thanks for this update. This morning, I held my breath and opened my iNat native app on iOS and…. RELIEF! You had pulled this rabbit out of the hat and everything had been put to rights. It must have been a challenging few days. I hope you all celebrate this achievement… Thanks again!
Thank you thank you thank you for all of your incredible work!
So many comments so I'll just add, great job!
There's a chorus of thank-yous, appreciation and kudos, and I wholeheartedly add my voice as one of the iNat addicts. who is grateful for your expertise and the resources you provide. Thank you so much for keeping us apprised of the situation and resolving it.
Muchas felicidades por este logro tan importante, mis felicitaciones a todo el equipo!!
Thank you guys for the hard work! You guys deserve the best of love.
Thank you for your hard work. I my earlier life, I have been there, done that, and know that as a result I missed out on family and life. We appreciate your work. I also now realize how much I depend on the iNat community.
Wow, power outage eh? Does Microsoft not believe in backup power systems?
Adding my thanks to all the Staff for getting iNat back on its feet so quickly. Job qualification for all new hires: must demonstrate dedication to iNaturalist equaling or exceeding that of current staff!!
Love you iNat staff! Thank you guys for working hard to keep the site so great!
I'd like to add my very big thank you - well done all, especially @pleary! A reminder on how much I depend on iNat and not to
take all the staff efforts and their hard work for granted but to appreciate them even more. Thank you guys!
A hundred thousand thank you's for all of the hard work --- especially what I'm sure must have been into the wee hours!
Just want to add a quick thank you for everything you do!
Thank you so so much! Much appreciation from NZ!
Muchísmas gracias por vuestro trabajo para sacar adelante esta maravillosa plataforma !!!!!!
Thank you so very much for your work! The more so because iNat was back just before I was to give a lecture on citizen science in biodiverity research :-)
Phew!! Thanks @pleary and all the team.
Thank you iNat staff!
Thank you!
Thanks for all the hard work, iNat team!
Thank you. What a relief to have iNaturalist back!
Thank you @pleary! Your work is much appreciated!
Thank you to the iNat team.
I had to wait…it seemed like forever.
What a relief to have iNat back. Thanks for your great work, dev team!
Special thanks to @pleary for all his hard work! So glad that iNaturalist is back online!
Thank you for all your hard work to keep this thing going, it's super important.
Thanks everyone for your patience during the downtime, and appreciation of our staff as we worked through the recovery process. Fortunately, aside from some notifications, no data was lost and the recovery process went smoothly albeit a little slow due to how large iNaturalist has become. Things could have been worse, but they also could have been better, and we're definitely going to learn from what happened to improve things like backup and recovery strategies.
Some changes have already been put into place. During this event we needed to re-index 7 days of data, and going forward we expect to always have a copy of indices from at most 24 hours ago. That won't mean recovery from a problem like this in the future will be 7 times as fast, but it will be faster and fewer notifications will potentially be lost. We will be exploring other options for high-availability of our systems through more redundancy within our service provider so we're less effected should an entire data center go down. We're also working on ways to speed up indexing to further minimize downtime.
This is the worst cloud provider failure I've experienced in my 8 years at iNaturalist. Hopefully events like this remain rare, but we're not taking that for granted. I'll note we've been working to grow the team here through some open job postings (https://www.inaturalist.org/pages/jobs). We're in a better place now than we were on Friday and we'll keep that up as we grow the team and improve backup, recovery, and redundancy strategies.
Finally - thanks to you all again for being patient and having a generally very positive attitude towards this. It was a busy weekend of unexpected work, but the work is much easier when there's not a lot of outside pressure and you know it will be appreciated. The iNaturalist community is really exceptional and it's all the more rewarding to help keep the platform running to see the amazing things you all do with it.
As @quokka2 mentioned, observing, identifying, researching is an addiction and now we see iNat has become quite a important part of our daily lifes. :) Thanks for your work, one weekend out of order is nothing compared to the services you provide us the whole time.
I am so glad that iNaturalist is back online. The ability to identify all of these wonderful creatures has given me a greater appreciation of the variety of wildlife in our world. I always look forward to finding something new to me and being able to contribute to the greater knowledge. Thanks for your efforts in getting back up!
There was a blank void in my life. Thank God it was only temporary and thank you for all the hard work to make this community accessible .
I think it was good to take a break from technology
And yet another thank you! Patrick Leary, great job on rebuilding the indices. Thank you all for your hard work.
Thanks! And no worries. Downtimes happen to many fancy banks and large companies which have IT armies at their disposal. You may wish to consider other cloud providers or an hybrid cloud approach.
For how long will I continue to receive zombie notifications? A week out and it's still happening.
Add a Comment