Derpibooru
Filters (Default) SettingsRegisterLogin

Maintenance 29-July-2013

Site and Policy » Maintenance 29-July-2013 Search Posts
Started by The Frowning Pony
Posted
5 replies
Subscribe
The Frowning Pony

Administrator
13 blinks per minute
News Tumblr
 
Basically, we have images missing from the data corruption, and images hidden from the DB desync of a few days earlier. The first problem we’ll fix by plugging in the images using another backup. The second problem requires a reindex. Neither of these can be done any time soon due to real life circumstances (ie. work), so don’t expect everything to be “back” until tomorrow.
 
As well, due to the way we had to re-set up the site, there’re a few nasty bugs, most notably the inability to upload any new images. That should be fixed today, but probably not for several hours.
Posted Report
The Frowning Pony

Administrator
13 blinks per minute
For those wanting a tl;dr of wtf happened, have some bullet points:
 
  1. Server handling arbitration, DNS and backups dies  
  2. Lack of proper arbitration causes data desync, hidden images  
  3. Backups temporarily set to a separate partition on cadance, the image server  
  4. New backup server goes online, but needs setting up  
  5. Datacenter where cadance lives suffers a network fault  
  6. cadance does not come back after network fault is resolved  
  7. Per host’s intervention guidelines, server is rebooted to ensure it’s not a software issue  
  8. Reboot triggers a file system check, as expected  
  9. Host tech notices a trivial fault with a remote control module and reboots the system during the check causing data corruption  
  10. After 8 hours on the phone with them insisting the machine was in perfect health and them looking at a login screen, we pay for KVM IP to have direct access to the machine  
  11. Machine is seen to obviously never have gotten anywhere near login; file system corrupted  
  12. Due to the gap in the backups, we begin very slow data recovery  
  13. To speed up the process, we order a USB drive for the server  
  14. After 18 hours, host plugs the drive in and reboots the server mid-data recovery, severely impacting the process  
  15. All data possible recovered between cadance and main backup, site goes back up  
  16. You are here
Posted Report
Clover the Clever
Princess of Love - Extra special version for those who participated in the Canterlot Wedding 10th anniversary event by contributing art.
A Perfectly Normal Pony - <@CloverTheClever> I'd pay to see Carcer in a fursuit
Always Codes Drunk - It explains a lot
From the Night - I have technically banned myself a bunch of times...
Since the Beginning - User number zero

Lord and Saviour
I can confirm that babies are employed by our hosts as server technicians. Possibly less confident than that fella appears to be. He at least knows which one’s the power cable. He’s even picked out the inlet.
 
Derpy’s far too competent, employing percussive maintenance like that.
Posted Report
Interested in advertising on Derpibooru? Click here for information!
Ministry of Image - Fanfiction Printing

Help fund the $15 daily operational cost of Derpibooru - support us financially!

Edit Preview
Syntax quick reference: **bold** *italic* ||hide text|| `code` __underline__ ~~strike~~ ^sup^ %sub%

Detailed syntax guide