关于4月4日的宕机事件

2014年4月4日(什么倒霉日子……),凌晨2:30到9:00,服务器宕机了。

你们做了什么把服务器给宕掉的?

前段时间, 亚太二号的海底光缆出现故障了(现在也没有修复),大家应该能感受到,有些网站打开的速度受到了影响。这条光缆,同样也对FarBox的访问产生了一定影响。

虽然我们之前已经在开发NaNo Network,但毕竟是测试期,并不宜所有人都默认转入这个Network。

我们打算在DNS系统上,优化线路的选择。说到线路的选择,大家可能会浮现DNSPOD里面的多线路匹配。但这样的多线路匹配,涉及到中美线路,同时到中国之后,还有电信、联通、移动这些ISP的时候,网络的质量就变得很诡异了。

我们的优化逻辑很简单,当一个IP查询FarBox的DNS系统时,我们也会反向去查询连接这个IP的线路质量,然后下次查询DNS的时候,就会根据线路质量,匹配针对来路IP,速度最稳定的一条线路。

即使在海外的机房,仍然可以为中国大陆的访问,提供稳定的速度!

这样的课题,用一个简单直接的方式解决,我们还是很兴奋的,就动手去改造了。

这也能把服务器弄挂掉?

四月四日的凌晨,我们更新了代码,部署之后,运行完美。然后,小伙伴们就屁颠屁颠滚去睡觉了。

早上起来的时候,吓得一个激灵,服务器全线阵亡。

赶紧重启主服务器,暂时恢复正常。然后查询系统日志。

确定了基本的时间表:

  1. 2:32, 主服务器阵亡;
  2. 自动故障转移生效,第二台主服务器承担了所有的请求;
  3. 第二台主服务器虽然开始没有阵亡,但之前,已经处于半瘫痪状态;
  4. 4:20 之后,第二台主服务器没有扛住,彻底挂了。

为什么会宕机?

开始的时候,我们也百思不得其解。

因为新部署的功能是针对DNS系统的,不至于连累Web服务器才是;而且如果未能命中最佳线路的前提下,DNS服务器也能正常运转的。

在探测来路IP线路质量的时候,我们使用的是ping。这引入了一个包,接下来查看这个package源码的时候,发现,每次ping完之后,端口没有自动关闭。而DNS的查询, 是有大量的来路IP涌入。

然后,运行的时间一久,端口就会被消耗殆尽。一台服务器可用的端口也就6W左右,而端口是互联网建立连接的口子。

于是,就挂了。

还有更糟糕的。

时间一久,端口本身也会被强制释放的;所以,宕机之后,我们仍然能登录服务器。

如果只是端口耗尽,网路堵塞,我们应该很容易发现问题所在;还不至于被吓了个激灵。

真正吓到我们的是:FarBox相关的所有系统服务、程序全部阵亡!

在查了系统日志,发现是oom-killer干的好事。所谓的oom-killer就是内存溢出的时候,Linux系统的一种自我保护。所谓的自我保护,就是杀掉一些进程,空出一些内存来。(但我们是服务器,随便杀掉个进程,服务就不正常了,囧……)

我们两台服务器加起来有40G的内存,可想而知,我们凌晨升级的代码是多么开心+愉快,竟然统统吃光这些内存,引发了一系列的错误。

最后

我们停掉了新增的代码。

即使在海外的机房,仍然可以为中国大陆的访问,提供稳定的速度!

但这样的课题,仍然是很令人兴奋的。我们下次需要更慎重一些,这些代码或许改造后才会重新回归。

但起码不会是最近这段时间。

这样的宕机,是一个不可承受之重。

很抱歉, 让大家失望了。

@2014-04-04 15:43
  • yurenchen2015-06-17 11:32

    写的感同身受的

  • homepage2019-10-16 15:30

    I actually wanted to make a simple remark to be able to say thanks to you for these pleasant instructions you are giving out on this site. My incredibly long internet investigation has at the end of the day been recognized with pleasant suggestions to exchange with my companions. I would assume that many of us website visitors are unquestionably blessed to be in a magnificent website with many perfect individuals with insightful guidelines. I feel pretty lucky to have seen your entire webpages and look forward to plenty of more fabulous times reading here. Thanks again for a lot of things.

  • Papers-Land2020-03-25 20:29

    Hello guys, thanks for your contribution in that blog
    Are you terribly tired of doing endless assignments? Do you lack time to complete your projects according to the set deadlines? Do you desire to devote more time to your family and friends and, at the same time, get excellent grades? You have chosen the right website, as we provide a valuable college paper writing service. Our main aim is to assist our clients in composing fascinating pieces of writing.

  • write my essay2020-06-19 04:20

    Thanks for ones marvelous posting!

  • Online biology essay services have come up with Biology Essay Writing Services for biology research paper writing service students in order for them to score straight A’s in their custom biology research paper services.

  • The ultimate goal of online sociology research paper writing services is to provide Sociology Assignment Writing Services and sociology essay writing services since most sociology term paper writing service students lack time to complete their custom sociology coursework writing services.

  • Online psychology essay writing services are very difficult to complete and many students are always searching for Help with Psychology Coursework Writing services to help them complete their psychology research paper writing services and psychology case study writing services.

  • Finding the best healthcare assignment writing services and Medical Assignment Writing Services is not easy unless one is keen to establish a reliable custom medical research paper provider & medical writing service help online.

  • It is important for theology & religion writing service students to seek Religion & Theology Research Writing Services from a reputable theology & religion research paper service provider for their custom theology & religion assignment writing services.

  • Make my assignment2020-07-08 13:15

    Allhomeworkhelp if you are looking for good work then we provide the best high-quality essay and other Online assignment help services.
    https://www.allhomeworkhelp.co.uk

  • Allhomeworkhelp.co.uk2020-07-08 13:16

    Best Assignment Help Online has been a frequent provider of online services in the Allhomeworkhelp.co.uk and other countries for more than five years.