关于4月4日的宕机事件

2014年4月4日(什么倒霉日子……),凌晨2:30到9:00,服务器宕机了。

你们做了什么把服务器给宕掉的?

前段时间, 亚太二号的海底光缆出现故障了(现在也没有修复),大家应该能感受到,有些网站打开的速度受到了影响。这条光缆,同样也对FarBox的访问产生了一定影响。

虽然我们之前已经在开发NaNo Network,但毕竟是测试期,并不宜所有人都默认转入这个Network。

我们打算在DNS系统上,优化线路的选择。说到线路的选择,大家可能会浮现DNSPOD里面的多线路匹配。但这样的多线路匹配,涉及到中美线路,同时到中国之后,还有电信、联通、移动这些ISP的时候,网络的质量就变得很诡异了。

我们的优化逻辑很简单,当一个IP查询FarBox的DNS系统时,我们也会反向去查询连接这个IP的线路质量,然后下次查询DNS的时候,就会根据线路质量,匹配针对来路IP,速度最稳定的一条线路。

即使在海外的机房,仍然可以为中国大陆的访问,提供稳定的速度!

这样的课题,用一个简单直接的方式解决,我们还是很兴奋的,就动手去改造了。

这也能把服务器弄挂掉?

四月四日的凌晨,我们更新了代码,部署之后,运行完美。然后,小伙伴们就屁颠屁颠滚去睡觉了。

早上起来的时候,吓得一个激灵,服务器全线阵亡。

赶紧重启主服务器,暂时恢复正常。然后查询系统日志。

确定了基本的时间表:

  1. 2:32, 主服务器阵亡;
  2. 自动故障转移生效,第二台主服务器承担了所有的请求;
  3. 第二台主服务器虽然开始没有阵亡,但之前,已经处于半瘫痪状态;
  4. 4:20 之后,第二台主服务器没有扛住,彻底挂了。

为什么会宕机?

开始的时候,我们也百思不得其解。

因为新部署的功能是针对DNS系统的,不至于连累Web服务器才是;而且如果未能命中最佳线路的前提下,DNS服务器也能正常运转的。

在探测来路IP线路质量的时候,我们使用的是ping。这引入了一个包,接下来查看这个package源码的时候,发现,每次ping完之后,端口没有自动关闭。而DNS的查询, 是有大量的来路IP涌入。

然后,运行的时间一久,端口就会被消耗殆尽。一台服务器可用的端口也就6W左右,而端口是互联网建立连接的口子。

于是,就挂了。

还有更糟糕的。

时间一久,端口本身也会被强制释放的;所以,宕机之后,我们仍然能登录服务器。

如果只是端口耗尽,网路堵塞,我们应该很容易发现问题所在;还不至于被吓了个激灵。

真正吓到我们的是:FarBox相关的所有系统服务、程序全部阵亡!

在查了系统日志,发现是oom-killer干的好事。所谓的oom-killer就是内存溢出的时候,Linux系统的一种自我保护。所谓的自我保护,就是杀掉一些进程,空出一些内存来。(但我们是服务器,随便杀掉个进程,服务就不正常了,囧……)

我们两台服务器加起来有40G的内存,可想而知,我们凌晨升级的代码是多么开心+愉快,竟然统统吃光这些内存,引发了一系列的错误。

最后

我们停掉了新增的代码。

即使在海外的机房,仍然可以为中国大陆的访问,提供稳定的速度!

但这样的课题,仍然是很令人兴奋的。我们下次需要更慎重一些,这些代码或许改造后才会重新回归。

但起码不会是最近这段时间。

这样的宕机,是一个不可承受之重。

很抱歉, 让大家失望了。

@2014-04-04 15:43
  • yurenchen2015-06-17 11:32

    写的感同身受的

  • homepage2019-10-16 15:30

    I actually wanted to make a simple remark to be able to say thanks to you for these pleasant instructions you are giving out on this site. My incredibly long internet investigation has at the end of the day been recognized with pleasant suggestions to exchange with my companions. I would assume that many of us website visitors are unquestionably blessed to be in a magnificent website with many perfect individuals with insightful guidelines. I feel pretty lucky to have seen your entire webpages and look forward to plenty of more fabulous times reading here. Thanks again for a lot of things.

  • Papers-Land2020-03-25 20:29

    Hello guys, thanks for your contribution in that blog
    Are you terribly tired of doing endless assignments? Do you lack time to complete your projects according to the set deadlines? Do you desire to devote more time to your family and friends and, at the same time, get excellent grades? You have chosen the right website, as we provide a valuable college paper writing service. Our main aim is to assist our clients in composing fascinating pieces of writing.

  • write my essay2020-06-19 04:20

    Thanks for ones marvelous posting!

  • Online biology essay services have come up with Biology Essay Writing Services for biology research paper writing service students in order for them to score straight A’s in their custom biology research paper services.

  • The ultimate goal of online sociology research paper writing services is to provide Sociology Assignment Writing Services and sociology essay writing services since most sociology term paper writing service students lack time to complete their custom sociology coursework writing services.

  • Online psychology essay writing services are very difficult to complete and many students are always searching for Help with Psychology Coursework Writing services to help them complete their psychology research paper writing services and psychology case study writing services.

  • Finding the best healthcare assignment writing services and Medical Assignment Writing Services is not easy unless one is keen to establish a reliable custom medical research paper provider & medical writing service help online.

  • It is important for theology & religion writing service students to seek Religion & Theology Research Writing Services from a reputable theology & religion research paper service provider for their custom theology & religion assignment writing services.

  • Make my assignment2020-07-08 13:15

    Allhomeworkhelp if you are looking for good work then we provide the best high-quality essay and other Online assignment help services.
    https://www.allhomeworkhelp.co.uk

  • Allhomeworkhelp.co.uk2020-07-08 13:16

    Best Assignment Help Online has been a frequent provider of online services in the Allhomeworkhelp.co.uk and other countries for more than five years.

  • essay writer websites2020-12-20 09:34

    Very energetic article, I loved that bit. Will there bbe a part
    2?
    https://yescheapessaywriting.com
    essay writer websites
    essay writer websites
    https://yescheapessaywriting.com

  • written assignment2020-12-25 09:01

    Thanks for your personal marvelous posting! I definitely enjoyed
    reading it, you could be a great author.I will bee sure to bookmark your blog and will oftsn come bafk
    sometime soon. I want to encourage you to continue your great writing, have a nice evening!

    https://huntleesq2he6g5v.medium.com/
    written assignment
    written assignment

  • write papers for me2020-12-31 09:46

    I every time used too study piece of writing in news papers but now as I am a user oof internet therefore from
    now I amm using net for content, thanks to web.
    https://foracademicwriting.com
    write papers for me
    write papers for me
    https://foracademicwriting.com

  • college essay writing2020-12-31 14:07

    I rally like ypur blog.. very nice colors & theme.
    Did you create this website yourself or did you hire someone to ddo it for you?
    Plz answer back as I'm looking to construct my own blog and would
    like to find out where u got this from. kudos
    https://ordertermpaperonline.com
    college essay writing
    college essay writing
    https://ordertermpaperonline.com

  • custom essay writing2020-12-31 19:11

    Whatt i don't understood is in fact how you're now not really much
    more smartly-liked than you might bee now. You're very intelligent.
    You recognize therefore significantly when it comes to this subject,
    made me for my part believe it from so many various angles.
    Its like women and men aren't fascinated unles it's someething to do with Woman gaga!

    Your individual stuffs great. Always handfle it up!

    https://topessaystore.com
    custgom essay writing
    custom essay writing
    https://topessaystore.com

  • write a essay online2020-12-31 19:32

    WOW just what I was searching for. Camme here by searching for essay paper
    https://servicesdigitalwriters.com
    wriite a essay online
    write a essay online
    https://servicesdigitalwriters.com

  • I like the valuabpe information youu provide in your articles.
    I will bookmark your blog and check once more here regularly.
    I'm quite suree I'll learn plenty of new stuff right right here!
    Good luck for thhe following!
    https://researchpaperorder.com
    cheap research paper writing
    cheap research paper writing
    https://researchpaperorder.com

  • paper writing service2020-12-31 19:45

    Heey There. I discovered your weblog using msn.
    This is a very neatly written article. I will be sure to bookmark it and
    retuen to learn more of your useful information. Thanks for the post.
    I'll certainly comeback.
    https://servicethesis.com
    paper writing service
    paper writing service
    https://servicethesis.com

  • I was curious if you evger thought of changing the page layout
    of your website? Its very weell written; I love what youve got to say.But maybe you could
    a little more in tthe way of content so people could connect with
    it better. Youve got an aawful lot of text for only having
    onne or two images. Maybe you could space it out better?
    https://orderessayonlineuk.com
    best paper writing service
    best paper writing service
    https://orderessayonlineuk.com

  • This blog was... how do you say it? Relevant!!
    Finally I have found something that helped me. Many thanks!

    https://globalwritinganessay.com
    top rated essay writing service
    top rayed essay writing service
    https://globalwritinganessay.com

  • buy custom essay online2020-12-31 20:52

    Does your blog have a contact page? I'm having problems locating it but, I'd lie tto send you an e-mail.
    I've got some ideas for your blo you might be interested in hearing.
    Either way, great site andd I look forward to seeing it improve over time.

    https://goodpaperwritingservices.com
    buy custom essay online
    buy custom esssy online
    https://goodpaperwritingservices.com

  • This iss a good tip particularly to those neew to the blogosphere.

    Shortt but very prtecise info… Many thanks forr sharing this one.

    A must rea post!
    https://essaywritingtld.com
    cheapest essqy writing service
    cheapest essay writing service
    https://essaywritingtld.com

  • buy essay writing2020-12-31 21:18

    Hello, I think your website might be having browser compatibility
    issues. When I look at your blog site in Chrome, it looks fie but when opening in Internet Explorer,
    it haas some overlapping. I just wanted to give you a quick heads up!
    Othger then that, terrific blog!
    https://essayserviceusukua.com
    buy essay writing
    buy essay writing
    https://essayserviceusukua.com

  • I'm gone to convey my little brother, that he should
    also go tto see this website on regular basis too get updated from
    latest information.
    https://essaygood.com
    essay writing elp service
    essay writing help service
    https://essaygood.com

  • Do you mind if I quote a couple of yoyr artticles as long as I provide credit and soufces back to your
    site? My blog site is in the exact same niche aas yours and
    my users would truly benefit from some of the informjation you provide here.
    Pleaxe let me know if this alright with you.
    Thanks a lot!
    https://essayhelpforall.com
    best paper writing services
    best paper writing services
    https://essayhelpforall.com

  • website to write essays2020-12-31 23:38

    Good write-up. I certainly love this website.
    Keep writing!
    https://buyessayonlinereviews.com
    webbsite to write essays
    website to write essays
    https://buyessayonlinereviews.com

  • online paper2021-01-01 00:03

    Really when someone doesn't understand afterward its up to other visitors that they will help, so here itt happens.

    https://amazingcustompaperwritingservice.com
    online paper
    online paper
    https://amazingcustompaperwritingservice.com

  • writing service2021-01-01 04:51

    What's up, after reading this amazing paragraph i am too delighted to share myy familiarity
    here with colleagues.
    https://mywritingexperience.com
    writing service
    writing service
    https://mywritingexperience.com

  • writing service2021-01-11 19:26

    You are so interesting! I do not think I've truly read
    something like this before. So wonderful to find another person with a few original thoughts on this subject matter.
    Really.. many thanks for starting this up. This website is something
    that is require on the internet, someone with a bit off originality!

    https://internationaltopwriting.com
    writing service
    writing service
    https://internationaltopwriting.com

  • You ought to be a part of a conhtest for one of the most useful websites
    on the web. I am going to recommend this site!
    https://webtheessaywriter.com
    college essay writing service reviews
    college essay writing service reviews
    https://webtheessaywriter.com

  • Magnificent goods from you, man. I have understand your stuff previous to
    and yoou are just too excellent. I really like
    what you've acauired here, crtainly like what you are sayibg and the
    way in which you say it. You make itt entertaining and you still care
    for to keep it smart. I can't wait to read muhch ore from you.
    This is actually a tremendous web site.
    https://dolf097z5tjo.medium.com/
    custom dissertation writers
    custom dissertation writers

  • 1v1 lol2021-02-04 14:37

    I really like your content because it is very informative and knowledgeable content.

  • slope unblocked2021-02-04 14:42

    Thank you very much for sharing this useful information. I was doing a project and for that, I was looking for related information. Some of the points are very useful. Do share some more material if you have.

  • Raleigh2021-03-08 12:39

    Hi there are using Wordpress for your site platform?
    I'm new to the blog world but I'm trying to get started and set up
    my own. Do you need any html coding knowledge to
    make your own blog? Any help would be greatly appreciated!

  • lo1maf222021-03-09 18:03

    This is a topic which is near to my heart... Cheers!
    Where are your contact details though?

  • Rozella2021-03-10 12:10

    If you want to improve your experience only keep
    visiting this site and be updated with the most up-to-date information posted here.

  • see here now2021-03-13 22:36

    Wow, that's what I was looking for, what a data!
    present here at this blog, thanks admin of this web page.

  • directory2021-03-14 17:21

    I absolutely love your blog.. Very nice colors & theme. Did you develop this amazing site yourself?
    Please reply back as I'm wanting to create my very own website and
    would love to learn where you got this from or just what the theme is called.
    Kudos!

  • 0mniartist2021-04-08 05:41

    Heya i'm for the first time here. I found this board and I to find It truly helpful
    & it helped me out much. I am hoping to present
    something back and help others such as you helped me.
    0mniartist asmr

  • free dating sites
    Having read this I believed it was rather enlightening.
    I appreciate you finding the time and energy to put this information together.
    I once again find myself personally spending way too much time both reading and commenting.
    But so what, it was still worthwhile! https://785days.tumblr.com/ dating sites

  • dating sites
    Just wish to say your article is as amazing. The clearness on your post is
    just cool and i could suppose you're an expert in this subject.
    Well with your permission allow me to grasp your feed to stay updated with
    drawing close post. Thanks a million and please carry
    on the gratifying work. free dating sites