关于4月4日的宕机事件

2014年4月4日(什么倒霉日子……),凌晨2:30到9:00,服务器宕机了。

你们做了什么把服务器给宕掉的?

前段时间, 亚太二号的海底光缆出现故障了(现在也没有修复),大家应该能感受到,有些网站打开的速度受到了影响。这条光缆,同样也对FarBox的访问产生了一定影响。

虽然我们之前已经在开发NaNo Network,但毕竟是测试期,并不宜所有人都默认转入这个Network。

我们打算在DNS系统上,优化线路的选择。说到线路的选择,大家可能会浮现DNSPOD里面的多线路匹配。但这样的多线路匹配,涉及到中美线路,同时到中国之后,还有电信、联通、移动这些ISP的时候,网络的质量就变得很诡异了。

我们的优化逻辑很简单,当一个IP查询FarBox的DNS系统时,我们也会反向去查询连接这个IP的线路质量,然后下次查询DNS的时候,就会根据线路质量,匹配针对来路IP,速度最稳定的一条线路。

即使在海外的机房,仍然可以为中国大陆的访问,提供稳定的速度!

这样的课题,用一个简单直接的方式解决,我们还是很兴奋的,就动手去改造了。

这也能把服务器弄挂掉?

四月四日的凌晨,我们更新了代码,部署之后,运行完美。然后,小伙伴们就屁颠屁颠滚去睡觉了。

早上起来的时候,吓得一个激灵,服务器全线阵亡。

赶紧重启主服务器,暂时恢复正常。然后查询系统日志。

确定了基本的时间表:

  1. 2:32, 主服务器阵亡;
  2. 自动故障转移生效,第二台主服务器承担了所有的请求;
  3. 第二台主服务器虽然开始没有阵亡,但之前,已经处于半瘫痪状态;
  4. 4:20 之后,第二台主服务器没有扛住,彻底挂了。

为什么会宕机?

开始的时候,我们也百思不得其解。

因为新部署的功能是针对DNS系统的,不至于连累Web服务器才是;而且如果未能命中最佳线路的前提下,DNS服务器也能正常运转的。

在探测来路IP线路质量的时候,我们使用的是ping。这引入了一个包,接下来查看这个package源码的时候,发现,每次ping完之后,端口没有自动关闭。而DNS的查询, 是有大量的来路IP涌入。

然后,运行的时间一久,端口就会被消耗殆尽。一台服务器可用的端口也就6W左右,而端口是互联网建立连接的口子。

于是,就挂了。

还有更糟糕的。

时间一久,端口本身也会被强制释放的;所以,宕机之后,我们仍然能登录服务器。

如果只是端口耗尽,网路堵塞,我们应该很容易发现问题所在;还不至于被吓了个激灵。

真正吓到我们的是:FarBox相关的所有系统服务、程序全部阵亡!

在查了系统日志,发现是oom-killer干的好事。所谓的oom-killer就是内存溢出的时候,Linux系统的一种自我保护。所谓的自我保护,就是杀掉一些进程,空出一些内存来。(但我们是服务器,随便杀掉个进程,服务就不正常了,囧……)

我们两台服务器加起来有40G的内存,可想而知,我们凌晨升级的代码是多么开心+愉快,竟然统统吃光这些内存,引发了一系列的错误。

最后

我们停掉了新增的代码。

即使在海外的机房,仍然可以为中国大陆的访问,提供稳定的速度!

但这样的课题,仍然是很令人兴奋的。我们下次需要更慎重一些,这些代码或许改造后才会重新回归。

但起码不会是最近这段时间。

这样的宕机,是一个不可承受之重。

很抱歉, 让大家失望了。

@2014-04-04 15:43
  • yurenchen2015-06-17 11:32

    写的感同身受的

  • homepage2019-10-16 15:30

    I actually wanted to make a simple remark to be able to say thanks to you for these pleasant instructions you are giving out on this site. My incredibly long internet investigation has at the end of the day been recognized with pleasant suggestions to exchange with my companions. I would assume that many of us website visitors are unquestionably blessed to be in a magnificent website with many perfect individuals with insightful guidelines. I feel pretty lucky to have seen your entire webpages and look forward to plenty of more fabulous times reading here. Thanks again for a lot of things.

  • Papers-Land2020-03-25 20:29

    Hello guys, thanks for your contribution in that blog
    Are you terribly tired of doing endless assignments? Do you lack time to complete your projects according to the set deadlines? Do you desire to devote more time to your family and friends and, at the same time, get excellent grades? You have chosen the right website, as we provide a valuable college paper writing service. Our main aim is to assist our clients in composing fascinating pieces of writing.

  • write my essay2020-06-19 04:20

    Thanks for ones marvelous posting!

  • Online biology essay services have come up with Biology Essay Writing Services for biology research paper writing service students in order for them to score straight A’s in their custom biology research paper services.

  • The ultimate goal of online sociology research paper writing services is to provide Sociology Assignment Writing Services and sociology essay writing services since most sociology term paper writing service students lack time to complete their custom sociology coursework writing services.

  • Online psychology essay writing services are very difficult to complete and many students are always searching for Help with Psychology Coursework Writing services to help them complete their psychology research paper writing services and psychology case study writing services.

  • Finding the best healthcare assignment writing services and Medical Assignment Writing Services is not easy unless one is keen to establish a reliable custom medical research paper provider & medical writing service help online.

  • It is important for theology & religion writing service students to seek Religion & Theology Research Writing Services from a reputable theology & religion research paper service provider for their custom theology & religion assignment writing services.

  • Make my assignment2020-07-08 13:15

    Allhomeworkhelp if you are looking for good work then we provide the best high-quality essay and other Online assignment help services.
    https://www.allhomeworkhelp.co.uk

  • Allhomeworkhelp.co.uk2020-07-08 13:16

    Best Assignment Help Online has been a frequent provider of online services in the Allhomeworkhelp.co.uk and other countries for more than five years.

  • essay writer websites2020-12-20 09:34

    Very energetic article, I loved that bit. Will there bbe a part
    2?
    https://yescheapessaywriting.com
    essay writer websites
    essay writer websites
    https://yescheapessaywriting.com

  • written assignment2020-12-25 09:01

    Thanks for your personal marvelous posting! I definitely enjoyed
    reading it, you could be a great author.I will bee sure to bookmark your blog and will oftsn come bafk
    sometime soon. I want to encourage you to continue your great writing, have a nice evening!

    https://huntleesq2he6g5v.medium.com/
    written assignment
    written assignment

  • write papers for me2020-12-31 09:46

    I every time used too study piece of writing in news papers but now as I am a user oof internet therefore from
    now I amm using net for content, thanks to web.
    https://foracademicwriting.com
    write papers for me
    write papers for me
    https://foracademicwriting.com

  • college essay writing2020-12-31 14:07

    I rally like ypur blog.. very nice colors & theme.
    Did you create this website yourself or did you hire someone to ddo it for you?
    Plz answer back as I'm looking to construct my own blog and would
    like to find out where u got this from. kudos
    https://ordertermpaperonline.com
    college essay writing
    college essay writing
    https://ordertermpaperonline.com

  • custom essay writing2020-12-31 19:11

    Whatt i don't understood is in fact how you're now not really much
    more smartly-liked than you might bee now. You're very intelligent.
    You recognize therefore significantly when it comes to this subject,
    made me for my part believe it from so many various angles.
    Its like women and men aren't fascinated unles it's someething to do with Woman gaga!

    Your individual stuffs great. Always handfle it up!

    https://topessaystore.com
    custgom essay writing
    custom essay writing
    https://topessaystore.com

  • write a essay online2020-12-31 19:32

    WOW just what I was searching for. Camme here by searching for essay paper
    https://servicesdigitalwriters.com
    wriite a essay online
    write a essay online
    https://servicesdigitalwriters.com

  • I like the valuabpe information youu provide in your articles.
    I will bookmark your blog and check once more here regularly.
    I'm quite suree I'll learn plenty of new stuff right right here!
    Good luck for thhe following!
    https://researchpaperorder.com
    cheap research paper writing
    cheap research paper writing
    https://researchpaperorder.com

  • paper writing service2020-12-31 19:45

    Heey There. I discovered your weblog using msn.
    This is a very neatly written article. I will be sure to bookmark it and
    retuen to learn more of your useful information. Thanks for the post.
    I'll certainly comeback.
    https://servicethesis.com
    paper writing service
    paper writing service
    https://servicethesis.com

  • I was curious if you evger thought of changing the page layout
    of your website? Its very weell written; I love what youve got to say.But maybe you could
    a little more in tthe way of content so people could connect with
    it better. Youve got an aawful lot of text for only having
    onne or two images. Maybe you could space it out better?
    https://orderessayonlineuk.com
    best paper writing service
    best paper writing service
    https://orderessayonlineuk.com

  • This blog was... how do you say it? Relevant!!
    Finally I have found something that helped me. Many thanks!

    https://globalwritinganessay.com
    top rated essay writing service
    top rayed essay writing service
    https://globalwritinganessay.com

  • buy custom essay online2020-12-31 20:52

    Does your blog have a contact page? I'm having problems locating it but, I'd lie tto send you an e-mail.
    I've got some ideas for your blo you might be interested in hearing.
    Either way, great site andd I look forward to seeing it improve over time.

    https://goodpaperwritingservices.com
    buy custom essay online
    buy custom esssy online
    https://goodpaperwritingservices.com

  • This iss a good tip particularly to those neew to the blogosphere.

    Shortt but very prtecise info… Many thanks forr sharing this one.

    A must rea post!
    https://essaywritingtld.com
    cheapest essqy writing service
    cheapest essay writing service
    https://essaywritingtld.com

  • buy essay writing2020-12-31 21:18

    Hello, I think your website might be having browser compatibility
    issues. When I look at your blog site in Chrome, it looks fie but when opening in Internet Explorer,
    it haas some overlapping. I just wanted to give you a quick heads up!
    Othger then that, terrific blog!
    https://essayserviceusukua.com
    buy essay writing
    buy essay writing
    https://essayserviceusukua.com

  • I'm gone to convey my little brother, that he should
    also go tto see this website on regular basis too get updated from
    latest information.
    https://essaygood.com
    essay writing elp service
    essay writing help service
    https://essaygood.com

  • Do you mind if I quote a couple of yoyr artticles as long as I provide credit and soufces back to your
    site? My blog site is in the exact same niche aas yours and
    my users would truly benefit from some of the informjation you provide here.
    Pleaxe let me know if this alright with you.
    Thanks a lot!
    https://essayhelpforall.com
    best paper writing services
    best paper writing services
    https://essayhelpforall.com

  • website to write essays2020-12-31 23:38

    Good write-up. I certainly love this website.
    Keep writing!
    https://buyessayonlinereviews.com
    webbsite to write essays
    website to write essays
    https://buyessayonlinereviews.com

  • online paper2021-01-01 00:03

    Really when someone doesn't understand afterward its up to other visitors that they will help, so here itt happens.

    https://amazingcustompaperwritingservice.com
    online paper
    online paper
    https://amazingcustompaperwritingservice.com

  • writing service2021-01-01 04:51

    What's up, after reading this amazing paragraph i am too delighted to share myy familiarity
    here with colleagues.
    https://mywritingexperience.com
    writing service
    writing service
    https://mywritingexperience.com

  • writing service2021-01-11 19:26

    You are so interesting! I do not think I've truly read
    something like this before. So wonderful to find another person with a few original thoughts on this subject matter.
    Really.. many thanks for starting this up. This website is something
    that is require on the internet, someone with a bit off originality!

    https://internationaltopwriting.com
    writing service
    writing service
    https://internationaltopwriting.com

  • You ought to be a part of a conhtest for one of the most useful websites
    on the web. I am going to recommend this site!
    https://webtheessaywriter.com
    college essay writing service reviews
    college essay writing service reviews
    https://webtheessaywriter.com

  • Magnificent goods from you, man. I have understand your stuff previous to
    and yoou are just too excellent. I really like
    what you've acauired here, crtainly like what you are sayibg and the
    way in which you say it. You make itt entertaining and you still care
    for to keep it smart. I can't wait to read muhch ore from you.
    This is actually a tremendous web site.
    https://dolf097z5tjo.medium.com/
    custom dissertation writers
    custom dissertation writers

  • 1v1 lol2021-02-04 14:37

    I really like your content because it is very informative and knowledgeable content.

  • slope unblocked2021-02-04 14:42

    Thank you very much for sharing this useful information. I was doing a project and for that, I was looking for related information. Some of the points are very useful. Do share some more material if you have.

  • Raleigh2021-03-08 12:39

    Hi there are using Wordpress for your site platform?
    I'm new to the blog world but I'm trying to get started and set up
    my own. Do you need any html coding knowledge to
    make your own blog? Any help would be greatly appreciated!

  • lo1maf222021-03-09 18:03

    This is a topic which is near to my heart... Cheers!
    Where are your contact details though?

  • Rozella2021-03-10 12:10

    If you want to improve your experience only keep
    visiting this site and be updated with the most up-to-date information posted here.

  • see here now2021-03-13 22:36

    Wow, that's what I was looking for, what a data!
    present here at this blog, thanks admin of this web page.

  • directory2021-03-14 17:21

    I absolutely love your blog.. Very nice colors & theme. Did you develop this amazing site yourself?
    Please reply back as I'm wanting to create my very own website and
    would love to learn where you got this from or just what the theme is called.
    Kudos!

  • 0mniartist2021-04-08 05:41

    Heya i'm for the first time here. I found this board and I to find It truly helpful
    & it helped me out much. I am hoping to present
    something back and help others such as you helped me.
    0mniartist asmr

  • free dating sites
    Having read this I believed it was rather enlightening.
    I appreciate you finding the time and energy to put this information together.
    I once again find myself personally spending way too much time both reading and commenting.
    But so what, it was still worthwhile! https://785days.tumblr.com/ dating sites

  • dating sites
    Just wish to say your article is as amazing. The clearness on your post is
    just cool and i could suppose you're an expert in this subject.
    Well with your permission allow me to grasp your feed to stay updated with
    drawing close post. Thanks a million and please carry
    on the gratifying work. free dating sites

  • Gail2021-08-29 00:42

    My family members always say that I am killing my time here at web, except I know I am getting know-how every day by reading such fastidious articles.

  • Hi thеre to all, the contents presеnt at this web site аre genuinely remarkable fоr people
    experience, ԝell, kеep up tһe good work fellows.

  • Bellueur Cream2021-08-29 00:51

    Sweet site, super design, really clean and utilise pleasant.

  • Hedofirm Male Enhancement2021-08-29 00:56

    I?m impressed, I have to admit. Rarely do I come across a blog that?s both educative
    and amusing, and let me tell you, you have hit the nail on the head.
    The issue is an issue that not enough men and women are speaking intelligently
    about. I am very happy I came across this in my search for something
    relating to this.

  • ToxyBurn2021-08-29 01:36

    I couldn?t resist commenting. Perfectly written!

  • ToxyBurn2021-08-29 01:39

    Attractive section of content. I just stumbled upon your weblog and in accession capital to assert
    that I get actually enjoyed account your blog posts.
    Any way I will be subscribing to your augment and even I achievement you access consistently fast.

  • It's actually very complicated in this full of activity
    life to listen news on TV, therefore I simply use world wide web
    for that purpose, and get the most recent information.

  • Pretty! This has been an incredibly wonderful article.
    Thanks for supplying these details.

  • Hi i am kavin, its my first occasion to commenting anyplace, when i read this article i
    thought i could also create comment due to this good

    paragraph.

  • dự án Empire City2021-08-29 14:59

    thương hiệu những bậc thang lơ lửng tỏa
    ra nhiều hướng quay quanh trục nội khu của
    tháp biểu trưng cho sự hòa hợp giữa tình trạng khu đô thị và thị
    trường tự nhiên của bán đảo, giữa thành thị
    và vùng quê. khu rừng rậm tại chiều cao, với sự liên hệ
    gợi cảm giữa thiết kế bên ngoài và nội thất, tiếp tục mang tới hưởng thụ kỳ diệu từ cảnh quan nhiệt đới của Việt Nam.

    có được Căn hộ Empire City không chỉ có được 1 sản phẩm nhà đất, việc đấy còn cho thấy được cao
    cấp của người chủ. 1 căn hộ triệu đô tới từ
    tên Keppel, toạ lạc mặt bờ con sông Sài Gòn.
    cạnh bên giá trị về mặt tài chính thì dự án công trình còn cho biết được sự thời thượng,
    tinh tế và đẳng cấp xứng khoảng thượng lưu.

  • I just couldn't leave your web site before suggesting that I really
    loved the usual information an individual provide on your visitors?
    Is gonna be back often in order to investigate cross-check new posts

  • Hey! Do you know if they make any plugins to safeguard against hackers?
    I'm kinda paranoid about losing everything I've worked hard on. Any recommendations?

  • Pretty! This has been a really wonderful post.
    Thanks for providing this info.

  • We are a group of volunteers and opening a new scheme in our community.
    Your website offered us with valuable information to work on. You have done an impressive
    job and our entire community will be grateful to you.

  • Have you ever considered publishing an e-book or guest authoring on other sites?
    I have a blog based on the same subjects you discuss and would really like to have you
    share some stories/information. I know my visitors would value your work.
    If you're even remotely interested, feel free to send me an email.

  • Hello to every single one, it's in fact a fastidious for me to pay a visit
    this web site, it includes priceless Information.

  • Now I am ready to do my breakfast, after having my breakfast coming
    again to read additional news.

  • Way cool! Some very valid points! I appreciate you penning this article and the rest
    of the site is really good.

  • Since the admin of this site is working, no doubt very soon it will be famous, due to its feature contents.

  • Thank you for the auspicious writeup. It
    in fact was a amusement account it. Look advanced to far added agreeable from
    you! However, how could we communicate?

  • Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
    You definitely know what youre talking about, why waste your intelligence on just posting videos
    to your weblog when you could be giving us something informative to read?

  • This piece of writing is actually a pleasant one it helps new net people,
    who are wishing for blogging.

  • summer superfoods2021-09-08 16:21

    Hello would you mind stating which blog platform you're using?

    I'm planning to start my own blog soon but I'm
    having a difficult time selecting between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your layout seems different then most blogs and
    I'm looking for something unique. P.S Apologies for being off-topic but I had to ask!

  • Terra Xtract CBD Review2021-09-09 17:23

    I happen to be commenting to let you know what a amazing experience
    my child encountered viewing your web site. She
    came to find a lot of things, most notably what it is like to possess an excellent helping spirit to make many people
    easily comprehend specific extremely tough subject areas.

    You undoubtedly did more than our expected results.
    Many thanks for churning out those valuable, trusted, revealing as well as easy guidance on this topic to Julie.

  • 7 keto weight loss2021-09-10 12:29

    When someone writes an piece of writing he/she maintains the image of a user in his/her brain that how a user
    can understand it. Therefore that's why this paragraph is perfect.
    Thanks!

  • mens health questions2021-09-11 02:16

    Thank you a bunch for sharing this with all of us you
    really realize what you are speaking about! Bookmarked.
    Kindly also consult with my website =). We will have a link exchange arrangement between us!

  • email marketing2021-09-12 04:15

    I have been reading out a few of your posts and i can state nice stuff.

    I will make sure to bookmark your site.

  • lose weight2021-09-12 05:07

    That is really interesting, You are a very skilled blogger.

    I've joined your rss feed and look ahead to in the hunt for more of your magnificent post.
    Additionally, I've shared your website in my social networks!

  • car rental2021-09-13 13:03

    Hi there! This post could not be written any better!

    Reading through this post reminds me of my good old room mate!
    He always kept chatting about this. I will forward this article to him.
    Pretty sure he will have a good read. Many thanks for sharing!

  • pubic hair removal2021-09-14 09:20

    I have been examinating out a few of your stories and it's clever stuff.
    I will definitely bookmark your site.

  • home remedies for acne2021-09-14 09:43

    Hey There. I discovered your blog the usage of msn. That is a really smartly written article.
    I'll be sure to bookmark it and come back to read extra
    of your helpful info. Thanks for the post. I will definitely comeback.

  • pubic hair removal2021-09-14 10:09

    No matter if some one searches for his necessary thing, thus he/she wishes to be
    available that in detail, thus that thing is maintained over here.

  • lose fat fast2021-09-14 12:09

    Hello there, simply become alert to your blog via Google, and located that it is really informative.
    I am going to be careful for brussels. I'll
    be grateful in the event you continue this in future. Lots of other folks can be benefited from your writing.
    Cheers!

  • preventing hair loss2021-09-14 12:43

    Awsome website! I am loving it!! Will come back again. I
    am bookmarking your feeds also

  • best low carb diet2021-09-14 14:15

    Howdy! This article couldn't be written any better!
    Looking at this post reminds me of my previous roommate!
    He constantly kept talking about this. I most certainly will forward this information to him.

    Fairly certain he will have a very good
    read. Thank you for sharing!

  • eco friendly hemp2021-09-14 17:16

    I like this post, enjoyed this one thanks for posting.

  • pillow pets2021-09-14 18:55

    Truly no matter if someone doesn't know then its up to other people
    that they will help, so here it takes place.

  • natural fabrics2021-09-14 20:04

    At this time I am going to do my breakfast, after having my
    breakfast coming yet again to read more news.

  • low carbohydr2021-09-15 03:04

    Hey, you used to write excellent, but the last several posts have been kinda boring?
    I miss your super writings. Past several posts are just a
    bit out of track! come on!

  • carslot882021-09-17 07:10

    Good web site you have got here.. It's difficult to find quality writing like yours these days.
    I really appreciate individuals like you! Take care!!

  • hair removal sugaring2021-09-20 19:15

    WOW just what I was looking for. Came here by searching for waxing hair removal

  • regular cyclical cyclical2021-09-21 20:24

    Hi my loved one! I want to say that this article is awesome, nice written and include approximately all significant infos.
    I would like to peer extra posts like this.

  • increase muscle mass2021-09-22 00:43

    Hi there, You have performed an incredible job. I will certainly
    digg it and in my view recommend to my friends.
    I am confident they will be benefited from this site.

  • handmade soap2021-09-23 01:53

    Hmm is anyone else encountering problems with the pictures
    on this blog loading? I'm trying to find out if its a problem on my end or if it's the blog.
    Any responses would be greatly appreciated.

  • Diaetoxil2021-10-05 19:03

    This web site is my inhalation, very excellent design and Perfect content material.

  • Matt2021-10-06 05:39

    I read this piece of writing completely on the topic of the comparison of most recent and previous technologies,
    it's amazing article.

  • Victoria2021-10-06 12:57

    Rattling excellent information can be found
    on weblog.

  • Right away I am going away to do my breakfast, when having my breakfast coming yet again to
    read other news.

  • slot online2021-11-23 02:42

    Wonderful items from you, man. I've bear in mind your stuff
    previous to and you are just too great. I really like what you have received
    here, really like what you're saying and the way during which
    you assert it. You're making it entertaining and you still take
    care of to stay it sensible. I cant wait to read far more from you.
    That is really a terrific web site.