![]() ![]() Usually, when you see high load on an otherwise unloaded machine, the reason is one of the following: ![]() Since the only real trouble was that we were receiving "false" positives from the monitoring system on non-production hosts, we postponed the investigation a bit and gathered some data. This was very strange, as we were not aware of anything being run on hosts at that interval and these hosts were really not in production yet, so we were quite sure that this was not being caused by external applications. While investigating the situation, we noticed a pattern: it seemed that the load spikes happened every seven hours. But then it happened on another host in a different DC, and as it turned out, it started to occasionally happen on all the virtual servers in all the datacenters – at different times. Yes, these were one-CPU virtual servers with 2GB of RAM, but even with one CPU you should be able to handle doing nothing, right? The first time we received the notification email, we were surprised, but thought it was just a temporary glitch on the hypervisor itself. When inspecting the plots, we noticed a one-minute load as high as seven or eight. except, from time to time, we received an alert email that their load was above a configured threshold. So, the virtual machines were sitting there, doing nothing. Having everything prepared, we were able to install the machines pretty quickly, but it took us some time to actually start using them. We chose an unbound DNS recursive resolver for this task, as we had positive experience with it from the past and we already had collected scripts for its performance monitoring, as well as the necessary puppet modules. ![]() We recently decided to make our DNS infrastructure inside each of our core data centers more robust and therefore installed three virtual servers on three different hypervisors to function as DC-local recursive DNS servers. Regular high load every 7 hours: Finding out how Linux measures load value. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |