Wednesday, April 27, 2016

Domain Controller high CPU - Service Host / Security Log

I had been having problems for sometime with our Windows Server 2012 and 2012 R2 domain controllers.  The CPU would spike to 50% and sit there occasionally dropping down to 2% and then shortly after back up to 50%.

The process in question was the Service Host: Local Service (Network Restricted).
The sub processes are:

  • TCP/IP NetBIOS Helper
  • Windows Event Log
  • DHCP Client

It doesn't take a lot of google fu to find the following post:
https://www.experts-exchange.com/questions/28440274/Why-does-security-logging-on-the-DC-eat-all-the-CPU.html

From this post it indicates that the Windows Security log could be at fault.  I took a look at ours and it was set to overwrite and had a maximum size of 1GB (actually well under the MS maximum size, but still large imo).  A quick test of clearing the log fixed the issue.

Of course the issue started up again a few days later when the log got full and started to overwrite again.  So, I reduced the maximum size of the security log to 10mb (not large enough to hold much of course, but we're just testing at this point).  Once the log started overwriting again no issue.  From this I made the conclusion that the issue isn't just overwriting events, but rather overwriting the events when the log is very large.

Solution:
  1. Ensured that our SIEM solution was collecting the logs every hour.
  2. Set the maximum log size for the security log to a value that will hold 12 hours of logs and then overwrite.  To determine this value I just had to wait and then check it's properties (for each DC!)
  3. If any logging event success / failures are ever changed I'll need to re-evaluate that the size is still sufficient.
Another solution that I didn't like as much was to throw hardware at the issue, ie add a CPU.

In addition here's a good table of MS recommendations on logging settings
https://technet.microsoft.com/en-us/library/dn487457.aspx