Problem Description

After installing KB2647170, DNS.exe (Version: 5.2.3790.4957) process can consume up to 2GB of Virtual Memory (VM Size) on Windows Server 2003 R2 x86 and as a result DNS.exe process may not able to process client requests.

MS12-017: Vulnerability in DNS Server could allow denial of service: March 13, 2012
http://support.microsoft.com/kb/2647170/en-us

Mentioned below events are registered.

Events

Event Type: Error
Event Source: DNS
Event Category: None
Event ID: 111
Date: 04.06.2013
Time: 16:00:34
User: N/A
Computer: XXX
Description:
The DNS server could not create a thread. System may be out of resources. You might close applications not in use, restart the DNS server or reboot your computer. The event data is the error code.

Event Type: Error
Event Source: DNS
Event Category: None
Event ID: 7502
Date: 04.06.2013
Time: 16:02:04
User: N/A
Computer: XXX
Description:
The DNS server was unable to service a client request due a shortage of available memory. Close any applications not in use or reboot the computer to free memory.

Solution

Mentioned below article from Tiger Li on technet forum is an Microsoft internal SOX which explains the design change in DNS.exe process and provides some suggestions to decrease the VM usage inside DNS.exe process.

In our case the provided solution from Tiger Li did not help and Microsoft verified that KB2647170 causes a high vm usage but they will not fix the bug DNS.exe process in Windows Server 2003 R2.

As a workaround Microsoft suggested either to upgrade to 2008 R2 or to schedule a batch job and restart the dns service every day to prevent to hit the 2GB VM Usage threshold.

Below you will find the statement from Tiger Li [MSFT] on TechNet :

DNS error 7502
http://social.technet.microsoft.com/Forums/windowsserver/en-US/e7c687a4-5f7c-49ae-89b7-ff72253d6934/dns-error-7502

Hi,
Thanks for posting here.

May I know what was the actual memory usage of DNS.exe when we got this error message ? we may check it form the task manager .

5.2.3790.4957 is the latest version number of DNS.exe on windows server 2003 X86 platform based on my knowledge. So I’d say that this issue might not been caused by potential system bug here . The best suggestion we can provide it to implement workaround about shortening DNS cache lifetime to help prevent excessive memory usage.Since we have KB2647170 applied, here are some notes for reference :

1. Once DNS.EXE has allocated memory for cache, it never frees that memory which appears has high node hash count in dnscmd /statistics.

2. DNS exe never releases memory back to the OS

MSKB 2647170 / MS12-17 allows DNS.EXE to re-allocate heap memory back to itself. The example if DNS allocates memory for cache, it will internal free and re-use that memory. However, DNS will not release memory back to the OS. For example, with MSKB 2647170 / MS12-017 installed, DNS.EXE with default cache settings installed may consume 2GB of memory in a high-stress situation. Once the high-stress dissipates, DNS.EXE will not free memory back to the OS but will continue to consume 2GB of memory until the service is restarted. This is by design.

So the workaround is Incrementally “walk” maxcachettl to to a minimum of 300 and / or reduce the value for maxnegativecachettl to as low a minimum threshold of “60” seconds by modifying both registry key “MaxCacheTtl” and “MaxNegativeCacheTtl”. Monitoring our network and the DNS server for increased query traffic and increased DNS Sever workload after the changing.

And yes, the workaround about setting EnableDuplicateQuerySuppression to zero is still applied.

Thanks.
Tiger Li

Good luck !