DetailPage-MSS-KB

Microsoft small business knowledge base

Article ID: 975363 - Last Review: November 1, 2012 - Revision: 17.0

Hotfix Download Available
View and request hotfix downloads
 

On This Page

Symptoms

You may experience one or more of the following symptoms. These symptoms may be intermittent or continuous. These symptoms are more likely and more widespread during "high usage" times, such as at the beginning of a business day when increased client load occurs on the servers in the environment.

You may experience the following issues in a web services scenario:
  • Web clients receive delayed responses from the web server.
  • Web clients are repeatedly prompted for credentials even if the correct credentials are entered.
You may experience the following issues in a web proxy scenario:
  • Web clients receive delayed responses from the web server.
  • Web clients are repeatedly prompted for credentials even if the correct credentials are entered.
You may experience the following issues in an Exchange client scenario:
  • Clients receive delayed responses from the server.
  • Clients are repeatedly prompted for credentials even if the correct credentials are entered.
You may experience the following issue in any scenario in which NTLM authentication is used for applications:

Line of business or custom applications that use NTLM authentication fail. Additionally, you may receive different errors that are intermittent and may include "access denied."
You may experience the following issue in a remote file access scenario:
Windows clients receive "access denied" errors or delayed responses from the file server.
You may experience the following issue in any scenario in which Kerberos delegation is being used in a middle-tier service:
The clients gain access successfully at first but then lose access to the same resources. Additionally, you may be repeatedly prompted for credentials or experience "access denied" errors.
Notes
  • This issue is more likely to occur if one or more of the following conditions are true:
    • There are highly transactional and heavily used services in the environment.
    • There is heavy use of scripts that use the WINNT provider.
    • There are applications and services that are not configured (or are not configurable) to use Kerberos authentication.
    • When the following three conditions are true at the same time:
      • There are many "accounts" domains (in other words, domains that have user accounts in them) in the environment.
      • There are Windows Server 2003-based domain controllers (DCs) .
      • There are applications or services that may authenticate without providing the domain name. For example, there are applications or services that provide <null>\username instead of domainname\username.
  • The following symptoms indicate that this issue is occurring in the environment:
    • A Kerberos source event is logged in the System log of application servers. This event indicates that Kerberos PAC validation is failing. The event resembles the following:

      Event Type: Error
      Event Source: Kerberos
      Event Category: None
      Event ID: 7
      User: N/A
      Description: The Kerberos subsystem encountered a PAC verification failure. This indicates that the PAC from the client computer name in realm AD DNS domain name had a PAC which did not verify or was modified. Contact your system administrator.
      Data: 0000: c0000192

    • Text in Netlogon service debug logs (Netlogon.log) matches the text "NlpUserValidateHigher: Can't allocate Client API slot." These entries may appear in any of the Netlogon debug logs of the following servers:
      • The application server
      • The domain controllers in the application servers domain
      • Trusting domain controllers
    • Perfmon performance logging of the Netlogon performance counter for Semaphore Timeouts during the time when the issue is occurring shows numbers greater than zero. This counter value may appear on any of the following servers in this scenario:
      • The application server
      • The domain controllers in the application servers domain
      • Trusting domain controllers

Cause

This issue occurs when a high volume of NTLM authentication or Kerberos PAC validation transactions (or both) occur on a Windows-based server, and that volume is greater than the volume that can be handled at one time by the member server or the domain controllers that are providing authentication. In other words, this is caused by an authentication resource bottleneck.

NTLM authentication and PAC validation are performed by dedicated threads in the Lsass.exe process on Windows-based computers. There is a maximum number of these threads that are available to handle these requests at the same time, and if the requests exceed the availability of the threads and the requests cannot wait any longer, this issue occurs.

By default, workstations have one of the threads available for use, and member servers have two of the threads available for use. Domain controllers have one available thread per security channel to trusted domains. This maximum number of threads that are dedicated to this purpose is known as "Maxconcurrentapi" and is configurable.

Resolution

To resolve the issue, use one or more of the following methods:
  • Install the following hotfix, and then follow the steps that are described in the "Registry information" section. After you install this hotfix on Windows Vista, Windows Server 2008, Windows 7, or Windows Server 2008 R2, the maximumlimit of concurrent connections between a client computer and another server or a domain controller for NTLM authentication or PAC validation may be changed up to 150. This should be done on all servers that exhibit Perfmon Netlogon “semaphore time-out” indications in their Performance logs or that have the “NlpUserValidateHigher: Can't allocate Client API slot” text in their Netlogon debug logs.
  • For applications and services that are using NTLM, just configure them to use Kerberos authentication instead. The methods to do that will be unique to those applications.
  • If Kerberos PAC validation is seen as a symptom, disable Kerberos PAC validation if the service allows this. This should be done on the server that has the Kerberos sourced system event 7’s appearing.

    Note Kerberos PAC validation cannot be disabled for IIS application pools or for some Exchange-related services.
Note In order to decide what value to set for the MaxConcurrentApi setting in your environment refer to the Knowledge Base article below.

2688798 (http://support.microsoft.com/kb/2688798) How to do performance tuning for NTLM authentication by using the MaxConcurrentApi setting

Hotfix information

A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing the problem that is described in this article. This hotfix might receive additional testing. Therefore, if you are not severely affected by this problem, we recommend that you wait for the next software update that contains this hotfix.

If the hotfix is available for download, there is a "Hotfix download available" section at the top of this Knowledge Base article. If this section does not appear, contact Microsoft Customer Service and Support to obtain the hotfix.

Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that do not qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, visit the following Microsoft website:
http://support.microsoft.com/contactus/?ws=support (http://support.microsoft.com/contactus/?ws=support)
Note The "Hotfix download available" form displays the languages for which the hotfix is available. If you do not see your language, it is because a hotfix is not available for that language.

Prerequisites

To apply this hotfix, your computer must be running Windows 7, Windows Server 2008 R2, Windows Vista Service Pack 2, or Windows Server 2008 Service Pack 2.

Registry information

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756  (http://support.microsoft.com/kb/322756/ ) How to back up and restore the registry in Windows
After you install the hotfix, increase the Maxconcurrentapi value to a larger number on all servers that have Perfmon Netlogon "semaphore timeout" indications in their Performance logs or that have "NlpUserValidateHigher: Can't allocate Client API slot" text in their Netlogon debug logs. To do this, follow these steps:
  1. Start Registry Editor.
  2. Locate the following registry subkey:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters
  3. Create the following registry entry:

    Name: MaxConcurrentApi
    Type: REG_DWORD
    Value:Set the value to the larger number, which you tested (any number greater than the default value).
  4. At a command prompt, run net stop netlogon, and then run net start netlogon.
Notes
  • The maximum value that can be configured depends on the operating system version and whether a hotfix is available.
    • The maximum configurable setting in Windows Server 2003 is 10.
    • The maximum configurable setting in Windows Server 2008 (without the hotfix in this article) is 10. With the hotfix, the maximum is 150.
    • The maximum configurable setting in Windows Server 2008 R2 (without the hotfix in this article) is 10. With the hotfix, the maximum is 150.
  • If you decide to increase the MaxConcurrentApivalue to greater than 10, the load and the performance of the desired setting should be tested in a nonproduction environment before you implement in production. This is recommended to make sure that increasing this value does not cause other resource bottlenecks.
To disable Kerberos PAC validation on a server, follow these steps.
  1. Start the Regedt32 program.
  2. Locate the registry key path HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters.
  3. At that registry location, create a DWORD value named ValidateKdcPacSignature.
  4. Set the value of ValidateKdcPacSignature value to 0.
  5. Restart the server.
Note Kerberos PAC validation cannot be disabled for IIS application pools or for some Exchange-related services.

Restart requirement

You must restart the computer after you apply this hotfix.

Hotfix replacement information

This hotfix does not replace a previously released hotfix.

File information

The English (United States) version of this hotfix installs files that have the attributes that are listed in the following tables. The dates and the times for these files are listed in Coordinated Universal Time (UTC). The dates and the times for these files on your local computer are displayed in your local time together with your current daylight saving time (DST) bias. Additionally, the dates and the times may change when you perform certain operations on the files.

  • The MANIFEST files (.manifest) and the MUM files (.mum) that are installed for each environment are listed separately in the "Additional file information for Windows Vista and Windows Server 2008" section. MUM and MANIFEST files, and the associated security catalog (.cat) files, are very important to maintaining the state of the updated component. The security catalog files, for which the attributes are not listed, are signed with a Microsoft digital signature.

File information for Windows Vista and for Windows Server 2008

File information for x86-based versions of Windows Server 2008 and of Windows Vista
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Netlogon.dll6.0.6002.22289592,89616-Dec-200912:09x86
Nlsvc.mofNot Applicable2,87303-Apr-200921:24Not Applicable
File information for x64-based versions of Windows Server 2008 and of Windows Vista
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Netlogon.dll6.0.6002.22289716,80016-Dec-200912:07x64
Nlsvc.mofNot Applicable2,87303-Apr-200920:58Not Applicable
File information for IA-64-based versions of Windows Server 2008
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Netlogon.dll6.0.6002.222891,216,51216-Dec-200912:05IA-64
Nlsvc.mofNot Applicable2,87303-Apr-200920:59Not Applicable

File Information for Windows 7 and for Windows Server 2008 R2

Windows 7 and Windows Server 2008 R2 file information notes Important Windows 7 hotfixes and Windows Server 2008 R2 hotfixes are included in the same packages. However, hotfixes on the Hotfix Request page are listed under both operating systems. To request the hotfix package that applies to one or to both operating systems, select the hotfix that is listed under "Windows 7/Windows Server 2008 R2" on the page. Always refer to the "Applies To" section in articles to determine the actual operating system to which each hotfix applies.
  • The MANIFEST files (.manifest) and the MUM files (.mum) that are installed for each environment are listed separately in the "Additional file information for Windows Server 2008 R2 and for Windows 7" section. MUM and MANIFEST files, and the associated security catalog (.cat) files, are very important to maintaining the state of the updated component. The security catalog files, for which the attributes are not listed, are signed with a Microsoft digital signature.
For all supported x86-based versions of Windows 7
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Netlogon.dll6.1.7600.20576563,71216-Nov-200906:40x86
Nlsvc.mofNot Applicable2,87310-Jun-200921:29Not Applicable
For all supported x64-based versions of Windows 7 and of Windows Server 2008 R2
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Netlogon.dll6.1.7600.20576692,73616-Nov-200907:45x64
Nlsvc.mofNot Applicable2,87310-Jun-200920:47Not Applicable
For all supported IA-64-based versions of Windows Server 2008 R2
Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Netlogon.dll6.1.7600.205761,148,41616-Nov-200906:10IA-64
Nlsvc.mofNot Applicable2,87310-Jun-200920:52Not Applicable


Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

More information

This hotfix is included in Windows 7 Service Pack 1 (SP1) and in Windows Server 2008 R2 Service Pack 1 (SP1).

The MaxConcurrentApi setting and the default settings for it are a legacy of Windows 2000 and of the limited hardware capabilities of that time. With older hardware, allowing for additional threads and the RPC traffic they would generate was a serious concern, and there was a possibility of performance bottlenecks if too many threads were created. With newer hardware platforms and improved performance, that hardware performance limitation is less likely to occur. As always, it is important to gauge and understand the performance of the servers in an environment before you increase the potential load by using a high MaxConcurrentApi setting.

For more information about how to use Netlogon service debug logging (Netlogon.log), click the following article number to view the article in the Microsoft Knowledge Base:
109626  (http://support.microsoft.com/kb/109626/ ) Enabling debug logging for the Net Logon service
An additional lessening step can be performed on Windows Server 2003-based domain controllers that have entries in their Netlogon service debug log that indicate that clients are submitting <null>\username instead of domainname\username. The steps are described in the following Microsoft Knowledge Base article:
923241  (http://support.microsoft.com/kb/923241/ ) The Lsass.exe process may stop responding if you have many external trusts on a Windows Server 2003-based domain controller
More information about how to use the Netlogon Performance monitoring object is available, together with an update to add that Performance object in Windows Server 2003. There is an update for Windows Server 2003 that lets you monitor the speed and the throughput of NTLM authentications. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
928576  (http://support.microsoft.com/kb/928576/ ) New performance counters for Windows Server 2003 let you monitor the performance of Netlogon authentication

There is an update for Windows Server 2008 R2 that introduces new events to track Netlogoan API overload:

New event log entries that track NTLM authentication delays and failures in Windows Server 2008 R2 are available
http://support.microsoft.com/default.aspx?scid=kb;EN-US;2654097 (http://support.microsoft.com/default.aspx?scid=kb;EN-US;2654097)
For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684  (http://support.microsoft.com/kb/824684/ ) Description of the standard terminology that is used to describe Microsoft software updates

Additional file information

Additional file information for Windows Vista and for Windows Server 2008

Additional file information for x86-based versions of Windows Vista and of Windows Server 2008
Collapse this tableExpand this table
File nameUpdate.mum
File versionNot Applicable
File size3,068
Date (UTC)16-Dec-2009
Time (UTC)21:16
PlatformNot Applicable
File nameX86_d097c3e62c5fe28649de747cfa96d8cc_31bf3856ad364e35_6.0.6002.22289_none_8f4d35d9370e9c53.manifest
File versionNot Applicable
File size705
Date (UTC)16-Dec-2009
Time (UTC)21:16
PlatformNot Applicable
File nameX86_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_ffda50c84e769578.manifest
File versionNot Applicable
File size22,701
Date (UTC)16-Dec-2009
Time (UTC)14:05
PlatformNot Applicable
Additional file information for x64-based versions of Windows Server 2008 and of Windows Vista
Collapse this tableExpand this table
File nameAmd64_0fe0181b07108c9de21c48d7ff24c52a_31bf3856ad364e35_6.0.6002.22289_none_f87d1e5f85e49530.manifest
File versionNot Applicable
File size1,060
Date (UTC)16-Dec-2009
Time (UTC)21:16
PlatformNot Applicable
File nameAmd64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_5bf8ec4c06d406ae.manifest
File versionNot Applicable
File size23,180
Date (UTC)16-Dec-2009
Time (UTC)15:52
PlatformNot Applicable
File nameUpdate.mum
File versionNot Applicable
File size3,092
Date (UTC)16-Dec-2009
Time (UTC)21:16
PlatformNot Applicable
File nameWow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_664d969e3b34c8a9.manifest
File versionNot Applicable
File size18,332
Date (UTC)16-Dec-2009
Time (UTC)14:00
PlatformNot Applicable
Additional file information for IA-64-based versions of Windows Server 2008
Collapse this tableExpand this table
File nameIa64_93a1c37632c96e8463a7fa3608662f92_31bf3856ad364e35_6.0.6002.22289_none_f505daf555102feb.manifest
File versionNot Applicable
File size1,058
Date (UTC)16-Dec-2009
Time (UTC)21:16
PlatformNot Applicable
File nameIa64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_ffdbf4be4e749e74.manifest
File versionNot Applicable
File size23,156
Date (UTC)16-Dec-2009
Time (UTC)16:08
PlatformNot Applicable
File nameUpdate.mum
File versionNot Applicable
File size2,247
Date (UTC)16-Dec-2009
Time (UTC)21:16
PlatformNot Applicable
File nameWow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.0.6002.22289_none_664d969e3b34c8a9.manifest
File versionNot Applicable
File size18,332
Date (UTC)16-Dec-2009
Time (UTC)14:00
PlatformNot Applicable

Additional file information for Windows 7 and for Windows Server 2008 R2

Additional files for all supported x86-based versions of Windows 7


Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Package_for_kb975363_rtm~31bf3856ad364e35~x86~~6.1.1.0.mumNot Applicable1,94716-Nov-200909:45Not Applicable
X86_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_fe237c4db262181f.manifestNot Applicable35,54116-Nov-200908:08Not Applicable
Additional files for all supported x64-based versions of Windows 7 and of Windows Server 2008 R2

Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Amd64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_5a4217d16abf8955.manifestNot Applicable35,54716-Nov-200908:11Not Applicable
Package_for_kb975363_rtm~31bf3856ad364e35~amd64~~6.1.1.0.mumNot Applicable2,18116-Nov-200909:45Not Applicable
Wow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_6496c2239f204b50.manifestNot Applicable16,59616-Nov-200908:01Not Applicable
Additional files for all supported IA-64-based versions of Windows Server 2008 R2

Collapse this tableExpand this table
File nameFile versionFile sizeDateTimePlatform
Ia64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_fe252043b260211b.manifestNot Applicable35,54416-Nov-200909:06Not Applicable
Package_for_kb975363_rtm~31bf3856ad364e35~ia64~~6.1.1.0.mumNot Applicable1,68316-Nov-200909:45Not Applicable
Wow64_microsoft-windows-security-netlogon_31bf3856ad364e35_6.1.7600.20576_none_6496c2239f204b50.manifestNot Applicable16,59616-Nov-200908:01Not Applicable
Keywords: 
kbqfe kbhotfixserver kbsurveynew kbautohotfix kbexpertiseinter kbbug kbfix KB975363
Share
Additional support options
Ask The Microsoft Small Business Support Community
Contact Microsoft Small Business Support
Find Microsoft Small Business Support Certified Partner
Find a Microsoft Store For In-Person Small Business Support