Security package error – Windows 10 RDP

About the issue

Users of the Remote Desktop client (MSTSC), may experience a “Security package error occurred in the transport layer” message when the connection is configured to use a Remote Desktop Gateway.  This condition is likely to occur when the client workstation access the internet via a proxy server.

The HTTP Protocol

The Remote Desktop Client which is part of Windows 10 uses a new method of connecting to a remote computer when the connection is configured to use a Remote Desktop Gateway connection.  This method is simply called “HTTP”.  The HTTP method promises to be more efficient and secure than the previous “RPC over HTTP” method that was used in Server 2008 and Server 2012.

The Remote Desktop client negotiates a connection method to send the Remote Desktop Protocol (RDP) data to the Gateway Server.  When connecting to a Server 2008 Gateway, the method will be negotiated to use RPC over HTTP.  If a Server 2016 Gateway is providing the bridge, the new HTTP method will be used.  The client will then fall back to the RPC over HTTP method if the negotiation fails.

Unfortunately it appears that the developers of the protocol have not documented how and when the client will fail over to the back up protocol.  It is also possible that the protocol is not proxy/IDS friendly which also results in a problem.

Workaround the “security package” error

If you get the message “Your computer can’t connect to the remote computer because a security package error occurred in the transport layer. Retry the connection or contact your network administrator for assistance.” 

On the Gateway Server

The Remote Desktop Gateway Server normally accepts HTTPS traffic from client computers. Where a client computer connects via a proxy server, there is a possibility for the data flow to be transformed.

Enable the following registry keys for each Gateway Server, restarting the appropriate services, or rebooting the server will apply the changes.

Switch off Channel Binding – Add or amend the EnforceChannelBinding registry key.

Key: HKLM\Software\Microsoft\Windows NT\CurrentVersion\TerminalServerGateway\Config\Core
Name: EnforceChannelBinding
Type: REG_DWORD
Value: 0 (zero)

Setting the EnforceChannelBinding registry value to 0, sets the Remote Desktop Gateway Service to ignore any missing channel bindings.

On the Client

To force the Remote Desktop Client to always use the RPC over HTTP method.  This registry key appears to be undocumented and may “switch off” other important security and performance features.

Key: HKCU\Software\Microsoft\Terminal Server Client
Name: RDPClientTransport 
Type: REG_DWORD
Value: 1

The next time that you use a Gateway Server, the Remote Desktop Client will switch to RPC over HTTP.

This registry key also appears to resolve or provide a work around for these other connection problems.

50331661 – Your computer can’t connect to the remote computer because the Remote Desktop Gateway server is temporarily unavailable. Try reconnecting later or contact your network administrator for assistance.

50331671 – Your computer can’t connect to the remote computer because a security package error occurred in the transport layer. Retry the connection or contact your network administrator for assistance.

On both Client and Server

If clients are still experiencing a problem, you can try to adjust the Network Security authentication level, keep in mind that Group Policy will override this setting.

Set the following registry key

Key: HKLM\System\CurrentContolSet\Control\LSA 
Name: LMCompatilityLevel 
Type: REG_DWORD 
Value: 3

Setting this registry key will force Windows to “send only NTLMv2 responses” and takes immediate effect. According to this Microsoft article, this setting should be the default for Windows 7 and higher operating systems.