About the issue
Users of the Remote Desktop client (MSTSC), may experience a “Security package error occurred in the transport layer” message when the connection is configured to use a Remote Desktop Gateway. The Remote Desktop Client which is part of Windows 10 uses a new method of connecting to a remote computer when the connection is configured to use a Remote Desktop Gateway connection. This method is simply called “HTTP”. The HTTP method promises to be more efficient and secure than the previous “RPC over HTTP” method that was used in Server 2008 and Server 2012.
The Remote Desktop client negotiates a connection method to send the Remote Desktop Protocol (RDP) data to the Gateway Server. When connecting to a Server 2008 Gateway, the method will be negotiated to use RPC over HTTP. If a Server 2016 Gateway is providing the bridge, the new HTTP method will be used. The client will then fall back to the RPC over HTTP method if the negotiation fails.
Unfortunately it appears that the developers of the protocol have not documented how and when the client will fail over to the back up protocol. It is also possible that the protocol is not proxy/IDS friendly which also results in a problem.
Workaround the “security package” error
If you get the message “Your computer can’t connect to the remote computer because a security package error occurred in the transport layer. Retry the connection or contact your network administrator for assistance.” Try this registry key to force the Remote Desktop Client to always use the RPC over HTTP method. This registry key appears to be undocumented and may “switch off” other important security and performance features.
To set the registry key,
- Open Regedit.exe
- Browse to HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client
- Add a new DWORD (32) entry called RDPClientTransport
- Set the value to 1
The next time that you use a Gateway Server, the Remote Desktop Client will switch to RPC over HTTP.
This registry key also appears to resolve or provide a work around for these other connection problems.
50331661 – Your computer can’t connect to the remote computer because the Remote Desktop Gateway server is temporarily unavailable. Try reconnecting later or contact your network administrator for assistance.
50331671 – Your computer can’t connect to the remote computer because a security package error occurred in the transport layer. Retry the connection or contact your network administrator for assistance.