While deploying an Azure MFA solution integrating with a Cisco AnyConnect VPN I discovered a very frustrating issue that burned an untold amount of time – in short the problem was due to the use of a RADIUS secret with symbols and when removed resolved the issue immediately. I’ve wanted to write this post because I discovered several diagnostic tools that massively helped tracking down the root cause.
I’ll start by saying this isn’t my first instance of using Azure MFA with Cisco AnyConnect, however is the first time I’ve needed to work with a 3rd party due to their ownership and control of the Cisco ASA firewalls. My usual process is to setup a Windows server with the NPS role, create the policies and RADIUS clients with a generated secret and then install the Azure MFA NPS extension via PowerShell. Then, on the RADIUS client I’d set that up accordingly to send authentication messages to my server although in this case the task was left to the 3rd party. Unfortunately as a result I’m unable to specify which model of ASA or which firmware version I encountered this problem with.
Testing of the process yielded bizarre results as the NPS DLL extension refused the connection every time despite the user existing in Azure AD, having a valid licence and being enrolled in MFA already.
Reason: An NPS extension dynamic link library (DLL) that is installed on the NPS server rejected the connection request.
The first tool in my arsenal was this PowerShell tool which was able to test and collect more detailed logs than I was seeing in the Network Policy and Access Services event log on the Windows server. Despite this all I was seeing was:
NPS Extension for Azure MFA: NPS Extension for Azure MFA only performs Secondary Auth for Radius requests in AccessAccept State. Request received for User email@example.com with response state AccessReject, ignoring request.
Indicating that RADIUS itself was refusing the connection, not the extension. I uninstalled the extension and tested again, this time seeing
Reason: Authentication failed due to a user credentials mismatch. Either the user name provided does not map to an existing user account or the password was incorrect.
Now I was certain that the username was correct as I was able to lock the account out every couple of trys, and the password was almost certainly correct as I’d reset it to various simple and complex variations to see if that had any effect but still was unable to get RADIUS to authenticate the user.
I then turned to another tool NTRadPing in order to test RADIUS authentication from another server on the LAN across to the NPS server using the exact same RADIUS secret, username / password, policies etc. This worked each and every time, returning success messages.
So for some reason at this stage, with the exact same details, the ASA was unable to receive authentication for users while the tool could.
A side by side of the NPS event log messages shown no obvious difference in the information coming in on the requests yet the ASA always resulted in the user account ultimately being locked out despite my certainty of using a valid password.
At this point I tried increasing the amount of characters in the RADIUS secret above 22 as a desperate Google led to a TechNet article where someone allegedly had success doing this, but even this was unsuccessful.
I then installed the NPS role onto a new server thinking it may be something wrong with the original, perhaps a missing patch, but the results were exactly the same – authentication for ASA sourced requests failed while the NTRadPing test tool was always a success.
Big guns time. I busted out Wireshark and captured the RADIUS requests from the ASA and also from the NTRadPing tool. Annoyingly there was no difference even here in the messages other than the PAP encrypted password changing for each request based on some hash of the packet and RADIUS secret. Another trick emerged that I’d not been aware of in Wireshark until now – you can enter the RADIUS key into the settings and it’ll decrypt the password in the capture! This is done under Edit, Preferences, Protocols, RADIUS and then entering the shared secret.
The difference was immediately obvious between the allowed request and the denied request, with the former displaying the correct password and the latter (from the ASA) showing a completely different decrypted password in a strange format. Bear in mind at this point the RADIUS key had been checked several times and even changed, and there were no log errors about an invalid RADIUS client I’ve seen previously when the RADIUS secret mismatched, yet somehow the ASA was incorrectly encrypting the password.
The thought occurred to simplify the RADIUS secret by removing the % symbol and testing again. Thankfully this worked and I’ve got to say the journey to get here while frustrating did yield some great discoveries in diagnostic tools and steps!