Welcome to the first blog of the new year, sorry for the late start. I’ve been stuck in the trenches trying to figure out the application flow for the posturing module used in the AnyConnect client.
A client of mine is having mixed results when an endpoint authenticates and then attempts to posture to their network. Some of the symptoms that are experienced are as followed:
- Device and user authenticates, AnyConnect reports incorrect policy server
- User reports intermittent network interrupts during their working day
- User receives “Untrusted server blocked!” (certificate issue)
I’ve invested a lot of time absorbing and digesting Cisco’s documentation for Cisco ISE and AnyConnect and believe at this point I have a good understanding of it.
Here is what I found and how you can perform the same troubleshooting steps to help resolve your issues.
We are focusing on Posturing. So after Authentication and Authorization is successful, posturing begins. Posturing is just a way to ensure that your endpoints are complying to your companies network policies before gaining network access.
The AnyConnect Posture Module begins by initiating policy server detection. This is accomplished through a series of probes which are known as discovery probes.
There are three probes in total, and I will show you how they look.
Probe 1 – AnyConnect sends first discovery probe to the clients default gateway. This discovery probe along with the next two are HTTP GET requests to /auth/discovery.
This request will be intercepted by the switch that the client is connected to and present a redirect-url to the client. This redirect is that of your policy node. If your AC is unsuccessful it will attempt a second probe.
Reminder: this is all done in the background and is not known to the user
Probe 2 – AC sends second probe. A HTTP GET /auth/discovery to enroll.cisco.com. This FQDN needs to be successfully resolvable by DNS server. In VPN scenario with split-tunnel, traffic to enroll.cisco.com has to be routed through the tunnel.
Expected result for the probe is redirect-url to your policy nodes.
Probe 3 – HTTP GET /auth/discovery to discovery host. Discovery host value is returned from ISE during installation in AC posture profile. Expected result for the probe is redirect-url
Your AC posture profile lives here.
C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\ISE Posture\ISEPosture.xml
Unfortunately none of these probes were successful for my client.
Posture would sometime succeed but not always. When it was successful the client was pointing to an old policy server in their old ISE 1.3 environment yet their clients were Authenticating to their new ISE 2.2 environment. hmm..
That turned out to be a key part in resolving their issue. My client had recently migrated from ISE 1.3 to ISE 2.2. ISE 2.2 changes the way posturing works.
Posturing in ISE 2.2 assumes that redirects are not needed but does support backwards compatibility for environment that use redirects.
I won’t be going into detail on how posture work in ISE 2.2. But do know that there are two stages.
Stage 1 uses the same traditional discovery probes that we listed above for backwards compatibility.
Stage 2 uses two discovery probes.
Probe 1 – Attempts to discover your PSN through IP/FQDN from the “CallHome list” that are defined in your posture profile located in
C:\ProgramData\Cisco\CiscoAnyConnect Secure Mobility Client\ISE Posture\ISEPostureCFG.xml
Probe 2 – AC tries the PSN FQDNs. It generates what should be a dynamically created file (ConnectionData.xml) upon first posture attempt. That file is located here.
C:\Users\<currentuser>\AppData\Local\Cisco\Cisco AnyConnect Secure Mobility Client\
The goal of both probes is to get FQDN. They are both there in the case where you don’t have your “CallHome List” defined for the first probe to succeed.
So after knowing this. I reference ConnectionData.xml. The .xml file had current ISE 2.3 and Old ISE 1.0 FQDNs.
I could delete and modify this file to exclude old ISE 1.0 FQDNs but it would reappear upon the next posture. I continued my troubleshooting.
I deleted ConnectionData.xml. I uninstalled the posture module from the client. I had the client re-posture so that I would receive the installation of the posture module.
I then referenced my newly dynamically generated ConnectionData.xml and bam! The .xml file only had ISE 2.3 FQDNs. It had no knowledge of old ise. Come to find out, the customer had a off line deployment package that included the ConnectionData.xml file and placed it in C:\Users\<currentuser>\AppData\Local\Cisco\Cisco AnyConnect Secure Mobility Client\…..When AnyConnect is deployed off-line and includes the .xml file, it creates a static entry in the file even though that file is rebuilt at each posture.
After fixing the discovery probes which was to enable http server on the default gateway where the hosts live and correcting the .xml file to exclude old ISE FQDNs. Clients were posturing to new ISE.
It’s been a long journey in this discovery but things are looking up.
Mike
Apronets 
Awesome stuff Mike!
Well explained and very useful for understanding the posture module working.
LikeLike
Thanks Ali, glad it helped!
LikeLike
very useful article Mike, how I ended up here is because I am dealing with MAC users who are having issue when trying to connect via Cisco Anyconnect and getting stuck in “searching for policy server” error. Cisco tshoot guides doesn’t explain much around this and I am currently looking to reproduce the issue myself however couldn’t reproduce it yet. The current work around is to restart MAC, launch Cisco Anyconnect and it seems start working again. I have checked the DART log files sent by affected users and also downloaded the ISE log file however it doesn’t tell me much as why it doesn’t move from searching for policy server. Would be glad to know if you have dealt with this sort of issue? 🙂
LikeLike
Hi Faisal! Thanks for the positive feedback. So based on a limited view of your issue I can only speculate. May it will help spawn some idea’s on your end? You state the restarted the MAC resolved the issue (most times), is MAR enforced in your ISE environment?
https://www.cisco.com/c/en/us/support/docs/lan-switching/8021x/116516-problemsolution-technology-00.html
What version of ISE? Posturing differs in post 2.2.
https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine-22/210523-ISE-posture-style-comparison-for-pre-and.html
This will feel like finding a needle in a haystack so I would consider opening a TAC case around it unless you don’t mind being bugged be end-users 🙂
Here are some helpful debug commands that you can use on the switch the endpoint is connected to actively see the authentications happening to ISE.
debug radius verbose
debug epm all
debug authentication all
debug dot1x all
debug authentication feature all
debug aaa authentication
debug aaa authorization
LikeLike
Hi Mike, thanks for your reply and providing the useful debug commands. We don’t have MAR enforced in our environment. I agree with you as I have already spent a lot of time to tshoot but couldn’t find anything in the logs, I think its time to contact TAC 🙂
Thanks again
LikeLike
Hey, I’m having the same problem in macOS (Catalina), getting stuck at “Searching for policy server” randomly, and after restarting usually work. This did not happen before Catalina update…maybe it’s a problem with Catalina and AnyConnect? (was on Mojave and El Capitan before). Did you get any advice for this problem?
Thanks
LikeLike
It is still happening in our case and not just with Catalina but also with Mojave as well, we pin point the issue with split tunneling as we are allowing a small range of network (/40) to use in local access LAN for VPN users. There were few recommendation made by Cisco Tac which we have made for example apply Tunnel-all configuration for Macs in ASA configuration
https://www.cisco.com/c/en/us/support/docs/security/anyconnect-secure-mobility-client/116016-technote-AnyConnect-00.html#anc11
I wouldn’t say it is completely gone but there are less reports we are receiving from end users now.
LikeLike
Hi Mike,,,we are running ise 3.0 version…can the switch and ise are on different segment seperated by Firewall.
we have download anyconnect manually on pcs and depend on redirect acl for complaince module download.
we are unable to download it show no policy server found. we have allowed port 80/8443 from user to ise.
i am seeing redirect url on show authentication session. but it doesn’t work.
when i have the user , switch and ise on same vlan than i am able to download the compaince module.any idea whats wrong ,,
LikeLike
Hello, Mustansir
Can the switch and ise are on different segment separated by Firewall?
[MD] – Generally, ISE is deployed in front of a firewall. Is there a business reason for deploying ISE behind a firewall?
To troubleshoot this, you could put a client (laptop) on the same segment as the ISE and perform authentication. Does it work then? Unfortunately, there are a lot of considerations that I am not purview of. It might be worth engaging TAC for assistance.
LikeLike
Hi Mike, Thanks for sharing the valuable info. This isn’t something we get in the Cisco community forums.
I’m running into a similar issue now where it’s been frequently popping up with the Old ISE PSN info as the ‘Policy Server’. I deleted ConnectionData.xml but it keeps re-adding with the old ISE node. You’ve given as below.
I uninstalled the posture module from the client. I had the client re-posture so that I would receive the installation of the posture module.
Can you please suggest how can this be achieved for 1000+users?
LikeLike
Hi Sri,
Thanks for reaching out! A couple of things: 1) Did the ConnetionData.xml get deployed via ISE or was it seeded the Desktop/Application team in their AnyConnect install packag?
2) Having to remove 1000+ AC from user device is going to be an issue. You’ll have to use a management tool like SCCM if you are a Microsoft shop or some other automation means. You could possible use Group Policy and script the removal of AC.
3) TAC may have a removal script for your to use. It might be worth asking if they have a script so you don’t have to recreate it.
Finally, old data found in ConnectionData.xml is usual due to a migration taking place and the FQDN of the ISE nodes change, leaving behind stale FQDN data. OR the ConnectionData.xml was seeded in during the install on the users devices. This File should be dynamically created rather than seeded into an install.
I hope that you found this helpful!
LikeLike
Hi, Great post! I have got a query regarding the posture. With the Anyconnect mobility client (pre-deploy package), we’ve got an ISE posture module. In addition to that, Cisco offers a Compliance module as well. As the compliance module (system scan) is performing the posture checks, I’d like to know about the ISE posture module (which is part of Anyconnect pre-deploy) and what is it responsible for? Thanks in advance.
LikeLike
Hi Kumar,
Simply put, the compliance module sets the benchmark or defines a set of security standards you want your devices to comply with. The posture module is used to validate that they are in compliance with the security standard that you defined.
Once the AnyConnect agents retrieve this support information, they check the latest definition information from the periodically updated se-checks.xml file (which is published along with the se-rules.xml file in the se-templates.tar.gz archive), and determine whether clients are compliant with the posture policies.
https://www.cisco.com/c/en/us/td/docs/security/ise/2-7/admin_guide/b_ise_27_admin_guide/b_ISE_admin_27_compliance.html#id_17065
LikeLike