VPN causes developer cert in keychain to become untrusted

After not happening to me for a few months, this issue has started hitting me again with currently %100 reproducability.

If turn on VPN on my Mac then instantly the developer certificates within the keychain become untrusted. But that's not all, if VPN is then turned off the certificates do not revert to their trusted status but remain untrusted.

The consequence of this is that if VPN is turned on and then a build is perform, it fails, and the only way to fix things is to delete the cert(s) from they keychain and re-install them.

As a remote worker, having this happen several or even dozens of times a day is incredibly annoying and frustrating.

This issue has been occurring for literally years, sometimes it occurs very often, others while its quiet for a while, and has spanned multiple versions of Xcode and Mac OS. So whatever the cause is its endemic. It doesn't just affect myself, but all the members in my development team.

I'm currently using Xcode 14.1 RC 2 and Monterey but I've seen this issue occur with many versions of Xcode and Mac. (I'm using Cisco AnyConnect Secure Mobility Client).

If the VPN is somehow interfering/affecting the connectivity aspect when an attempt is made by Xcode to validate the certificate, then why does it not rectify itself after turning off VPN?

This is so so so so annoying.

Can somebody please comment on why this happens and if there's a way to prevent it.

Post not yet marked as solved Up vote post of mungbeans Down vote post of mungbeans
1.6k views
Posted by
mungbeans
Reply
Add a Comment

Replies

If the VPN is somehow interfering/affecting the connectivity aspect when an attempt is made by Xcode to validate the certificate, then why does it not rectify itself after turning off VPN?

Probably because the system caches negative results aggressively.

It doesn't just affect myself, but all the members in my development team.

…

I've seen this issue occur with many versions of Xcode and Mac.

Right. So the one common factor is your VPN setup. Speaking as someone who’s been working remotely for 20-ish years, and who’s never seen this problem, I think it’s safe to assume that your specific VPN setup is the issue here (-: I suspect that macOS is reaching out to check the CRL [1] or OCSP status and your VPN setup is causing that to fail.

I recommend that you start by trying to isolate this from Xcode [2]. If you sign some code — it doesn’t really matter what code — with your Apple Development signing identity using codesign while the VPN is up, does that trigger the same problem?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] To be clear, modern versions of macOS do not use CRL checks.

[2] Because Xcode is huge and do all sorts of weird and wonderful things.

Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo

  • @eskimo. Thanks for the reply. "Modern versions of macOS do not use CRL checks". I currently have Monterey 12.5.1, so presumably that's not doing a CRL check?

    The problem actually manifests before even attempting to do any code signing - if I open the keychain and click on a dev cert it says its valid, then if I connect the VPN and then click on another cert and then back to the first then it says not valid. So presumably Xcode is not even entering the picture here at this point?

    —
    mungbeans
Add a Comment

I'm facing this exact same problem. And the only solution is to delete the certificate from Keychain Access, which lets Xcode prompt me to revoke it, then Xcode generates a trusted one the next time, then the cycle repeats and it happens every time.

I don't understand the root cause and seem to be stuck to where @mungbeans latest findings are.

Posted by
alobaili
Up vote reply of alobaili Down vote reply of alobaili
Add a Comment

My team came up with a "certificate kick" to workaround this.

Open Keychain Access.app Double-click on the said certificate Change to Always Trust, close the panel Open the certificate again Change it back to Use System Defaults Close the panel.

This is annoying but these steps works _(ツ)_/¯ At least it's better than rebooting or revoke and get new certificates.

I wish there is a way to script this. (or just fix whatever needs to be fixed somewhere) :).

Posted by
julian99
Up vote reply of julian99 Down vote reply of julian99
Add a Comment

I wish there is a way to script this.

The trust settings in Keychain Access are available both at the API level and from the security command line tool. See:

Normally I recommend that folks stay away from this stuff — well, from trust settings entirely — but in this case they might help with the workaround.

or just fix whatever needs to be fixed somewhere

Apropos that, has anyone filed a bug about this? If so, what was the bug number?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo
Add a Comment