async/await back deployment crash in system extension

The problem

I have a MacOS app that hosts a content filtering system extension, like SimpleFirewall. The app has been in production for a couple years.

I'm working on a new version, and in testing the release candidate, I'm getting a consistent crash that I believe is related to swift concurrency back deployment. Here are the key details:

  • building using Xcode 14.2, from a machine running Monterrey, Swift 5.7.2
  • crash does not happen when building and testing from Xcode, locally
  • crash does not happen on test machine running Ventura
  • crash DOES happen always on a test machine running Big Sur
  • only the root-user system extension crashes, not the host application
  • the new version introduced async/await into the system extension
  • crash report shows identical stack trace to well-known issue that had to do with concurrency back deployment

Is there a known issue/limitation with concurrency back deployment in the context of a system extension? Is there any reason why async/await shouldn't work in that context when deployed to Big Sur?

More details, context

The key lines of the crash stack trace are:

0   libswiftCore.dylib            	0x00007fff2cdacdc7 swift::ResolveAsSymbolicReference::operator()(swift::Demangle::__runtime::SymbolicReferenceKind, swift::Demangle::__runtime::Directness, int, void const*) + 55
1   libswiftCore.dylib            	0x00007fff2cdcf2dd swift::Demangle::__runtime::Demangler::demangleSymbolicReference(unsigned char) + 141
2   libswiftCore.dylib            	0x00007fff2cdcc2a8 swift::Demangle::__runtime::Demangler::demangleType(__swift::__runtime::llvm::StringRef, std::__1::function<swift::Demangle::__runtime::Node* (swift::Demangle::__runtime::SymbolicReferenceKind, swift::Demangle::__runtime::Directness, int, void const*)>) + 168
3   libswiftCore.dylib            	0x00007fff2cdb25a4 swift_getTypeByMangledNameImpl(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 516
4   libswiftCore.dylib            	0x00007fff2cdafd6d swift::swift_getTypeByMangledName(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 477
5   libswiftCore.dylib            	0x00007fff2cdaff9b swift_getTypeByMangledNameInContext + 171
6   com.myorg.app.filter-extension	0x000000010db2b8b7 0x10db02000 + 170167
7   libdispatch.dylib             	0x00007fff20516806 _dispatch_client_callout + 8
8   libdispatch.dylib             	0x00007fff2051798c _dispatch_once_callout + 20
9   libswiftCore.dylib            	0x00007fff2cdbe16a swift_once + 26
10  com.myorg.app.filter-extension	0x000000010db2c5e3 0x10db02000 + 173539
11  com.myorg.app.filter-extension	0x000000010dbbd708 0x10db02000 + 767752
12  com.myorg.app.filter-extension	0x000000010db073cc 0x10db02000 + 21452
13  com.apple.NetworkExtension    	0x00007fff2dfdd4d8 -[NEExtensionProviderContext createWithCompletionHandler:] + 377
14  com.apple.Foundation          	0x00007fff215a7c96 __NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT_S1__ + 10
15  com.apple.Foundation          	0x00007fff21552b98 -[NSXPCConnection _decodeAndInvokeMessageWithEvent:flags:] + 2271
16  com.apple.Foundation          	0x00007fff2150a049 message_handler + 206
17  libxpc.dylib                  	0x00007fff20406c24 _xpc_connection_call_event_handler + 56
18  libxpc.dylib                  	0x00007fff20405a9b _xpc_connection_mach_event + 938

The first five lines are identical to an issue from Xcode 13.2.1, discussed in depth on the swift forums:

https://forums.swift.org/t/async-await-crash-on-ios14-with-xcode-13-2-1/54541

...except I'm using Xcode 14.2. Which makes me think that it's not exactly the same bug, but another manifestation of a failure to link against the back-deployed currency lib, possibly having to do with the fact that the system extension isn't able to access the back-deployed lib.

The archived app does have libswift_Concurrency.dylib at MyApp.app/Contents/Frameworks/libswift_Concurrency.dylib.

What I've Tried

I tested the workaround in the above mentioned thread, using lipo to remove arm64 arch, but it didn't work.

I also tested adding -Xllvm -sil-disable-pass=alloc-stack-hoisting to Other Swift settings, as suggested in https://developer.apple.com/forums/thread/697070.

I would greatly appreciate any assistance.

Up vote post of jaredh159 Down vote post of jaredh159
1.7k views
Posted by
jaredh159
Reply
Add a Comment

Accepted Reply

@ilis544 I finally got a reply back from DTS, and they provided a working solution. Here's the relevant part, for you and future googlers - I can confirm the workaround mentioned below fixed the issue for my app:

"This isn’t due to a bug with Swift Concurrency, but a side-effect of how System Extensions work, with an easy solution. Your intuition about a simple test case with the Simple Firewall sample project was correct, so I was using that project built with Xcode 14.3.1 and testing on macOS 11.4 to reproduce and test solutions.

Speaking generally of Swift Concurrency, when an app is built with Swift Concurrency code that needs to back deploy, Xcode detects this and deploys the library supporting back deployment to a standard location inside of the app bundle so that any binary inside of the app bundle can refer to that singular copy of the library. This library is weakly linked, which means that at process launch, if the system doesn’t locate the library in one of the standard locations, the system doesn’t abort the process launch due to a missing library.

In the crash here, the top frame is:

0   libswiftCore.dylib             0x00007fff2cdacdc7 swift::ResolveAsSymbolicReference::operator()(swift::Demangle::__runtime::SymbolicReferenceKind, swift::Demangle::__runtime::Directness, int, void const*) + 55

This is the Swift runtime failing to locate a symbol it needs for a some code that is executing, including in other scenarios outside of Swift concurrency. While there have been real issues that Apple has addressed with Swift Concurrency that have this frame in the crash report, what happened here is that the system couldn’t locate the Swift Concurrency symbol because we never loaded the library, because the system couldn’t find it. Since the library is weakly linked, this wasn’t a crash on launch.

System extensions are unique compared to other types of extension points because the copy inside your app is inert and not executed. When the system extension activates through the user consent process, macOS copies it from your app to a location under /Library/SystemExtensions, and then executes this copy. When the system extension process launches, the search mechanism to locate the Swift Concurrency back deployment library doesn’t find it, because the library is in a shared location inside of your app bundle but outside of the system extension files are are copied. There is no mechanism for connecting the copied system extension back to its original copy inside the app for this library loading purpose, so there is no way for the system to locate the library inside of the original app bundle.

The solution here is two build settings on the system extension target:

  • Set ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES to Yes. This will copy the Swift Concurrency library into the System Extension during the build. This means there will be two copies, one in the standard location for the app for any concurrency needs in your app code, and a second copy inside the system extension for its own needs.

  • Ensure that LD_RUNPATH_SEARCH_PATHS is not overridden. It should have its default value of $(inherited) @executable_path/../Frameworks @executable_path/../../../../Frameworks.

This second build setting is to make sure that Xcode inserts /usr/lib/swift into the right place in the search path list the system uses to locate libraries, so that on newer systems which have the Swift Concurrency library built into the macOS, your app uses that copy instead of the back-deployment library. To confirm the search list order on your system extension, you can run otool -l, and ensure that /usr/lib/swift is listed first of the multiple LC_RPATH entries."

Posted by
jaredh159
Up vote reply of jaredh159 Down vote reply of jaredh159
Post marked as solved

  • Thanks for closing the loop here.

    —
    eskimo
Add a Comment

Replies

Hello everyone.

We have got the same issue on BigSur. I would like to join this thread and receive any reply from Apple Support :)

Posted by
ilis544
Up vote reply of ilis544 Down vote reply of ilis544
Add a Comment

I would like to join this thread and receive any reply from Apple Support

Keep in mind that DevForums isn’t a formal support mechanism. See tip 1 in Quinn’s Top Ten DevForums Tips. Apple offers formal code-level support via DTS tech support incidents (TSI).

I can’t say whether jaredh159 opened a TSIs for this; if they did, it’d be nice if they could post a summary of their conclusions here.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo
Add a Comment

I can’t say whether jaredh159 opened a TSIs for this; if they did, it’d be nice if they could post a summary of their conclusions here.

I did open a TSI, but because of WWDC and some personal time off, not much progress has been made yet. The only thing notable so far is that I was asked to retry with Xcode 14.3.1 (I had been on 14.2, because I hadn't upgraded to Ventura). I did retry with 14.3.1, but the same crash manifested the exact same way. Still waiting for a response to that. I'll update this thread if there's anything useful I find out.

Posted by
jaredh159
Up vote reply of jaredh159 Down vote reply of jaredh159
Add a Comment

@eskimo @ilis544 I guess I can also add this: since I first posted this, I modified my filter-side network extension code to remove every single async await (basically to be an empty FilterDataProvider), and rebuilt, notarized, and tested on Big Sur -- no crash.

Then I added one single Task { ... } and tried again, it immediately crashed. So I'm pretty much totally confident it's a swift concurrency back deployment issue.

I told the TSI engineer that I'm almost certain they could repro using the SimpleFirewall sample app from "Filtering Network Traffic". All they'd need to do is add a single Task with an os_log or something, build in release mode, and test on a machine running Big Sur.

Posted by
jaredh159
Up vote reply of jaredh159 Down vote reply of jaredh159
Add a Comment

@ilis544 I finally got a reply back from DTS, and they provided a working solution. Here's the relevant part, for you and future googlers - I can confirm the workaround mentioned below fixed the issue for my app:

"This isn’t due to a bug with Swift Concurrency, but a side-effect of how System Extensions work, with an easy solution. Your intuition about a simple test case with the Simple Firewall sample project was correct, so I was using that project built with Xcode 14.3.1 and testing on macOS 11.4 to reproduce and test solutions.

Speaking generally of Swift Concurrency, when an app is built with Swift Concurrency code that needs to back deploy, Xcode detects this and deploys the library supporting back deployment to a standard location inside of the app bundle so that any binary inside of the app bundle can refer to that singular copy of the library. This library is weakly linked, which means that at process launch, if the system doesn’t locate the library in one of the standard locations, the system doesn’t abort the process launch due to a missing library.

In the crash here, the top frame is:

0   libswiftCore.dylib             0x00007fff2cdacdc7 swift::ResolveAsSymbolicReference::operator()(swift::Demangle::__runtime::SymbolicReferenceKind, swift::Demangle::__runtime::Directness, int, void const*) + 55

This is the Swift runtime failing to locate a symbol it needs for a some code that is executing, including in other scenarios outside of Swift concurrency. While there have been real issues that Apple has addressed with Swift Concurrency that have this frame in the crash report, what happened here is that the system couldn’t locate the Swift Concurrency symbol because we never loaded the library, because the system couldn’t find it. Since the library is weakly linked, this wasn’t a crash on launch.

System extensions are unique compared to other types of extension points because the copy inside your app is inert and not executed. When the system extension activates through the user consent process, macOS copies it from your app to a location under /Library/SystemExtensions, and then executes this copy. When the system extension process launches, the search mechanism to locate the Swift Concurrency back deployment library doesn’t find it, because the library is in a shared location inside of your app bundle but outside of the system extension files are are copied. There is no mechanism for connecting the copied system extension back to its original copy inside the app for this library loading purpose, so there is no way for the system to locate the library inside of the original app bundle.

The solution here is two build settings on the system extension target:

  • Set ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES to Yes. This will copy the Swift Concurrency library into the System Extension during the build. This means there will be two copies, one in the standard location for the app for any concurrency needs in your app code, and a second copy inside the system extension for its own needs.

  • Ensure that LD_RUNPATH_SEARCH_PATHS is not overridden. It should have its default value of $(inherited) @executable_path/../Frameworks @executable_path/../../../../Frameworks.

This second build setting is to make sure that Xcode inserts /usr/lib/swift into the right place in the search path list the system uses to locate libraries, so that on newer systems which have the Swift Concurrency library built into the macOS, your app uses that copy instead of the back-deployment library. To confirm the search list order on your system extension, you can run otool -l, and ensure that /usr/lib/swift is listed first of the multiple LC_RPATH entries."

Posted by
jaredh159
Up vote reply of jaredh159 Down vote reply of jaredh159
Post marked as solved

  • Thanks for closing the loop here.

    —
    eskimo
Add a Comment

Hey everyone,

@jaredh159’s answer helped me fix the can't resolve symbol _swift_defaultActor_initialize in /Applications/<REDACTED>/Contents/MacOS/<REDACTED> because dependent dylib @rpath/libswift_Concurrency.dylib could not be loaded crash that had occurred on macOS Big Sur (possibly Catalina too—yet to be confirmed). Thanks!

I’ll add a few details specific to my case because it seems to me that Xcode may have issues with detecting whether back-deployment is needed.

  • Native Cocoa app
  • Deployment target: macOS 10.15
  • libswift_Concurrency.dylib is supposed to be back-deployed
  • The library was in fact injected into the .app package (Contents/Frameworks/libswift_Concurrency.dylib), but the executable didn’t attempt to call it (I checked with otool -l)
  • Both Xcode 14.3.1 and 15 beta 6 had the following default value for LD_RUNPATH_SEARCH_PATHS (at target level): $(LD_RUNPATH_SEARCH_PATHS_$(IS_MACCATALYST)) (I don’t remember seeing this value ever before)
  • The crash stopped occurring after I changed LD_RUNPATH_SEARCH_PATHS’s value to $(inherited) @executable_path/../Frameworks @executable_path/../../../../Frameworks (from jaredh159’s answer)
  • Changing the ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES value (it had the default $(EMBEDDED_CONTENT_CONTAINS_SWIFT) value) didn’t seem to result in any change

I’m concerned with Xcode failing to apply the correct build settings automatically, even though it did copy the library. Could this be a bug?

CC @eskimo

Posted by
XMaster
Up vote reply of XMaster Down vote reply of XMaster
Add a Comment

Does your app use a system extension? Because jaredh159’s issue was specifically related to system extensions.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo
Add a Comment

No, it’s just a regular Cocoa app, no extensions—but similar symptoms.

Posted by
XMaster
Up vote reply of XMaster Down vote reply of XMaster
Add a Comment

it’s just a regular Cocoa app

OK, then this is a different problem. I recommend that you start a new thread about it. Tag it with Concurrency so that I see it.

Before you do that, however, try reproducing the issue with a small test project. That’ll inform you as to whether this is a general Xcode issue or something specific to how your main project is set up.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo
Add a Comment