creating python modules that can call back to python

I'm doing the python dynamic module loading dance.

As you likely know, this is not the typical shared library scenario described on all the docs, where an executable is linked against an existing shared lib. Instead cythonized python modules need to call back to the interpreter that loaded them for various functions. Thus, when creating the interpreter, some manner of symbol export must happen, which can then later be linked into a module's dynamic lib to resolve interpreter symbols.

With GNU/GCC this is all magic with the -shared flag (perhaps GNU's weak symbol feature, unsupported on macOS, helps out?). With a MingW GCC variant on Windows -Wl,--output-def python.def is required, followed by dlltool -d python.def -l python.dll.a, which creates some object thingy that can be linked into the subsequent module .so files. I played with lipo python -create -output libpython.dylib on macOS, but saw that no one else uses this and abandoned it.

The python3 on Darwin uses clang -bundle -undefined dynamic_lookup, but when I roll my own custom interpreter, building a module by hand with the above gives: ld: warning: -undefined dynamic_lookup may not work with chained fixups ... whatever that means.

Link args of -bundle -bundle_loader ../python result in NO errors or warnings, but my interpreter cannot load these module .so files either, probably due to some error I have not yet unmasked.

I am hoping to find a solution that works for both Xcode and Homebrew, and that will last for a decade or so, but for now would be happy getting something to work today.

Replies

I’m not 100% confident I understand your problem here, but I think the idea is:

  1. You want to create an executable.

  2. That executable loads a plug-in at runtime using dlopen.

  3. You want the plug-in to be able to import symbols from the executable.

Is that right?

If so, there are a few ways to do this on Apple platforms but the easiest option is:

  • Move the symbols that you want the plug-in to be able to access into a dynamic library.

  • Give it an rpath-relative install name, per Dynamic Library Standard Setup for Apps.

  • Export your symbols from that dynamic library. The ld man page explains how to control what symbols get exported from a dynamic libarry.

  • Give the plug-in developers a copy of that dynamic library or, better yet, a stub library (.tbd). An Apple Library Primer explains stub libraries.

  • Or, if you expect those developers to use Xcode, give them an XCFramework.

  • Have the plug-in developer link their plug-in to this dynamic library (or stub library) but not embed it.

  • Have your main executable link to and embed that dynamic library.


when I roll my own custom interpreter, building a module by hand with the above gives: ld: warning: -undefined dynamic_lookup may not work with chained fixups ... whatever that means.

See this thread, and specifically the link in the comment that I just added to my response.

The approach I’ve described will avoid this completely, but it does require you to do extra work. As I mentioned above, there “are a few ways to do this” and the approach used by Python is one of the alternatives.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  • I've followed the links you've given (thanks!) and done some more tests.

    To clarify, I have issues similar to the cpython folks (https://openradar.appspot.com/radar?id=5536824084660224), except that I am my own customer; I am the only one linking loadable module .so dynamic libraries to my interpreter. No one else will need to build a module for my interpreter to load - which may simplify things a bit (I don't think I'll need a framework).

    (continued... after stupid posting length rules)

  • I've tried a lot of flags, -no_fixup_chains, -no_compact_unwind, -undefined suppress , -undefined dynamic_lookup. -bundle with -bundle_loader <my python interpreter>, and also placing all 300+ symbols on the link command line with -Wl,-U,<sym>. -bundle -bundle_loader <python> -Wl,-no_compact_unwind and -undefined dynamic_lookup -Wl,-no_compact_unwind -Wl,-no_fixup_chains can produce a loadable module .so without errors (and all of the above with and without -flat_namespace), ....

  • ... but my interpreter cannot load either at runtime. Most frustrating is that I don't see any errors. If there are missing symbols, I should be able to find something, right? Is there a system log, or something like LD_DEUBG, that can provide a bit more information?

    When linking the interpreter, I assume I need -Wl,-export_dynamic so the system dyld(?) loader can find and resolve the loaded module .so against the interpreter ... but clearly I'm still missing something.

Add a Comment

(sorry, missed the reply vs comment UI - that last set of comments should have been a reply - I posted both (duplicate), reviewers chose the former :( and looks like we don't get to delete or edit.)