4

How do I properly configure pkgconf and libffi to allow the python3 build process to correctly use my libffi version at every step of the build process, in order to import the _ctypes module correctly? Which piece am I missing here?

Some background

I am trying to build Python3 from source to build a GUI with PyQt5, and one of the requirements is a functional libffi-dev library. I don't have sudo permission on this SLES11 machine or write access to the typical library directory, nor are those likely to be granted. I reached out to the team who manages the machines, but they were unwilling to update the whole system set of libraries for my project. I have to resort to building ~30 dependencies from source, and I have little experience in this, but I'm familiar with the configure-->make-->install process. I'm stuck on one final piece of the process (failure to import the _ctypes module), which can be traced back to a missing symbol from the FFI library.

*** WARNING: renaming "_ctypes" since importing it failed: 
build/lib.linux-x86_64-3.9/_ctypes.cpython-39-x86_64-linux-gnu.so: 
undefined symbol: ffi_prep_cif

The libffi library built and installed without any issues, and I can see the files in the local library path, so I have reached the conclusion that there is either a mismatch between the libffi version and another dependency, or pkgconf is unable to locate the library. Based on my observations of the behavior of pkgconf when isolated and instructed to validate the libffi.so file, it is most likely the latter. But, I am virtually a complete novice with this, I've been at this whole build for about a week now, and I'm here typing this question, so I'm clearly open to hearing some other ideas!

Some useful debug

pkgconf --version 1.7.3 https://distfiles.dereferenced.org/pkgconf/pkgconf-1.7.3.tar.gz
libffi 3.3 ftp://sourceware.org/pub/libffi/libffi-3.3.tar.gz
Python 3.9.1 https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tgz

I provided the options to specify a local library directory while making the pkgconf source

./configure --prefix=$HOME/LIBRARIES/--with-system-libdir=$HOME/LIBRARIES/lib:$HOME/LIBRARIES/lib64:/usr/lib:/lib --with-system-includedir=$HOME/LIBRARIES/include:/usr/include

My PKG_CONFIG, PKG_CONFIG_PATH, LD_LIBRARY_PATH, LDFLAGS, and PATH are updated to reflect where the libffi pc files and pkgconf files are located

$ echo $PKG_CONFIG
$HOME/LIBRARIES/bin/pkgconf
$ echo $PKG_CONFIG_PATH
$HOME/LIBRARIES/lib/pkgconfig:$HOME/LIBRARIES/lib64/pkgconfig
$ echo $LD_LIBRARY_PATH
$HOME/LIBRARIES/lib:$HOME/LIBRARIES/lib64
$ echo $LDFLAGS
-L$HOME/LIBRARIES/lib64/ -L$HOME/LIBRARIES/lib
$ echo $PATH
$HOME//LIBRARIES/bin:/usr/local/bin:/usr/bin
$ ls $HOME/LIBRARIES/lib64/libff*
libffi.a libffi.la libffi.so libffi.so.7 libffi.so.7.1.0

AND YET

pkgconf --validate validation of the library appears to fail, and the Python3 make script notes the undefined symbol. I'm more concerned about the make script; I'm not sure whether pkgconf is actually supposed to error out here.

Update here: the library is valid according to pkgconf, so this rules out that suspicion. Thank you, telcoM

pkgconf --validate libffi 
$HOME/LIBRARIES/lib/pkgconfig/libffi.pc:9

Adding the configure command for Python3 for clarity

./configure --prefix=$HOME/LIBRARIES --enable-shared --with-system-ffi=$HOME/LIBRARIES/lib
muru
  • 69,900
  • 13
  • 192
  • 292
szofar
  • 43
  • 6

3 Answers3

2

I had exactly the same problem.

build/lib.linux-x86_64-3.9/_ctypes.cpython-39-x86_64-linux-gnu.so

would be generated by make, but wasn't linked with libffi (as I found out with ldd). When, subsequently, make runs setup.py, I'd get exactly the same error:

*** WARNING: renaming "_ctypes" since importing it failed:
build/lib.linux-x86_64-3.9/_ctypes.cpython-39-x86_64-linux-gnu.so:
undefined symbol: ffi_prep_cif

Following modules built successfully but were removed because they could not be imported:
_ctypes

But in my case, exporting C_INCLUDE_PATH wasn't the problem. The problem was that _ctypes.cpython-39-x86_64-linux-gnu.so was compiled without -lffi. I had to hack setup.py by adding the line

ext.libraries.append('ffi')

at the end of the definition of the function

def detect_ctypes(self):

For the record, I ran the configure script with

CPPFLAGS="-I/my/path/include" LDFLAGS="-Wl,-rpath=/my/path/lib64 -Wl,-rpath=/my/path/lib" ./configure --prefix=/my/path --build=x86_64-redhat-linux --enable-shared --enable-optimizations

I'm not sure if explicit CPPFLAGS and LDFLAGS would be necessary in all cases.

SU3
  • 174
  • 1
  • 1
  • 10
  • This sounds like a better solution. In my case at the time, the cppflags I used may have included some system packages set up by the admin, so I probably lucked out and libffi might have been linked correctly somehow in one of those in a convoluted way. I haven’t revisited this in a while, but I will try to take a look this week – szofar Sep 06 '21 at 01:55
  • Worked like a charm, thanks for the hack! – szofar Sep 14 '21 at 17:54
  • For my case, compiling Python 3.9.7, I simply had to tweak `LDFLAGS` : `../Python-3.9.7/configure --with-pydebug --prefix=$HOME/.local LDFLAGS="-L $HOME/.local/lib64 -Wl,-rpath $HOME/.local/lib64" ` – philb Oct 06 '21 at 21:11
  • Any idea why this hack has to be done, and doesn't get taken care of by python's build logic automatically? (It did work for me! I had to set the LIBRARY_PATH variable apart from LD_LIBRARY_PATH though) – Parth Thakkar Sep 14 '22 at 18:56
1

You should specify just the name of the library package for pkgconf --validate, not the pathname to a .so library file. In other words, try:

pkgconf --validate libffi
telcoM
  • 87,318
  • 3
  • 112
  • 232
  • Thank you! It appears pkgconf can properly locate the libffi library package. and I edited that in the debug steps above. This rules out the pkgconf piece of the issue, but I still see the "undefined symbol: ffi_prep_cif" in the Python3 make output. – szofar Jan 30 '21 at 15:28
0

I know OP's problem has been solved, but I had the same issue, with the solution that OP had not working for me. I went with downloading an AppImage version, which worked for me.

Minek Po1
  • 1
  • 1