One triple, many ABIs: multilib and multiarch

Source references in this chapter are to GCC 15.2.0 (releases/gcc-15.2.0).

You pass -m32. You supply no extra flags, no extra -L, nothing. And somehow the link picks up a 32-bit libgcc.a and a 32-bit libc.so, out of an installation whose default is 64-bit.

Then you pass -mcpu=cortex-m4 -mfloat-abi=hard on a bare-metal toolchain and the same magic fails: the link succeeds, and the resulting binary is built against the wrong float ABI.

One target triple can have several mutually incompatible ABIs, and multilib is how one toolchain ships a separate copy of every target library for each of them. The mechanism is a four-way expansion of every search prefix, plus one fallback pass that is the reason the second case fails quietly rather than loudly.

The problem, in the manual's words

Write options that are mutually incompatible side by side, separated by a slash. Write options that may be used together separated by a space. The build procedure will build all combinations of compatible options.

For example, if you set MULTILIB_OPTIONS to m68000/m68020 msoft-float, Makefile will build special versions of libgcc.a using the following sets of options: -m68000, -m68020, -msoft-float, -m68000 -msoft-float, and -m68020 -msoft-float.

— gcc/doc/fragments.texi:63-83

Five variants of libgcc.a from two lines of makefile. And not just libgcc: libstdc++, libatomic, newlib's libc.a, every crt*.o — each has to be built and installed once per variant, and the driver has to pick the matching set on every single invocation.

You mostly do not notice this working. You notice when it does not.

Which variant am I getting?

Three commands, and the distinction between them is the whole chapter:

$ gcc -print-multi-lib                  # every variant this toolchain HAS
$ gcc -m32 -print-multi-directory       # GCC-side dir for THIS invocation
$ gcc -m32 -print-multi-os-directory    # OS-side dir for THIS invocation
$ gcc -m32 -print-file-name=libgcc.a    # which variant actually won

-print-multi-lib prints one line per variant, in a format the driver's own comment describes:

The format of multilib_select is a list of elements. Each element is a subdirectory name followed by a list of options followed by a semicolon. […] A subdirectory name is optionally followed by a colon and the corresponding multiarch name.

— gcc/gcc.cc:9750-9761

so you get something shaped like:

.;
32;@m32
x32;@mx32

First field the directory, then the options that select it, @-separated.

If the variant you expect is missing from -print-multi-lib, the library was never built. No amount of -B or --sysroot will conjure it into existence. You need a toolchain built with that variant, or you need to build it. This is the single most useful check in this chapter, and it is much faster than reasoning about paths.

A short -print-multi-lib usually means --disable-multilib at configure time, which gcc -v will show you.

Two directory names, not one

Here is the asymmetry that causes most of the confusion.

The driver keeps them as three separate variables with three separate comments:

/* Subdirectory to use for locating libraries.  Set by
   set_multilib_dir based on the compilation options.  */

static const char *multilib_dir;

/* Subdirectory to use for locating libraries in OS conventions.  Set by
   set_multilib_dir based on the compilation options.  */

static const char *multilib_os_dir;

/* Subdirectory to use for locating libraries in multiarch conventions.  Set by
   set_multilib_dir based on the compilation options.  */

static const char *multiarch_dir;

Listing 7-1: the three subdirectory variables (gcc/gcc.cc:1666-1679).

Why there are two — three, with multiarch — is worth stating plainly, because it is not arbitrary. GCC's own tree may name its variants however it likes; it owns that directory. But the sysroot's layout is dictated by whoever built the distribution: /usr/lib32 on one, Debian's /usr/lib/i386-linux-gnu on another, /usr/lib64 on a third. The target's build configuration is the mapping between GCC's convention and the OS's.

The practical consequence is that when you go hunting for a missing library by hand, you have to look in both shapes. lib/gcc/<t>/<v>/32/ and <sysroot>/usr/lib32/ are the same variant.

All three are chosen once per invocation by one function, set_multilib_dir (gcc/gcc.cc:9764), which matches your command-line switches against the multilib spec. And it is a spec — the variant table is a string in the spec table like any other (gcc.cc:1733), built at run time because it can be too long for some compilers' string limits (gcc.cc:1326-1330).

In spec text, %M expands to the OS-side name, with . as the no-variant case:

	  case 'M':
	    if (multilib_os_dir == NULL)
	      obstack_1grow (&obstack, '.');
	    else
	      obstack_grow (&obstack, multilib_os_dir,
			    strlen (multilib_os_dir));

Listing 7-2: %M (gcc/gcc.cc:6805-6811).

The four-way expansion

Now the mechanism. Every prefix on the list Chapter 1.5 built is tried up to four ways, by one function, in a fixed order. This is for_each_path (gcc/gcc.cc:2778), and each attempt is a labelled block:

	  /* Look first in MACHINE/VERSION subdirectory.  */
	  if (!skip_multi_dir)
	    {
	      memcpy (path + len, multi_suffix, suffix_len + 1);

Listing 7-3: pass one, machine and version plus multilib_dir (gcc/gcc.cc:2838-2841).

	  /* Some paths are tried with just the machine (ie. target)
	     subdir.  This is used for finding as, ld, etc.  */
	  if (!skip_multi_dir
	      && pl->require_machine_suffix == 2)

Listing 7-4: pass two, just the machine (gcc/gcc.cc:2847-2850).

	  /* Now try the multiarch path.  */
	  if (!skip_multi_dir
	      && !pl->require_machine_suffix && multiarch_dir)

Listing 7-5: pass three, multiarch (gcc/gcc.cc:2858-2860).

	  /* Now try the base path.  */
	  if (!pl->require_machine_suffix
	      && !(pl->os_multilib ? skip_multi_os_dir : skip_multi_dir))
	    {
	      const char *this_multi;
	      ...
	      if (pl->os_multilib)
		{
		  this_multi = multi_os_dir;

Listing 7-6: pass four, the bare prefix plus multilib_os_dir (gcc/gcc.cc:2868-2884).

Read Listing 7-6 carefully, because it is where the two directory names finally diverge. The choice between multi_os_dir and multi_dir is made by pl->os_multilib, a per-prefix flag — the last argument to add_prefix. Look back at Listing 5-6 in Chapter 1.5: the tool directory was registered with a final argument of 1. That is what marks it as an OS-convention prefix, and therefore what makes ../lib32 rather than 32 get appended to it.

So the expansion, concretely:

$prefix/lib/gcc/<target>/<ver>/32/     ← multilib_dir, on a GCC-convention prefix
<sysroot>/usr/lib/../lib32/            ← multilib_os_dir, on an OS-convention prefix

Yes, /usr/lib/../lib32/ really is the shape of the path. .. is not normalised away, which is why -print-multi-os-directory prints things like ../lib32 and why a strace of a link shows paths that look wrong and are not.

require_machine_suffix == 2 in Listing 7-4 is the "just the machine" case, used for finding as and ld — which is how a toolchain finds $prefix/<target>/bin/as without a version component. It applies to exec_prefixes, not to libraries.

The fallback pass, and why a wrong link succeeds

The whole prefix list is walked twice: once with the multilib subdirectories, and once without.

      /* Run through the paths again, this time without multilibs.
	 Don't repeat any we have already seen.  */
      if (multi_dir)
	{
	  free (CONST_CAST (char *, multi_dir));
	  multi_dir = NULL;
	  ...
	}
      else
	skip_multi_dir = true;

Listing 7-7: the second pass (gcc/gcc.cc:2902-2913).

That second pass is a deliberate fallback. It lets a target ship only the default variant of some library rather than every combination — which is genuinely useful, because the combinatorial explosion of ARM's -mcpu and float-ABI axes is enormous.

It is also why a link can succeed while silently picking up the wrong ABI's library. The variant directory was empty, so the search fell through to the default one, and ld was handed an archive it was perfectly willing to read. Sometimes you get a diagnostic about incompatible attributes; on many targets you do not.

This is documented, and the manual states both halves:

If the osdir part begins with a !, GCC will not search in the non-multilib directory and use exclusively the multilib directory. Otherwise, the compiler will examine the search path for libraries and crt files twice; the first time it will add multilib to each directory in the search path, the second it will not.

— gcc/doc/fragments.texi:205-210

So a target that wants the fallback off — because falling through is worse than failing — prefixes its MULTILIB_OSDIRNAMES entry with !.

When you suspect this has happened, do not reason about directories. Ask which file:

$ gcc -mcpu=cortex-m4 -mfloat-abi=hard -print-file-name=libgcc.a
$ gcc -mcpu=cortex-m4 -mfloat-abi=hard -print-multi-directory

If the second prints a variant directory and the first returns a path that does not contain it, you have just watched the fallback happen.

Multiarch is a different thing

Debian's /usr/lib/x86_64-linux-gnu scheme puts one directory per triple so that several architectures' libraries coexist in one filesystem. It is orthogonal to multilib — a configuration can be multiarch without being multilibbed — but it shares the plumbing, and it gets its own pass in for_each_path (Listing 7-5).

The ordering is fixed and documented:

Each multiarch subdirectory will be searched before the corresponding OS multilib directory, for example /lib/i386-linux-gnu before /lib/../lib32.

— gcc/doc/fragments.texi:218-220

$ gcc -print-multiarch          # e.g. x86_64-linux-gnu

Note that this one prints an empty line rather than . when there is no multiarch directory (gcc.cc:8867-8874), unlike its two siblings.

Multilib and the sysroot: two independent schemes

Knowing which one your target uses matters, because they behave completely differently under -print-sysroot.

Scheme one, subdirectories inside one sysroot. The common case, and everything above. The OS-side directory is appended to paths within a single sysroot: one /arm-rootfs, containing usr/lib and usr/lib32.

Scheme two, a different sysroot per variant. Some targets compute a suffix on the sysroot itself, through SYSROOT_SUFFIX_SPEC (declared at gcc.cc:1229, with the default empty at :1193-1195). MIPS is the canonical user:

#undef SYSROOT_SUFFIX_SPEC
#define SYSROOT_SUFFIX_SPEC						\
    "/%{mmicromips:micro}mips%{mel|EL:el}-" MIPS_SYSVERSION_SPEC	\
    "%{msoft-float:-soft;:-hard}"					\
    "%{!mips32r6:%{!mips64r6:%{mnan=2008:-nan2008}}}%{muclibc:-uclibc}"

Listing 7-8: MIPS computing a sysroot suffix from your flags (gcc/config/mips/mti-linux.h:29-33).

That is ordinary spec text, using the if/else form from Chapter 1.4. So --sysroot=/opt/rootfs -EL -msoft-float really searches /opt/rootfs/mipsel-r2-soft/…. The suffix is evaluated once at startup (gcc.cc:8542-8552) and glued on inside add_sysrooted_prefix (gcc.cc:3194-3196) — the same function from Chapter 1.2, which turns out to concatenate three pieces rather than two.

Three consequences if you are on such a target:

• -print-sysroot prints the suffixed path, so its output changes with -EL or -msoft-float. Listing 3-5 in Chapter 1.3 is the code. Do not cache it.

• Your "sysroot" is not one root filesystem but a family of them, and --with-sysroot only gives the base. The suffix is invisible in the configure line, so gcc -v will not warn you.

• The same target also makes the startfile prefix multilib-conditional, through the STARTFILE_PREFIX_SPEC branch of Chapter 1.5:

  #undef STARTFILE_PREFIX_SPEC
  #define STARTFILE_PREFIX_SPEC                          \
    "%{mabi=32: /usr/local/lib/ /lib/ /usr/lib/}         \
     %{mabi=n32: /usr/local/lib32/ /lib32/ /usr/lib32/}  \
     %{mabi=64: /usr/local/lib64/ /lib64/ /usr/lib64/}"
  

  Listing 7-9: MIPS replacing the standard prefix chain entirely (gcc/config/mips/mti-linux.h:37-41).

Header search gets the same treatment through SYSROOT_HEADERS_SUFFIX_SPEC, which MIPS simply defines as the same spec (mti-linux.h:35), processed at gcc.cc:8568-8577.

Declaring the variant set

If you are building a toolchain rather than using one, the variant set comes from your target's t-* makefile fragment. The variables, all documented in fragments.texi:

A real one, and it is worth seeing the real text rather than a tidied version:

comma=,
MULTILIB_OPTIONS    = $(subst $(comma),/,$(TM_MULTILIB_CONFIG))
MULTILIB_DIRNAMES   = $(patsubst m%, %, $(subst /, ,$(MULTILIB_OPTIONS)))
MULTILIB_OSDIRNAMES = m64=../lib64$(call if_multiarch,:x86_64-linux-gnu)
MULTILIB_OSDIRNAMES+= m32=$(if $(wildcard $(shell echo $(SYSTEM_HEADER_DIR))/../../usr/lib32),../lib32,../lib)$(call if_multiarch,:i386-linux-gnu)
MULTILIB_OSDIRNAMES+= mx32=../libx32$(call if_multiarch,:x86_64-linux-gnux32)

Listing 7-10: x86-64 GNU/Linux (gcc/config/i386/t-linux64:33-38).

Three pieces of indirection in six lines, and each is instructive. The option list comes from TM_MULTILIB_CONFIG, which config.gcc sets from the triple — so x86_64-linux-gnu and i686-linux-gnu get different axes from the same fragment. if_multiarch emits the :name part only on a multiarch-enabled configuration, so the same line serves Debian and Fedora. And the m32 line probes the filesystem at build time for usr/lib32, falling back to ../lib when it is absent, with the reasoning in a comment above:

On Debian, Ubuntu and other derivative distributions, the 32bit libraries are found in /lib32 and /usr/lib32, /lib64 and /usr/lib64 are symlinks to /lib and /usr/lib, while other distributions install libraries into /lib64 and /usr/lib64. The LSB does not enforce the use of /lib64 and /usr/lib64, it doesn't tell anything about the 32bit libraries on those systems.

— gcc/config/i386/t-linux64:19-24

That fragment is compiled into the driver by a shell script, genmultilib, which generates multilib.h (gcc/Makefile.in:2476-2483) — and the driver includes it with a pointed comment about ordering:

#include "multilib.h" /* before tm.h */

Listing 7-11: (gcc/gcc.cc:37).

Prune multilibs in GCC and you prune them everywhere. newlib, and anything else built in a combined tree, takes its variant matrix straight from xgcc -print-multi-lib. A --disable-multilib GCC produces a single-variant newlib whether you asked for that or not. Part III returns to this.

What to take from this

multilib_dir and multilib_os_dir are different things with different roles, and which one a prefix gets depends on a flag set when the prefix was registered. Every prefix is tried four ways, then the whole list is tried again with no variant at all — and that last fallback is why a wrong-ABI link can succeed silently.

Before debugging paths, check that the variant exists:

$ gcc -print-multi-lib

Documentation coverage

This is the best-documented mechanism in Part I. fragments.texi covers the fragment variables thoroughly, including the two-pass walk (:205-210), the ! prefix that disables it, the multiarch ordering (:218-220) and the fact that MULTILIB_OSDIRNAMES subsumes MULTIARCH_DIRNAME (:212-216). The -print-multi-* options are in Overall Options.

What is not documented is the consequence rather than the mechanism: that the fallback pass turns a missing variant into a wrong-library link rather than an error, and that -print-search-dirs shows the list before any of this expansion happens, so its output is not the set of directories actually searched.

Next: what all these paths are searched for.

All source references in this chapter are to GCC 15.2.0 (releases/gcc-15.2.0). Line numbers in other releases will differ; the surrounding code rarely does. Where behaviour itself changed across a major version, it is flagged inline.