Cross-Compilation

Someone once asked me: “Why do I need this complicated cross-compilation environment. Why can’t I just build the program on my Ubuntu Laptop, copy it to my Android phone and run it there?”

I’m sure you have thought about what a compiler does, it makes a binary from source code. But what is actually in that binary depends a lot on where it will be run.

CPU Instruction Set

First there is the CPU instruction set, like x86_64 or aarch64. As the name says, this defines what instructions your CPU can understand. So the first reason the program you built on/for your laptop can’t run on your phone just like that, is that they probably have very different CPUs, and use different instruction sets. The compiler has to build the binary specifically with the instructions your CPU can process.

For example when you consider the following C code:

int i = 0;
i++;

In assembly, it will (simplified) translate to something containing this:

x86_64:

c7 45 fc 00 00 00 00    movl   $0x0,-0x4(%rbp)
83 45 fc 01             addl   $0x1,-0x4(%rbp)

aarch64:

52800000       mov      w0, #0x0
11000400       add      w0, w0, #0x1

The left column is the actual binary, written as hexadecimal. The assembly instructions that are meant by these bytes are added on the right. One could notice is that x86 instructions have variable length while the aarch64 instructions all have the same length. But also the instructions for setting a register or incrementing it have totally different binary representations.

Kernel / syscall ABI

But even when two computers share the same CPU instruction set, they can have other differences. For example consider an Intel (x86_64) Mac, an x86_64 PC running Linux and an x86_64 PC running Windows. They all have the same CPU instruction set, but if you build a binary on/for one of them, it will still not run on the others without some compatibility layer.

One reason for this might be because of some differences when communicating with the operating systems kernel. For example, printing some text requires talking to the kernel and on x86_64-apple-darwin it might contain these instructions:

b8 04 00 00 02                 mov     $0x2000004,%eax   ; sys_write syscall
bf 01 00 00 00                 mov     $0x1,%edi         ; file handle 1 -> stdout
48 be 00 10 00 00 01 00 00 00  movabs  $0x100001000,%rsi ; address of the message
ba 0e 00 00 00                 mov     $0xf,%edx         ; length of the message
0f 05                          syscall

Where 0x2000004 is the number of the sys_write syscall. 0x100001000 is the address of the text to be printed.

On x86_64-pc-linux-*, the same thing might look like this:

b8 01 00 00 00                 mov     $0x1,%eax         ; write syscall
bf 01 00 00 00                 mov     $0x1,%edi         ; file handle 1 -> stdout
48 be 00 20 40 00 00 00 00 00  movabs  $0x402000,%rsi    ; address of the message
ba 0d 00 00 00                 mov     $0xe,%edx         ; length of the message
0f 05                          syscall

Where 1 is the number of the write syscall on linux (see, totally different than Mac OS!), and 0x402000 is the address of the text to be printed (different because of the way the executable is put together).

On x86_64-windows-* the way to do syscalls is completely different, but I won’t go into details here.

External libraries / linking

It is also important to note, that when building for other systems all external libraries need to be built for the correct target architecture. By default your system might dynamically link against shared libraries. This means the executable file produced does not actually contain all the code needed to run the program, but instead some parts are loaded from .so files when you run the program. If you copy the dynamically linked executable to another system, that system must have compatible .so files for all the libraries used.

You might find it easier to statically link the executable. This means including the libraries that are required to run the program in one executable, so fully statically linked executables should be able to run on all systems with the instruction set and kernel they were built for.

However, not all libraries can be statically linked. For example, on various systems you are expected to use the system libc instead of bringing your own, or certain libcs might not like being statically linked. This is why many Rust and Golang binaries might not run across systems of the same kernel and instruction set. To build fully-static binaries on Linux, one needs to build against musl libc or uClibc instead of glibc. The details of this restriction are again out of scope for this post, but I might write another one specifically on that topic.

Conclusion

So now you know, why you need cross-compilers; A normal compiler will usually produce binaries for the environment it’s running on. If you want to build something on your x86_64 Laptop and run it on your aarch64 phone later, you will need a cross-compiler to specifically produce instructions compatible with your phone’s environment.