When Make Can't Find Your Executable

I recently encountered a puzzling issue: zig build worked perfectly, but make build failed with "Permission denied". What followed was an interesting debugging session that uncovered a quirky interaction between GNU Make's PATH search and directory structures.

This article documents the investigation process and some debugging techniques that proved useful along the way.

The Mystery

I had the world's simplest Makefile:

build:
	zig build

Running zig build directly? Works great:

$ zig build
# ... builds successfully ...

Running make build? Hmm, not so much:

$ make build
make: zig: Permission denied
make: *** [Makefile:2: build] Error 127

Permission denied? That seemed odd. The zig executable definitely had execute permissions, and I could run it just fine from the shell. Something else was going on here.

First instinct — check the obvious

My first thought was to check the Makefile itself. Make's requirement for tabs instead of spaces has caught me before, so I verified:

$ cat -A Makefile
build:$
^Izig build$

The ^I shows it's a proper tab (not spaces), and the $ shows line endings. Everything looks correct.

I also verified that zig was properly in my PATH and executable:

$ which zig
/home/sam/bin/zig/zig

$ zig version
0.15.0-dev.471+369177f0b

Everything looked correct. The zig command worked fine when called directly, so the issue had to be specific to how Make was executing it.

Enter strace, the system call detective

When something works in one context but not another, strace becomes invaluable. It shows every system call a program makes, which is perfect for understanding these kinds of environmental differences.

$ strace -f -e trace=execve,execveat,access,stat,openat make build 2>&1 | grep -A2 -B2 "zig"

The flags serve specific purposes:

• -f: Follow child processes (necessary since Make spawns shells)
• -e trace=...: Filter to only show specific system calls
• 2>&1: Redirect stderr to stdout for easier grepping

The output revealed the issue:

openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
openat(AT_FDCWD, "Makefile", O_RDONLY)  = 3
zig build
access("/home/sam/bin/zig", X_OK)       = 0
strace: Process 4864 attached
[pid  4864] execve("/home/sam/bin/zig", ["zig", "build"], 0x5d803c4a6260 /* 47 vars */) = -1 EACCES (Permission denied)
[pid  4864] +++ exited with 127 +++
make: zig: Permission denied
make: *** [Makefile:2: build] Error 127

This was surprising: Make was trying to execute /home/sam/bin/zig, but which zig returned /home/sam/bin/zig/zig. I needed to check what was actually at /home/sam/bin/zig:

$ ls -la /home/sam/bin/zig
total 166040
drwxr-xr-x  4 sam sam      4096 May  6 06:57 .
drwxr-xr-x  3 sam sam      4096 May  9 10:49 ..
-rw-r--r--  1 sam sam      1080 May  6 06:57 LICENSE
-rw-r--r--  1 sam sam      5853 May  6 06:57 README.md
drwxr-xr-x  2 sam sam      4096 May  6 06:57 doc
drwxr-xr-x 15 sam sam      4096 May  6 06:57 lib
-rwxr-xr-x  1 sam sam 169990400 May  6 06:57 zig

Ah! /home/sam/bin/zig was a directory containing the zig executable. Make was attempting to execute the directory itself, which explained the permission denied error.

The PATH Investigation

This suggested a PATH configuration issue. I examined my PATH more closely:

$ echo $PATH | tr ':' '\n' | grep -n bin
1:/home/sam/.pyenv/bin
2:/home/sam/.local/bin
3:/usr/local/go/bin
4:/home/sam/bin         # <-- Here
5:/home/sam/bin/zig     # <-- And here!
6:/home/sam/.bun/bin
10:/home/sam/.pyenv/bin
11:/usr/local/sbin
12:/usr/local/bin
13:/usr/sbin
14:/usr/bin
15:/sbin
16:/bin
# ... and many more ...

Aha! Both /home/sam/bin and /home/sam/bin/zig are in my PATH. When Make searches for zig, it's finding the directory /home/sam/bin/zig before it finds the actual executable at /home/sam/bin/zig/zig.

But wait, what's actually in /home/sam/bin?

$ ls -la /home/sam/bin/
total 100952
drwxr-xr-x  3 sam sam     4096 May  9 10:49 .
drwxr-xr-x 52 sam sam     4096 May 29 19:47 ..
-rwxr-xr-x  1 sam sam 31212126 Nov 17  2023 cloud-sql-proxy
drwxr-xr-x  4 sam sam     4096 May  6 06:57 zig
-rw-r--r--  1 sam sam 50434148 May  6 07:13 zig-linux-x86_64-0.15.0-dev.471+369177f0b.tar.xz
-rwxr-xr-x  1 sam sam 17991552 Jan  1  1970 zls
-rw-r--r--  1 sam sam  3707392 May  8 21:56 zls-linux-x86_64-0.15.0-dev.98+09794038.tar.xz

So I had a zig directory inside /home/sam/bin/. When Make searched PATH, it was finding this directory and mistaking it for the executable.

Comparing Behaviors — Make vs Shell

But why did it work when running zig directly? I traced the shell's behavior to compare:

$ strace -f -e execve /bin/sh -c 'zig build' 2>&1 | grep execve
execve("/bin/sh", ["/bin/sh", "-c", "zig build"], 0x7ffc80a20c98 /* 44 vars */) = 0
[pid  4956] execve("/home/sam/bin/zig/zig", ["zig", "build"], 0x5c3be2732cd8 /* 44 vars */) = 0

The shell found the correct executable at /home/sam/bin/zig/zig. Clearly, Make and the shell were using different PATH search algorithms.

I checked which shell Make was using:

$ make -p -f/dev/null | grep '^SHELL'
SHELL = /bin/sh
make: *** No targets.  Stop.

$ ls -la /bin/sh
lrwxrwxrwx 1 root root 4 Mar 31  2024 /bin/sh -> dash

Make was using dash (Debian Almquist Shell) by default. I tested whether dash itself had the same issue:

$ /bin/sh -c 'which zig && zig version'
/home/sam/bin/zig/zig
0.15.0-dev.471+369177f0b

Dash found zig correctly when run directly. The issue was specific to Make's command execution mechanism.

Let's understand the behaviour with test programs

To better understand PATH searching behavior, I wrote a small C program to test how access() behaves with directories:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

int main() {
    char *path = getenv("PATH");
    char *cmd = "zig";
    char pathcopy[4096];
    strcpy(pathcopy, path);

    char *dir = strtok(pathcopy, ":");
    while (dir != NULL) {
        char fullpath[1024];
        snprintf(fullpath, sizeof(fullpath), "%s/%s", dir, cmd);

        if (access(fullpath, X_OK) == 0) {
            printf("Found executable: %s\n", fullpath);
        } else if (access(fullpath, F_OK) == 0) {
            printf("Found but not executable: %s\n", fullpath);
        }

        dir = strtok(NULL, ":");
    }
    return 0;
}

The output was revealing:

$ gcc debug.c -o debug && ./debug | grep zig
Found executable: /home/sam/bin/zig
Found executable: /home/sam/bin/zig/zig
Found executable: /mnt/c/Users/Sam/bin/zig

Both paths showed execute permissions, as expected — directories have execute permission to allow traversal, not because they can be executed as programs.

I tested how the standard C library's execvp function handles this situation:

#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

int main() {
    char *args[] = {"zig", "version", NULL};

    printf("Attempting execvp(\"zig\", ...)\n");
    execvp("zig", args);

    // If we get here, execvp failed
    printf("execvp failed: %s\n", strerror(errno));
    return 1;
}

$ gcc test_execvp.c -o test_execvp && ./test_execvp
Attempting execvp("zig", ...)
0.15.0-dev.471+369177f0b

So execvp worked correctly, which meant Make was implementing its own PATH search rather than using the standard library.

I also considered using ltrace to trace library function calls and see whether Make was calling execvp or similar functions, but it wasn't installed on this system. ltrace is complementary to strace - while strace shows system calls (kernel interface), ltrace shows library function calls (like execvp, malloc, printf). It would have quickly revealed whether Make was using standard library functions or implementing everything from scratch.

Digging deeper into Make's execution behavior

At that point i needed more test data, I created various test Makefiles to better understand Make's behavior:

# test2.mk
test:
	@echo "PATH search for zig:"
	@which -a zig
	@echo "Type of /home/sam/bin/zig:"
	@file /home/sam/bin/zig
	@echo "Trying to execute:"
	zig --version

$ make -f test2.mk test
PATH search for zig:
/home/sam/bin/zig/zig
Type of /home/sam/bin/zig:
/home/sam/bin/zig: directory
Trying to execute:
zig --version
make: zig: Permission denied
make: *** [test2.mk:7: test] Error 127

Notice that which -a found the correct zig, but Make still failed. I tested different execution methods:

# test3.mk
.ONESHELL:
test1:
	zig version

test2:
	/bin/sh -c 'zig version'

test3:
	exec zig version

$ make -f test3.mk test1
zig version
make: zig: Permission denied
make: *** [test3.mk:3: test1] Error 127

$ make -f test3.mk test2
/bin/sh -c 'zig version'
0.15.0-dev.471+369177f0b

$ make -f test3.mk test3
exec zig version
make: zig: Permission denied
make: *** [test3.mk:9: test3] Error 127

When Make explicitly invoked a shell with -c, it worked. The failure only occurred when Make tried to execute commands directly.

Using Make's debug mode

I also tried Make's debug mode to get more insight:

$ make --debug=v build 2>&1 | tail -20
GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'build'.
 File 'build' does not exist.
 Finished prerequisites of target file 'build'.
Must remake target 'build'.
zig build
make: zig: Permission denied
make: *** [Makefile:3: build] Error 127

While not particularly revealing, this confirmed that Make was attempting to execute zig build directly without shell interpretation.

Finally, the root cause

After all this investigation, here's what's happening:

1. GNU Make has its own PATH search implementation (it doesn't seem to be just using execvp)
2. When searching PATH, Make stops at the first match where access(path, X_OK) succeeds
3. Since /home/sam/bin/zig is a directory with execute permission (for traversal), Make thinks it found the executable
4. Make tries to execve() the directory, which fails with EACCES
5. The shell (dash/bash) is smarter - it continues searching if it finds a directory

This behavior could be considered a bug in GNU Make. From what I understand the POSIX specification indicates that implementations should skip non-regular files, but Make doesn't seem to follow this guidance. Given that this behavior has persisted for decades, it's unlikely to change, and since it's been this way for decades, it's more accurately described as a long-standing quirk.

Testing workarounds

Workaround 1: Change Make's shell

SHELL := /bin/bash

.PHONY: build
build:
	zig build

$ make build
zig build
# Success!

This works because bash handles PATH searching correctly.

Workaround 2: Use explicit shell invocation

build:
	$(shell which zig) build

$ make build
/home/sam/bin/zig/zig build
# Success!

Workaround 3: Be explicit about the shell

build:
	/bin/sh -c 'zig build'

While all of these workarounds functioned correctly, they were addressing the symptom rather than the root cause.

Other debugging techniques I used

During the investigation, I employed several other debugging techniques worth documenting:

Checking for hidden characters with hexdump

To rule out hidden character issues in the Makefile, I used hexdump:

$ hexdump -C Makefile
00000000  62 75 69 6c 64 3a 0a 09  7a 69 67 20 62 75 69 6c  |build:..zig buil|
00000010  64 0a                                             |d.|
00000012

The 09 represented a tab character (required by Make), and 0a represented newlines. No hidden characters were present.

Checking file attributes

Sometimes files have extended attributes that can cause weird issues:

$ ls -la /home/sam/bin/zig/zig
-rwxr-xr-x 1 sam sam 169990400 May  6 06:57 /home/sam/bin/zig/zig

$ file /home/sam/bin/zig/zig
/home/sam/bin/zig/zig: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

The file was a standard ELF executable with no unusual attributes.

Environment variable checks

Sometimes environment variables can affect behavior:

$ env | grep -i make
# (no output - no MAKE-specific environment variables set)

$ make -f test.mk test SHELL=/bin/bash
# This works, confirming SHELL is the key variable

Learnings

1. strace is your best friend - When something works in one context but not another, trace the system calls. The -f flag is crucial for following child processes.

2. PATH conflicts are sneaky - Having a directory with the same name as an executable in your PATH is asking for trouble. Different tools handle it differently.

3. Make is weird - Make does its own thing instead of using standard library functions. This leads to surprising behavior.

4. "Permission denied" lies - It doesn't always mean file permissions. Sometimes it means you're trying to execute something that's not a regular file.

The "real" fix

The proper solution is to clean up the PATH configuration. Having both /home/sam/bin and /home/sam/bin/zig in PATH when there's a zig directory inside /home/sam/bin creates this ambiguity.

Several solutions are available:

1. Remove /home/sam/bin from PATH (though I need other executables from there)
2. Remove /home/sam/bin/zig from PATH and create a symlink elsewhere
3. Rename the directory: mv /home/sam/bin/zig /home/sam/bin/zig-dist
4. Add SHELL := /bin/bash to Makefiles that need it

Option 3 or 4 seem most practical for my use case.

Some final thoughts

This was one of those bugs that initially seemed impossible. How could Make fail to find an executable that clearly existed? Through systematic debugging, the root cause emerged: an interaction between Make's simplistic PATH search algorithm and an ambiguous PATH configuration.

The debugging process proved quite educational. Each tool provided a different perspective on the problem:

• strace showed the actual system calls
• Custom C programs let me test specific behaviors
• Make's debug output confirmed how it was executing commands
• Simple shell commands revealed the PATH conflict

Next time you hit a weird "works here but not there" bug:

• Start with strace to see what's really happening
• Write small test programs to isolate behaviors
• Don't trust error messages - "Permission denied" might be lying to you
• Check for conflicts between what different tools think is "correct"

So if Make ever reports "Permission denied" on an executable you know works elsewhere, consider whether it might be attempting to execute a directory. It's more common than you might think!