NASM x86_64: File opening (SYS_OPEN) error list? - linux

Question

I'm coding a linux x64 assembly program that read a file and I want to handle errors like File Not Found or permission errors.
Where can I find a list of SYS_OPEN error codes?
Approaches to find codes (kinda fun)
My code to open a file:
SYS_OPEN equ 2
O_RDONLY equ 0
section .data
filename db "file.txt", 0
section .text
global _start
_start:
mov rax, SYS_OPEN
mov rdi, filename
mov rsi, O_RDONLY
mov rdx, 0644o
syscall
[...]
When the file is successfully opened the RAX register points to the file descriptor (positive integer), if fails RAX point to an error (negative integer). I managed to raise a permission error by removing all permissions for all users:
chmod 0000 file.txt
This causes an error with code -13. By deleting the file, I managed to get error -2. Where can I find a list of SYS_OPEN error codes?
PS: Maybe my googling skills are rusty

Answer 1

Linux system call return values from -4095 to -1 are -errno codes. (The actual highest error number that Linux has actually defined is currently about 133, EHWPOISON, but that's the official range.)
strace ./myprog can decode them for you so you don't need to actually write error checking in your toy programs when playing around with system calls.
For example:
$ strace touch /tmp/xyjklj/bar
... (dynamic linker / process startup stuff)
openat(AT_FDCWD, "/tmp/xyjklj/bar", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = -1 ENOENT (No such file or directory)
utimensat(AT_FDCWD, "/tmp/xyjklj/bar", NULL, 0) = -1 ENOENT (No such file or directory)
... (more system calls as touch(1) finds a locale-specific set of error messages and prints
(The -1 is what the libc wrapper function actually returns; the errno code is what strace decoded from the asm syscall return value, which the glibc wrapper will store in errno. When using raw system calls in asm, you don't have to waste instructions doing that. But strace will still say "-1", not the numeric error code)
Documentation of most ways SYS_open can fail
Each system call man page documents which error codes that particular system call can fail with, and in which cases that can happen. (Those list aren't fully exhaustive, for example not covering weird things a specific filesystem like NFS could return, like EMULTIHOP (see comments).)
For your case, see the ERRORs section of the open(2) man page. e.g. there are several entries for ENOENT, covering all the cases which can lead to that return value.
ENOENT - O_CREAT is not set and the named file does not exist.
ENOENT - A directory component in pathname does not exist or is a
dangling symbolic link.
ENOENT - pathname refers to a nonexistent directory, O_TMPFILE and
one of O_WRONLY or O_RDWR were specified in flags, but
this kernel version does not provide the O_TMPFILE
functionality.
(Spoiler alert, 2 is ENOENT, so -2 is -ENOENT.)
There are of course lots of other fun ways that pathname and file access stuff (and open(2) in particular) can error, including:
EACCES (-13) - The requested access to the file is not allowed, or search
permission is denied for one of the directories in the
path prefix of pathname, or the file did not exist yet and
write access to the parent directory is not allowed. (See
also path_resolution(7).)
EFAULT - pathname points outside your accessible address space.
ENAMETOOLONG -
pathname was too long.
EBUSY - O_EXCL was specified in flags and pathname refers to a
block device that is in use by the system (e.g., it is
mounted).
[this would require root, otherwise you'd get EACCESS]
ETXTBSY - pathname refers to an executable image which is currently
being executed and write access was requested.
EWOULDBLOCK -
The O_NONBLOCK flag was specified, and an incompatible
lease was held on the file (see fcntl(2)).
ENODEV - pathname refers to a device special file and no
corresponding device exists. (This is a Linux kernel bug;
in this situation ENXIO must be returned.)
ELOOP - Too many symbolic links were encountered in resolving
pathname.
EISDIR - pathname refers to a directory and the access requested
involved writing (that is, O_WRONLY or O_RDWR is set).
ENOTDIR -
A component used as a directory in pathname is not, in
fact, a directory, or O_DIRECTORY was specified and
pathname was not a directory.
EPERM - The O_NOATIME flag was specified, but the effective user
ID of the caller did not match the owner of the file and
the caller was not privileged.
As well as various limits like number of open files (ENFILE, EMFILE), or ENOSPC disk space full. The above is not a complete list, I just took one each the ways to get many (but not all) of the error codes.
As per funnydman's answer, you can look up the number -> symbolic meaning of error values in man pages. Or look in /usr/include/asm-generic/errno-base.h (The full path may differ on some systems, and you'd only include this file indirectly, via #include <errno.h>)

Answer 2

You can interpret this as values of errno, here is the table (to list all of the codes use errno -l), also take a look at the docs. A part of the table:
number
hex
symbol
description
2
0x02
ENOENT
No such file or directory
13
0x0d
EACCES
Permission denied
There is described a reason of such decision: https://stackoverflow.com/a/6008711/9926721