Linux kernel system call returns -1 instead of {-1, -256} - linux

Question

I'm a kernel newbie and am facing a weird issue. I have written a proof-of-concept calculator syscall and while it works fine for most computations, it is returning -1 when the SUBTRACTION result is between -1 to -256. If someone can throw some light on what could be happening, would appreciate it. Below is the syscall code.
SYSCALL_DEFINE3(calc, int, a, int, b , char, op) {
int res_int;
switch(op) {
case '+': res_int = a + b;
break;
case '-': res_int = a - b;
break;
case '*': res_int = a * b;
break;
case '/': res_int = (a*1000) / b;
break;
}
printk(KERN_INFO "KERNEL CALC RESULT : %d %c %d = %ld",a, op, b, res_int);
return res_int;
}
EDIT:
Kernel version: Android Linux Kernel 3.10.xxx.
Platform: Nexus7 ARM EABI.
What I don't understand is why it is failing. The errno is not useful at all since it is setting -res_int to errno. Also, I don't understand why it is failing only when res_int is {-1, -256}.
a=1200, b=1300 op='-' => res_int=-100 is an example case where printk prints -100, but in my userspace app, I receive -1.
It looks like when res_int is {-1, -256}, errno is being set to -res_int.
root#android:/data/local # ./calc
Please enter request in 'num1 oper num2' format:
2.45 - 2.2
returned from syscall with res_int = 250
errno = 22, strerror(errno) = Invalid argument
Calculator result = 0.250000
root#android:/data/local # ./calc
Please enter request in 'num1 oper num2' format:
2.2 - 2.45
returned from syscall with res_int = -1
errno = 250, strerror(errno) = Unknown error 250
Calculator result = -0.001000
root#android:/data/local #

Answer 1

You didnt' mention your kernel version and platform but fwiw, from kernel's perspective a syscall usually returns zero on success and a negative number on error. Generally this is the ABI convention between kernel folks and application developers so the code can understand each other.
But user program always uses C library as a wrapper to make system calls and the C API should comply with the API standard. That is, the C library will check the return value from kernel space and set the errno based on that. (Note that errno is in user space and kernel doesn't know it.)
Say, if syscall foo(), from user's view, returns -1 on failure and should set the errno to indicate the error (ERR1 for failure 1, ERR2 for failure2, etc), the kernel syscall implementation would return -ERR1 or -ERR2 accordingly and the C library would check the return value against zero and if it's negative, it will return -1 to user and also set errno to ERR1/ERR2.
Hope this helps.
Update: I checked the x86_64 code of glibc, and this may help understand this:
.text
ENTRY (syscall)
movq %rdi, %rax /* Syscall number -> rax. */
movq %rsi, %rdi /* shift arg1 - arg5. */
movq %rdx, %rsi
movq %rcx, %rdx
movq %r8, %r10
movq %r9, %r8
movq 8(%rsp),%r9 /* arg6 is on the stack. */
syscall /* Do the system call. */
cmpq $-4095, %rax /* Check %rax for error. */
jae SYSCALL_ERROR_LABEL /* Jump to error handler if error. */
L(pseudo_end):
ret /* Return to caller. */
PSEUDO_END (syscall)
File: "sysdeps/unix/sysv/linux/x86_64/syscall.S"
Linus said he will make sure the no syscall returns a value in -1 .. -4095 as a valid result so we can savely test with -4095.
Update: so I guess it's your c library that converts the return value. You platform ABI may define that syscall returning {-1, -256} on failure so the C wrapper sets the return value to -1 in such cases and set the errno accordingly.