Procedures, Functions and Calls in Windows and Linux

The procedure is a separate fragment of the code that can be called from another part of the program. The function is similar to a procedure, but it also returns a result when it is finished. Procedures and functions are standalone elements. In object-oriented programming, a function which is a member of a class or object is named a method. The name depends on the programming language. In assembler, the common name for a separate fragment of the code called from another part is a procedure. The procedure can be separately tested and used multiple times in the same program or in other projects. Using procedures makes the code easier to manage and reusable, increasing the overall efficiency of software creation.

Procedures can be defined using a pair of directives. The PROC directive is used at the beginning of the procedure, and the ENDP directive is used at the end. The PROC directive can automatically:

• Preserve the contents of the registers whose values should not change, but are needed for use in the procedure.
• Set up local variables on the stack.
• Set up parameters placed on the stack.
• Adjust the stack when the procedure ends.

With the use of additional directives, it is possible to provide information about stack utilisation for stack unwinding.

Procedures can have parameters. In general, parameters can be passed through the stack, registers, common memory or a combination of these. In different operating systems, the rules of passing parameters differ. In 64-bit Windows, the fast call calling convention is used. In this convention, the first four parameters are passed through registers, and each subsequent parameter is passed through the stack. If the parameters are integers, they are passed through general-purpose registers. If parameters are floating-point numbers, they are passed through XMM registers as scalars. If the procedure plays the role of a function, it returns the resulting value. Integers are returned through the accumulator (RAX), and floating-point values are returned through XMM0. Parameters passing in Windows x64 ABI is summarised in a table 1.

If the procedure has integer parameters only, XMM registers will not be modified. If the procedure has floating-point arguments, general-purpose registers remain unchanged. If the types of parameters are mixed, they are passed through the corresponding registers. It is shown in the following example written in C.

function_1(int a, int b, int c, int d, int e);
// a in RCX, b in RDX, c in R8, d in R9, e is pushed on the stack
 
function_2(float a, double b, float c, double d, float e);
// a in XMM0, b in XMM1, c in XMM2, d in XMM3, e is pushed on the stack
 
function_3(int a, double b, int c, double d, int e);
// a in RCX, b in XMM1, c in R8, d in XMM3, e is pushed on the stack

The example of the procedure that adds two arguments and returns the sum can look like the following code.

AddProc PROC
  mov RAX, RCX  ; First parameter is in RCX
  add RAX, RDX  ; Second parameter is in RDX, RAX returns the result
  ret
AddProc ENDP

The Microsoft Windows x64 calling convention requires that even when the parameters are passed through registers, a 32-byte space for them should be reserved on the stack. It is referred to as a shadow space or home space. The shadow space size can be increased to store local variables of the procedure. Why does the x64 calling convention require the shadow space to be explained in the Microsoft blog article^[1].
Another requirement is that the stack must be aligned to the 16-byte boundaries. This is done for the performance, because data transfer between the processor and memory at aligned addresses can be done with faster versions of instructions (MOVAPD instead of MOVUPD). Note that the returning address, which is pushed on the stack automatically when the procedure is called, is 8 bytes long, so even without the shadow space, the stack pointer adjustment is required. Our simple AddProc function doesn't adjust the stack because it doesn't call any function which needs the stack to be aligned. If we decide to call a system function, we must align the stack before making a call. Before modification of the stack pointer, we must preserve its content and restore it before returning.

AlignProc PROC
  push RBP             ; preserve RBP
  mov RBP, RSP         ; store RSP in RBP
  and RSP, NOT 0Fh     ; align stack to the nearest address divisible by 16
  call SystemFunction  ; call any system function
  leave                ; restore RSP and RBP back
  ret
AlignProc ENDP

Certainly, these rules are to be used if there is a need to call a system function or to maintain compatibility with a high-level compiler. If the procedure is written in pure assembly and called from an assembly program, it is the programmer's decision whether they want to follow these rules.
The rules of passing parameters, stack and registers use, and data storage layout in 64-bit Microsoft Windows are described in the document about x64 Application Binary Interface (ABI)^[2].
In the Linux x64 Calling Convention, the first six arguments of type integer/pointers are passed in registers and subsequent arguments through the stack. For the floating point arguments, the first eight are passed in XMM registers, and the subsequent ones through the stack. Parameters passing in Linux is summarised in a table 2.

The operating systems offer a set of functions which help write an application. These functions include reading characters and text from standard input, usually the keyboard, displaying characters or text on standard output, usually the monitor, handling files, data streams and many others. In previous generations of operating systems, the software interrupt mechanism was used. In Microsoft DOS, it was int 21h while in 32-bit versions of Linux it was int 80h (or in the C-style hex notation int 0x80). Calling the system function required preparing the arguments in scratch registers and signalling the software interrupt.

Modern 64-bit operating systems use alternative methods for calling system functions. They significantly differ between Linux and Windows, so we'll briefly summarise both.

The Microsoft Windows operating system implements functions visible to programmers in the API (Application Programming Interface). Functions are identifiable by names, and they can be called as any other function in the program. Windows API functions for 32 and 64-bit Windows are documented on the Microsoft website ^[3]. Let's see the Hello World example written in the Windows API.

; include the library with system functions
includelib kernel32.lib
 
; define function names as external symbols
EXTERN GetStdHandle: PROC
EXTERN WriteConsoleA: PROC
 
; data section with constants and variables definitions
.DATA
 
STD_OUTPUT_HANDLE = -11
stdout_handle     dq 0
hello_msg         db "Hello World", 0
dummy             dq 0
 
; code section
.CODE 
MyAssemblerFunction PROC
 
; the stack must be aligned to an address divisible by 16 - mod(16)
; after the function call is aligned to mod(8)
; the Windows requires the shadow space on the stack
    push  rbp        ; push rpb to the stack
    mov   rbp, rsp   ; store rsp to rbp
    sub   rsp, 48    ; shadow space (32 bytes) and stack alignment (additional 8 bytes)
 
; we need the handle of the console window
    mov   rcx, STD_OUTPUT_HANDLE
    call  GetStdHandle
    mov   stdout_handle, rax
 
; display the text in the console window
    mov   rcx, stdout_handle
    mov   rdx, offset hello_msg
    mov   r8,  sizeof hello_msg
    mov   r9,  dummy
    call  WriteConsoleA
 
; restore the stack pointer and rbp
    mov   rsp, rbp
    pop   rbp
 
; return from the function
    ret
MyAssemblerFunction ENDP
END

The Linux operating system still supports the traditional calling of system functions using software interrupts. It is based on the int 0x80 interrupt, which recognises the number of the function in the EAX register and up to six arguments in EBX, ECX, EDX, ESI, EDI, and EBP. The example of the Hello World program in Linux interrupt-based system call is shown in the following code.

section   .text
global    _start
_start:    
; write function
     mov   ebx, 1    ; first argument - stdio
     mov   ecx, msg  ; second argument - text buffer
     mov   edx, len  ; third argument - text length
     mov   eax, 4    ; function number - write
     int   0x80
 
; exit from program
     mov   eax, 1    ; function number - exit
     int   0x80
 
section    .data
msg  db    "Hello World!", 10
len  equ   $ - msg

Modern processors have new instructions especially designed for calling system functions. They are supported in the Linux operating system. The syscall instruction doesn't use the interrupt mechanism. It uses registers only to provide the address of the function, store the return address and flags register, and load the instruction pointer. This makes the syscall instruction execution significantly faster than int 80h, and is the preferred mechanism for system calls in 64-bit Linux systems. The RIP is stored in RCX, and RFLAGS is stored in R11. The RIP is loaded with the content of the special register IA32_LSTAR MSR, which is an element of Architectural Model-Specific Registers, implemented starting from certain models of 32-bit processors. The Linux system sets this register, and the programmer selects the function using the RAX register, as in the previous model of system calls.

global	_start
section	 .text
 
_start:
 
; write function
     mov   rdi, 1    ; first argument of the function - stdout
     mov   rsi, msg  ; second argument - text buffer
     mov   rdx, len  ; third argument - number of characters
     mov   rax, 1    ; write function
     syscall
 
; exit from program
     mov   rdi, 0    ; result code of the program
     mov   rax, 60   ; exit function
     syscall
 
msg: db    "Hello World!", 10
len  equ   $ - msg