Procedures, Functions and Calls in Windows and Linux

The procedure is a separate fragment of the code that can be called from another part of the program. The function is similar to a procedure, but it also returns a result when it is finished. Procedures and functions are standalone elements. In object-oriented programming, a function which is a member of a class or object is named a method. The name depends on the programming language. In assembler, the common name for a separate fragment of the code called from another part is a procedure. The procedure can be separately tested and used multiple times in the same program or in other projects. Using procedures makes the code easier to manage and reusable, increasing the overall efficiency of software creation.

Procedures can be defined using a pair of directives. The PROC directive is used at the beginning of the procedure, and the ENDP directive is used at the end. The PROC directive can automatically:

• Preserve the contents of the registers whose values should not change, but are needed for use in the procedure.
• Set up local variables on the stack.
• Set up parameters placed on the stack.
• Adjust the stack when the procedure ends.

With the use of additional directives, it is possible to provide information about stack utilisation for stack unwinding.

Procedures can have parameters. In general, parameters can be passed through the stack, registers, common memory or a combination of these. In different operating systems, the rules of passing parameters differ. In 64-bit Windows, the fast call calling convention is used. In this convention, the first four parameters are passed through registers, and each subsequent parameter is passed through the stack. If the parameters are integers, they are passed through general-purpose registers. If parameters are floating-point numbers, they are passed through XMM registers as scalars. If the procedure plays the role of a function, it returns the resulting value. Integers are returned through the accumulator (RAX), and floating-point values are returned through XMM0. Parameters passing in Windows x64 ABI is summarised in a table 1.

If the procedure has integer parameters only, XMM registers will not be modified. If the procedure has floating-point arguments, general-purpose registers remain unchanged. If the types of parameters are mixed, they are passed through the corresponding registers. It is shown in the following example written in C.

function_1(int a, int b, int c, int d, int e);
// a in RCX, b in RDX, c in R8, d in R9, e is pushed on the stack
 
function_2(float a, double b, float c, double d, float e);
// a in XMM0, b in XMM1, c in XMM2, d in XMM3, e is pushed on the stack
 
function_3(int a, double b, int c, double d, int e);
// a in RCX, b in XMM1, c in R8, d in XMM3, e is pushed on the stack

The example of the procedure that adds two arguments and returns the sum can look like the following code.

AddProc PROC
  mov RAX, RCX  ; First parameter is in RCX
  add RAX, RDX  ; Second parameter is in RDX, RAX returns the result
  ret
AddProc ENDP

The Microsoft Windows x64 calling convention requires that even when the parameters are passed through registers, a 32-byte space for them should be reserved on the stack. It is referred to as a shadow space or home space. The shadow space size can be increased to store local variables of the procedure. Why does the x64 calling convention require the shadow space to be explained in the Microsoft blog article^[1].
Another requirement is that the stack must be aligned to the 16-byte boundaries. This is done for the performance, because data transfer between the processor and memory at aligned addresses can be done with faster versions of instructions (MOVAPD instead of MOVUPD). Note that the returning address, which is pushed on the stack automatically when the procedure is called, is 8 bytes long, so even without the shadow space, the stack pointer adjustment is required. Our simple AddProc function doesn't adjust the stack because it doesn't call any function which needs the stack to be aligned. If we decide to call a system function, we must align the stack before making a call. Before modification of the stack pointer, we must preserve its content and restore it before returning.

AlignProc PROC
  push RBP             ; preserve RBP
  mov RBP, RSP         ; store RSP in RBP
  and RSP, NOT 0Fh     ; align stack to the nearest address divisible by 16
  call SystemFunction  ; call any system function
  leave                ; restore RSP and RBP back
  ret
AlignProc ENDP

Certainly, these rules are to be used if there is a need to call a system function or to maintain compatibility with a high-level compiler. If the procedure is written in pure assembly and called from an assembly program, it is the programmer's decision whether they want to follow these rules.
The rules of passing parameters, stack and registers use, and data storage layout in 64-bit Microsoft Windows are described in the document about x64 Application Binary Interface (ABI)^[2].
In the Linux x64 Calling Convention, the first six arguments of type integer/pointers are passed in registers and subsequent arguments through the stack. For the floating point arguments, the first eight are passed in XMM registers, and the subsequent ones through the stack. Parameters passing in Linux is summarised in a table 2.

The operating systems offer a set of functions which help write an application. These functions include reading characters and text from standard input, usually the keyboard, displaying characters or text on standard output, usually the monitor, handling files, data streams and many others. In previous generations of operating systems, the software interrupt mechanism was used. In Microsoft DOS, it was int 21h while in 32-bit versions of Linux it was int 80h. Calling the system function required preparing the arguments in scratch registers and signalling the software interrupt.

Modern 64-bit operating systems use alternative methods for calling system functions. They significantly differ between Linux and Windows, so we'll briefly summarise both.

The Microsoft Windows operating system implements functions visible to programmers in the API (Application Programming Interface). Functions are identifiable by names, and they can be called as any other function in the program. Windows API functions for 32 and 64-bit Windows are documented on the Microsoft website ^[3].