Procedures, Functions and Calls in Windows and Linux

The procedure is a separate fragment of the code that can be called from another part of the program. The function is similar to a procedure, but it also returns a result when it is finished. Procedures and functions are standalone elements. In object-oriented programming, a function which is a member of a class or object is named a method. The name depends on the programming language. In assembler, the common name for a separate fragment of the code called from another part is a procedure. The procedure can be separately tested and used multiple times in the same program or in other projects. Using procedures makes the code easier to manage and reusable, increasing the overall efficiency of software creation.

Procedures can be defined using a pair of directives. The PROC directive is used at the beginning of the procedure, and the ENDP directive is used at the end. The PROC directive can automatically:

• Preserve the contents of the registers whose values should not change, but are needed for use in the procedure.
• Set up local variables on the stack.
• Set up parameters placed on the stack.
• Adjust the stack when the procedure ends.

With the use of additional directives, it is possible to provide information about stack utilisation for stack unwinding.

Procedures can have parameters. In general, parameters can be passed through the stack, registers, common memory or a combination of these. In different operating systems, the rules of passing parameters differ. In 64-bit Windows, the fast call calling convention is used. In this convention, the first four parameters are passed through registers, and each subsequent parameter is passed through the stack. If the parameters are integers, they are passed through general-purpose registers. If parameters are floating-point numbers, they are passed through XMM registers as scalars. If the procedure plays the role of a function, it returns the resulting value. Integers are returned through the accumulator (AL, AX, EAX or RAX), and floating-point values are returned through XMM0. Parameter passing and returning the result are summarised in a table 1.

If the procedure has integer parameters only, XMM registers will not be modified. If the procedure has floating-point arguments, general-purpose registers remain unchanged. If the types of parameters are mixed, they are passed through the corresponding registers. It is shown in the following example written in C.

function_1(int a, int b, int c, int d, int e);
// a in RCX, b in RDX, c in R8, d in R9, e is pushed on the stack
 
function_2(float a, double b, float c, double d, float e);
// a in XMM0, b in XMM1, c in XMM2, d in XMM3, e is pushed on the stack
 
function_3(int a, double b, int c, double d, int e);
// a in RCX, b in XMM1, c in R8, d in XMM3, e is pushed on the stack

The example of the procedure that adds two arguments and returns the sum can look like the following code.

AddProc PROC
  mov RAX, RCX  ; First parameter is in RCX
  add RAX, RDX  ; Second parameter is in RDX, RAX returns the result
  ret
AddProc ENDP

The Microsoft Windows x64 calling convention requires that even when the parameters are passed through registers, a 32-byte space for them should be reserved on the stack. It is referred to as a shadow space or home space. The shadow space size can be increased to store local variables of the procedure. Why does the x64 calling convention require the shadow space to be explained in the Microsoft blog article^[1].
Another requirement is that the stack must be aligned to the 16-byte boundaries. This is done for the performance, because data transfer between the processor and memory at aligned addresses can be done with faster versions of instructions (MOVAPD instead of MOVUPD). Note that the returning address, which is pushed on the stack automatically when the procedure is called, is 8 bytes long, so even without the shadow space, the stack pointer adjustment is required. Our simple AddProc function doesn't adjust the stack because it doesn't call any function which needs the stack to be aligned. If we decide to call a system function such as MessageBox, we must align the stack before making a call. If we modify the stack pointer, we must preserve its content and restore it before returning.

AlignProc PROC
  push RBP             ; preserve RBP
  mov RBP, RSP         ; store RSP in RBP
  and RSP, NOT 0Fh     ; align stack to the nearest address divisible by 16
  call SystemFunction  ; call any system function
  leave                ; restore RSP and RBP back
  ret
AlignProc ENDP

<note> It is OK to use the AND logic function to align the stack because the stack grows towards lower addresses. Clearing four least significant bits ensures that the address is 16-byte aligned and is lower than the previous one. <note> Certainly, these rules are to be used if there is a need to call a system function or to maintain compatibility with a high-level compiler. If the procedure is written in pure assembly and called from an assembly program, it is the programmer's decision whether they want to follow these rules.
The rules of passing parameters, stack and registers use, and data storage layout in 64-bit Microsoft Windows are described in the document about x64 Application Binary Interface (ABI)^[2].