====== Principles of Instructions Encoding ======
In x86 processors, instructions are encoded using varying numbers of bytes. Some instructions are encoded in just a single byte, others require several bytes, and some are as long as 15 bytes. In general, instructions are composed with some fields as presented in figure {{ref>instr_fields}}.
{{ :en:multiasm:cs:instruction_fields.png?600 |Illustration of fields of the instrution}}
Fields of the instructions
The fields are as follows
* Instruction prefixes
* Opcode
* MODR/M byte
* SIB byte (Scale Index Base byte)
* Displacement
* Immediate data
The only field which is present in every instruction is the opcode. Other fields appear depending on the detailed function of the instruction, its arguments and modifications. Their appearance also depends on the target version of the processor (16, 32 or 64-bit). In the following sections, we will show some principles of each field encoding.
=====Registers encoding in the instructions=====
Registers are used in the instructions as the operands which hold the data for calculations, or as elements of the address expression to calculate the indirect address in memory. Wherever the register appears, it is encoded with the same bit pattern. Initially, in 16 and 32-bit machines, three bits were used. In 64-bit processors, an additional, leading bit is used, which doubles the number of available registers. In table {{ref>register_16}}, the encoding of general-purpose registers in 16 and 32-bit processors is presented.
Encoding of registers in 16- and 32-bit x86 processors
In table {{ref>register_64_0}}, the encoding of general-purpose registers in x64 processors is shown. The additional leading bit = 0 comes from the REX prefix. For instructions without the REX prefix, the encoding of 8-bit registers is the same as in x86.
Encoding of registers in x64 processors with a leading bit = 0
In table {{ref>register_64_1}}, the encoding of general-purpose registers in x64 processors is shown. The additional bit = 1 comes from the REX prefix.
Encoding of registers in x64 processors with a leading bit = 1
In table {{ref>register_ext_0}} encoding of extension registers is presented. Please note that FPU and MMX registers, which are available on 32-bit processors, encoding is the same as with the additional leading bit = 0.
Encoding of registers in x64 processors with a leading bit = 0
In table {{ref>register_ext_1}} encoding of extension registers is presented. As there are only 8 MMX registers, for these registers, there is no difference between a leading bit equal to 0 or 1. FPU registers are not available with the leading bit = 1.
Encoding of additional registers in x64 processors with a leading bit = 1
=====Instruction prefixes=====
There are four groups of prefixes:
* Lock and repeat
* Segment override and branch hints
* Operand size override
* Address size override
In modern 64-bit processors, REX and VEX prefixes were introduced to extend the number of registers and to enable the RISC-like three-argument instructions.
The **lock prefix** is valid for instructions that work in a read-modify-write manner. An example of such an instruction can be adding a constant or register content to the variable in the memory.
lock add QWORD PTR [rax], 5
The lock prefix appears as a single byte with the value 0x0F before the opcode. It disables DMA requests (or any other requests that gain control of the buses) during the execution of the instruction to prevent accidental modification of the memory contents at the same address by both the processor and DMA controller.
The **repeat prefixes** are valid for string instructions. They modify the behaviour of instruction from single to repeated execution. The REP prefix is encoded as a single byte with the value 0xF3. It causes the instruction to repeat up to CX times, decreasing CX with every repetition and appropriately modifying the index registers. The REPE/REPZ (0xF3) and REPNE/REPNZ (0xF2) prefixes are used with string instructions which compare or scan strings. They cause the instruction to repeat with the possibility of finishing the execution if the zero flag is set (REPE) or cleared (REPNE).
The **segment override** prefix is used in the segmented mode of operation. It causes the instruction to access data placed in a segment other than the default. For example, the mov instruction accesses data in the segment pointed by DS, which can be changed to ES with a prefix.
mov BYTE PTR [ebx], 5 ;DS as the default segment
mov BYTE PTR ES:[ebx], 5 ;ES segment override (results in appearance of the byte 0x26 as the prefix)
* 0x2E – CS segment override
* 0x36 – SS segment override
* 0x3E – DS segment override
* 0x26 – ES segment override
* 0x64 – FS segment override
* 0x65 – GS segment override
In 64-bit mode, the CS, SS, DS and ES segment overrides are ignored.
The **branch hint** prefixes can appear together with conditional jump instructions. These prefixes can be used to support the branch prediction unit of the processor in determining if the branch shall be taken (0x3E) or not taken (0x2E). This function is enabled if there is no history in the branch prediction unit yet, and static branch prediction is used.
The **operand size** and **address size override** prefixes can change the default size of operands and addresses. For example, if the processor operates in 32-bit mode, using the 0x66 prefix changes the size of an operand to 16 bits, and using the 0x67 prefix changes the address encoding from 32 bits to 16 bits. To better understand the behaviour of prefixes, let us consider a simple instruction with different variants. Let's start with a 32-bit processor.
mov BYTE PTR [ebx], 0x5 ;encoded as 0xC6, 0x03, 0x05
mov WORD PTR [ebx], 0x5 ;encoded as 0x66, 0xC7, 0x03, 0x05, 0x00
mov DWORD PTR [ebx], 0x5 ;encoded as 0xC7, 0x03, 0x05, 0x00, 0x00, 0x00
We can notice that because the default operand size is a 32-bit doubleword, so prefix 0x66 appears in the 16-bit version (WORD PTR). It is also visible that the 8-bit version (BYTE PTR) has a different opcode (0xC6, 0x03 instead of 0xC7, 0x03). Also, the size of the argument is different.
The address override prefix (0x67) appears if we change the register to a 16-bit bx.
mov BYTE PTR [bx], 0x5 ;encoded as 0x67, 0xC6, 0x07, 0x05
mov WORD PTR [bx], 0x5 ;encoded as 0x67, 0x66, 0xC7, 0x07, 0x05, 0x00
mov DWORD PTR [bx], 0x5 ;encoded as 0x67, 0xC7, 0x07, 0x05, 0x00, 0x00, 0x00
The same situation can be observed if we use a 32-bit address register (ebx) and assemble the same instructions for a 64-bit processor.
mov BYTE PTR [ebx], 0x5 ;encoded as 0x67, 0xC6, 0x03, 0x05
mov WORD PTR [ebx], 0x5 ;encoded as 0x67, 0x66, 0xC7, 0x03, 0x05, 0x00
mov DWORD PTR [ebx], 0x5 ;encoded as 0x67, 0xC7, 0x03, 0x05, 0x00, 0x00, 0x00
While we use a native 64-bit address register in a 64-bit processor, the address size override prefix disappears.
mov BYTE PTR [rbx], 0x5 ;encoded as 0xC6, 0x03, 0x05
mov WORD PTR [rbx], 0x5 ;encoded as 0x66, 0xC7, 0x03, 0x05, 0x00
mov DWORD PTR [rbx], 0x5 ;encoded as 0xC7, 0x03, 0x05, 0x00, 0x00, 0x00
**REX** prefix (0x41, 0x48, 0x49, etc.) appears in long mode, e.g. if we try to enlarge the argument size to 64 bits or use any of the new registers. Long mode enables the possibility of using new registers and 64-bit calculations with the original set of instructions. The REX prefix contains four fixed bits and four control bits as shown in figure {{ref>rex_fields}}.
{{ :en:multiasm:cs:rex_prefix.png?400 |Illustration of REX prefix}}
REX prefix
* bit W: operand size override, it is "1" for 64-bit operand, "0" for default operand size
* bit R: extension to ModRM.reg
* bit X: extension to SIB.index
* bir B: extension to ModRM.rm or the SIB.base
Bits R, X and B enable the use of new registers.
mov BYTE PTR [r8], 0x5 ;encoded as 0x41, 0xC6, 0x00, 0x05
mov BYTE PTR [r9], 0x5 ;encoded as 0x41, 0xC6, 0x01, 0x05
mov BYTE PTR [r10], 0x5 ;encoded as 0x41, 0xC6, 0x02, 0x05
mov DWORD PTR [r8], 0x5 ;encoded as 0x41, 0xC7, 0x00, 0x05, 0x00, 0x00, 0x00
mov QWORD PTR [rbx], 0x5 ;encoded as 0x48, 0xC7, 0x03, 0x05, 0x00, 0x00, 0x00
mov QWORD PTR [r8], 0x5 ;encoded as 0x49, 0xC7, 0x00, 0x05, 0x00, 0x00, 0x00
The REX prefix
From the code shown in the last example, you can observe where the register used in the instructions is encoded.
We encourage you to assemble or disassemble some further instructions. You can do it with an Online x86 / x64 Assembler and Disassembler ((https://defuse.ca/online-x86-assembler.htm#disassembly)) by Taylor Hornby. Experimentation can give you valuable information on where some information is encoded in the instruction.
=====Instruction opcode=====
The instruction opcode is the mandatory field in every instruction. It encodes the main function of the operation. Expanding the processor's capabilities by adding new instructions required defining longer opcodes. The opcode can be 1, 2 or 3 bytes in length. New instructions usually contain an additional byte or two bytes at the beginning called an escape sequence. Possible opcode sequences are:
opcode
0x0F opcode
0x0F 0x38 opcode
0x0F 0x3A opcode
In the new instructions, the VEX or XOP prefix is used as the escape sequence.
* 0xC4 Three-byte VEX
* 0xC5 Two-byte VEX
* 0x8F Three-byte XOP
VEX-encoded instructions are written with V at the beginning. Let's look at the example of the blending instruction.
blendvpd xmm0, xmm1 ; encoded as 0x66, 0x0F, 0x38, 0x15, 0xC1
vblendvpd xmm0, xmm1, xmm2, xmm3 ; encoded as 0xC4, 0xE3, 0x71, 0x4B, 0xC2, 0x30
The first blendvpd instruction has only two arguments; in this encoding scheme is not possible to encode more. It uses the mandatory prefix 0x66 and 0x0F, 0x38 escape sequence. The second version, vblendvpd, has four arguments. It is encoded with a three-byte VEX escape sequence 0xC4, 0xE3, 0x71.
=====MOD R/M byte=====
The ModR/M byte encodes the addressing mode, a register which is used as an operand in the instruction, and registers used for addressing of memory. It can also extend the opcode in some instructions. It has the fields as shown in the figure {{ref>modrm_byte}}.
{{ :en:multiasm:cs:modrm_byte.png?300 |Illustration of MOD R/M byte}}
MOD R/M byte
In general, the fields have meaning:
* Mod - Mode. This 2-bit field gives the register/memory mode.
* Reg - Register. This 3-bit field specifies one of the general-purpose registers used as the operand. It can also be the opcode extension.
* R/M - Register/memory. This 3-bit field specifies the second register as the operand or combination of registers used in the calculation of the address in memory.
In x86, the Mod field specifies one of four possible memory addressing modes, and the R/M field specifies which register, or pair of registers, is used for address calculation. If the Mod field is 11 (binary), the R/M field specifies the second register in the instruction. Details are shown in table {{ref>modrm_16}}
Encoding of addressing mode with MOD R/M byte in 16-bit x86 processors
Let's look at some examples of instruction encoding. First, look at the data transfer between two registers.
; MOD REG R/M MOD REG R/M
mov al, dl ;encoded as 0x88, 0xD0 11 010 000 Register operand DL AL
mov ax, dx ;encoded as 0x89, 0xD0 11 010 000 Register operand DX AX
mov dx, si ;encoded as 0x89, 0xF2 11 110 010 Register operand SI DX
mov si, dx ;encoded as 0x89, 0xD6 11 010 110 Register operand DX SI
Notice that in the first and second lines, different opcodes are used, but the MOD R/M bytes are identical. The type of instruction determines the order of data transfer.
Now, a few examples of indirect addressing without displacement.
; MOD REG R/M MOD REG R/M
mov dx,[si] ;encoded as 0x8B, 0x14 00 010 100 Reg. only addr. DX [SI]
mov dx,[di] ;encoded as 0x8B, 0x15 00 010 101 Reg. only addr. DX [DI]
mov dx,[bx+di];encoded as 0x8B, 0x11 00 010 001 Reg. only addr. DX [BX+DI]
mov cx,[bx+di];encoded as 0x8B, 0x09 00 001 001 Reg. only addr. CX [BX+DI]
Now, a few examples of indirect addressing with displacement.
; MOD REG R/M MOD REG R/M Disp
mov dx,[bp+62];encoded as 0x8B, 0x56, 0x3E 01 010 110 Reg.+disp addr. DX [BP+disp] 0x3E
mov [bp+62],dx;encoded as 0x89, 0x56, 0x3E 01 010 110 Reg.+disp addr. DX [BP+disp] 0x3E
mov dx,[si+13];encoded as 0x8B, 0x54, 0x0D 01 010 100 Reg.+disp addr. DX [SI+disp] 0x0D
mov si,[bp] ;encoded as 0x8B, 0x76, 0x00 01 110 110 Reg.+disp addr. SI [BP+disp] 0x00
If we look in fort two lines, we can observe that the MOD R/M bytes are identical. The only difference is the opcode, which determines the direction of the data transfer.
Notice also that the last instruction is encoded as BP + displacement, even if there is no displacement in the mnemonic. If you look into the table {{ref>modrm_16}}, you can observe that there is no addressing mode with [BP] only. It must appear with the displacement.
In 32-bit mode, registers used for addressing can be specified with the SIB byte, but this is not always the case. If a single register is used (with some exceptions), it still can be encoded with the MOD R/M byte, and there is no SIB byte in the instruction.
In 64-bit long mode, the MOD R/M byte encoding works in a similar manner to 32-bit, with the difference that the MOD R/M byte is extended with R and B bits from the REX prefix, enabling the use of more registers in instructions.
Encoding of addressing mode with MOD R/M byte in x64 processors for leading bit =0
=====Scale Index Base byte=====
The SIB byte was added in 32-bit machines to enable the use of any register in addressing. It encodes the scaling factor, index and base register used in address calculations. As a result, the choice of registers for calculations is significantly wider. For details, please refer to the addressing modes section. The SIB byte has the fields as shown in the figure {{ref>sib_byte}}.
{{ :en:multiasm:cs:sib_byte.png?300 |Illustration of Scale Index Base byte}}
Scale Index Base byte
The scaling factor specifies the number 1, 2, 4 or 8 by which the content of the index register is multiplied in the process of the address calculation as presented in table {{ref>SIB_scale}}. For details on addressing, please refer to the section Addressing Modes in Instructions.
The index and base fields specify the index and base registers, respectively. In 32-bit processors, the register encoding is similar to that presented in table {{ref>register_16}}. Details for 32-bit addressing are presented in tables {{ref>SIB_index}} and {{ref>SIB_base}}.
Encoding of registers in the index field of the SIB byte in 32-bit processors
In 64-bit long mode, the SIB byte is extended with X and B bits from the REX prefix, enabling the use of more registers in instructions. Details are presented in tables {{ref>SIB_index64}} and {{ref>SIB_base64}}.
Encoding of registers in the index field of the SIB byte extended with X bit in 64-bit processors
In the tables {{ref>SIB_index}} and {{ref>SIB_index64}}, the * means that there is no index register encoded in the instruction. It results in base-only or direct addressing.
Let's look at some code examples, considering the 32-bit version first. In all instructions, there is a MOD R/M byte and an SIB byte. MOD R/M is identical for all instructions. REG field indicates eax register and the combination of MOD and R/M indicates that registers are specified with the SIB byte.
;MOD R/M (second byte) is 0x04 for all instructions:
; MOD REG R/M REG MOD & R/M
; 00 000 100 eax SIB is present
;SIB (third byte) is 0x0B, 0x4B, 0x8B or 0xCB:
; Scale Index Base Scale Index Base
mov eax, [ebx+ecx] ;0x8B, 0x04, 0x0B 00 001 011 x1 ecx ebx
mov eax, [ebx+ecx*2] ;0x8B, 0x04, 0x4B 01 001 011 x2 ecx ebx
mov eax, [ebx+ecx*4] ;0x8B, 0x04, 0x8B 10 001 011 x4 ecx ebx
mov eax, [ebx+ecx*8] ;0x8B, 0x04, 0xCB 11 001 011 x8 ecx ebx
And other examples for x64 processors. The SIB byte is extended with bits from the REX prefix. We'll start with the similar examples as shown for 32-bit machines.
;REX prefix (first byte) is 0x48 for all instructions:
; 7 0
; +---+---+---+---+---+---+---+---+
; | 0 1 0 0 | W | R | X | B |
; +---+---+---+---+---+---+---+---+
; 1 0 0 0
;MOD R/M (second byte) is 0x04 for all instructions:
; MOD R.REG R/M REG MOD & R/M
; 00 0.000 100 eax SIB is present
; Scale X.Index B.Base Scale Index Base
mov rax, [rbx+rcx] ;0x48, 0x8B, 0x04, 0x0B 00 0.001 0.011 x1 rcx rbx
mov rax, [rbx+rcx*2] ;0x48, 0x8B, 0x04, 0x4B 01 0.001 0.011 x2 rcx rbx
mov rax, [rbx+rcx*4] ;0x48, 0x8B, 0x04, 0x8B 10 0.001 0.011 x4 rcx rbx
mov rax, [rbx+rcx*8] ;0x48, 0x8B, 0x04, 0xCB 11 0.001 0.011 x8 rcx rbx
If any of the new registers (R8-R15) is used in the instruction, it changes the bits in the REX prefix.
; Scale X.Index B.Base Scale Index Base
mov rax, [r10+rcx] ;0x49, 0x8B, 0x04, 0x0A 00 0.001 1.010 x1 rcx r10
mov rax, [rbx+r11] ;0x4A, 0x8B, 0x04, 0x1B 00 1.001 0.011 x1 r11 rbx
mov r12, [rbx+rcx] ;0x4C, 0x8B, 0x24, 0x0B 10 0.001 0.011 x1 rcx rbx
;Last instruction has the MOD R/M REG field extended
;by the R bit from the REX prefix.
; MOD R.REG R/M REG MOD & R/M
; 00 1.100 100 r12 SIB is present
Certainly, the presented examples do not exhaust all possible situations. For a more detailed explanation, please refer to the documentation by AMD((https://docs.amd.com/v/u/en-US/40332-PUB_4.08)), Intel((https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html)), OSDev wiki((https://wiki.osdev.org/X86-64)) or other interesting sources mentioned at the bottom of this section.
=====Displacement=====
Displacement gives the offset for memory operands. Depending on the addressing mode, it can be the direct memory address or an additional offset added to the contents of the base, index register or both. Displacement can be 1, 2, or 4 bytes long. Some instructions allow using an 8-byte displacement. In these instructions, there is no immediate field.
=====Immediate=====
Some instructions require an immediate value. The instruction determines the length of the immediate value. The immediate can be 1, 2, 4 or 8 bytes long. When an 8-byte immediate value is encoded, no displacement can be encoded.
For detailed information on instructions encoding, please refer to the documentation provided by Intel and AMD. You can also find interesting information on websites: X86 Opcode and Instruction Reference ((http://ref.x86asm.net/index.html)) by
MazeGen, x86 and amd64 instruction reference ((https://www.felixcloutier.com/x86/)) by Félix Cloutier.