Build Your Own OS #09

Niroshan Pushparaj
6 min readSep 27, 2021

Hello everyone! Hope you are following this article series. Welcome back to the 9th part of our article. Today we are going to discuss about how to run a simple program in our operating system in user mode. Let’s get start!

User Mode

I think you already have some knowledge about user mode. We explained that in the 6th part of the series. Please read this again.

Segmentation for user mode

The very first thing that we need to do is add two more segments to the GDT. They are very similar to what we have done in segmentation stage when we were setting up the GDT for the first time.

You need to add these two segments to enable the user mode

I hope you can see where is this change happens. Now the DPL is PL3. But before that all the DPL were set to PL0(kernel mode). But I like to remind you that user program cannot access to kernel space in it’s address space! We need to prevent the user program interfering the kernel space for the security of our operating system which is mandatory. We can user paging for achieve this.

Setting Up For User Mode

There are few things that user mode process required.

  1. Page frames from the RAM for store instructions, data and stack. In here since we are focusing on basic implementation it is fine one page frame for the stack and enough page frames for the instructions or program codes.
  2. The binary from the GRUB module need to copied to the above allocated page frames for the program codes.
  3. Above mentioned page frames need to be mapped using page dictionary and page tables. For that we need at least two page tables since code and data must mapped in at 0x00000000 and increasing from there, and the stack should start from below the kernel, at 0xBFFFFFFB, growing towards lower addresses. (The U/S flag needs set to allow PL3 access.)

For our convenient development it is good to store this process information in a struct. you can allocate it dynamically using kernel’s malloc function.

Entering User Mode

The only way to execute code with a lower privilege level than the current privilege level (CPL) is to execute an iret or lret instruction - interrupt return or long return, respectively.

To enter user mode we set up the stack as if the processor had raised an inter-privilege level interrupt. The stack should look like the following:

[esp + 16]  ss     ;the stack segment selector we want for user mode
[esp + 12] esp ;the user mode stack pointer
[esp + 8] eflags ;the control flags we want to use in user mode
[esp + 4] cs ;the code segment selector
[esp + 0] eip ;the instruction pointer of user mode code to execute

See the Intel manual [33], section 6.2.1, figure 6–4 for more information.

The instruction iret will then read these values from the stack and fill in the corresponding registers. Before we execute iret we need to change to the page directory we setup for the user mode process. It is important to remember that to continue executing kernel code after we’ve switched PDT, the kernel needs to be mapped in. One way to accomplish this is to have a separate PDT for the kernel, which maps all data at 0xC0000000 and above, and merge it with the user PDT (which only maps below 0xC0000000) when performing the switch. Remember that physical address of the PDT has to be used when setting the register cr3.

The register eflags contains a set of different flags, specified in section 2.3 of the Intel manual [33]. Most important for us is the interrupt enable (IF) flag. The assembly code instruction sti can’t be used in privilege level 3 for enabling interrupts. If interrupts are disabled when entering user mode, then interrupts can’t enabled once user mode is entered. Setting the IF flag in the eflags entry on the stack will enable interrupts in user mode, since the assembly code instruction iret will set the register eflags to the corresponding value on the stack.

For now, we should have interrupts disabled, as it requires a little more work to get inter-privilege level interrupts to work properly (see the section “System calls”).

The value eip on the stack should point to the entry point for the user code - 0x00000000 in our case. The value esp on the stack should be where the stack starts - 0xBFFFFFFB (0xC0000000 - 4).

The values cs and ss on the stack should be the segment selectors for the user code and user data segments, respectively. As we saw in the segmentation chapter, the lowest two bits of a segment selector is the RPL - the Requested Privilege Level. When using iret to enter PL3, the RPL of cs and ss should be 0x3. The following code shows an example:


The register ds, and the other data segment registers, should be set to the same segment selector as ss. They can be set the ordinary way, with the mov assembly code instruction.

We are now ready to execute iret. If everything has been set up right, we should now have a kernel that can enter user mode.

Using C for User Mode Programs

When C is used as the programming language for user mode programs, it is important to think about the structure of the file that will be the result of the compilation.

The reason we can use ELF [18] as the file format for for the kernel executable is because GRUB knows how to parse and interpret the ELF file format. If we implemented an ELF parser, we could compile the user mode programs into ELF binaries as well. We leave this as an exercise for the reader.

One thing we can do to make it easier to develop user mode programs is to allow the programs to be written in C, but compile them to flat binaries instead of ELF binaries. In C the layout of the generated code is more unpredictable and the entry point, main, might not be at offset 0 in the binary. One common way to work around this is to add a few assembly code lines placed at offset 0 which calls main:

extern main    section .text
; push argv
; push argc
call main
; main has returned, eax is return value
jmp $ ; loop forever

If this code is saved in a file called start.s, then the following code show an example of a linker script that places these instructions first in executable (remember that start.s gets compiled to start.o):

OUTPUT_FORMAT("binary")    /* output flat binary */

. = 0; /* relocate to address 0 */

.text ALIGN(4):
start.o(.text) /* include the .text section of start.o */
*(.text) /* include all other .text sections */

.data ALIGN(4):

.rodata ALIGN(4):

Note: *(.text) will not include the .text section of start.o again.

With this script we can write programs in C or assembler (or any other language that compiles to object files linkable with ld), and it is easy to load and map for the kernel (.rodata will be mapped in as writeable, though).

When we compile user programs we want the following GCC flags:

-m32 -nostdlib -nostdinc -fno-builtin -fno-stack-protector -nostartfiles -nodefaultlibs

For linking, the followings flags should be used:

-T link.ld -melf_i386  # emulate 32 bits ELF, the binary output is specified              # in the linker script

The option -T instructs the linker to use the linker script link.ld.

Now congratulations! your operating system can switch between user mode and kernel mode! Hope you find this article valuable for you.

Thank you so much for reading this article!