Posted on

In this example I will use Ubuntu 18.04 LTS(AMD64) Desktop with Linux 5.0.0 Kernel and the 5.4.2 Kenel source file as an example to build a linux kernel with a customized system call, and then update the linux kernel of the Desktop. The road to a successful compilation and load can be arduous, so be careful, which should make the process much easier for you.

Video Tutorial

If you prefer a video tutorial, I’ve made one and you can go to YouTube for it.

Get the Kernel Source

You can download the kernel from the website The Linux Kernel Archives.

Or Simply

$ wget

After downloading the kernel, you should get a linux-x.x.tar.xz(in my case, linux-5.4). You should move the file to your home directory and then extract it.

$ mv linux-5.4.tar.xz ~/
$ tar -xvf linux-5.4.tar.xz

Now, a linux-5.4 will be in your home directory, to make the later process easier, we can create a soft link to the directory.

$ ln -s ~/linux-5.4 ~/linux

Install the Compile Tools

$ sudo apt-get install git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache libncurses-dev bison flex gcc make git vim

In the later process, in case there is some tool missing, you can just run the apt-get install command. For example

With the gcc: not found error, you should run the following command.

$ sudo apt-get install gcc

While this time

You should run

$ sudo apt-get install flex

To note, flex and bison are 2 most easily forgotten packages.

Transfer the Current Kernel Configuration

Firstly, clean all the previous configurations of the kernel source code

$ cd linux
$ make mrproper

To make it easier, we can transfer our current kernel configuration to the new kernel.

$ cp /boot/config-`uname -r` .config
$ yes '' | make oldconfig

The above commands should make your life easier. No extra steps are required

Add Customized System Call

Now with all of this configured, let’s add some small feature to our kernel. This syscall is about the kernel ram page management and it will allow our kernel to print the number of page faults in the whole system, the number of page faults and dirty pages in the system log.

Add the System Call Number

I am using an x64 Ubuntu, so I will not be modifying the syscall_32.tbl in the same directory.

$ vim ~/linux/arch/x86/entry/syscalls/syscall_64.tbl

Then find an unused syscall number in the gap of the common/64 and x32 ABI’s. To know more about how to add a syscall number in should refer to [Ref 2].

In my case, I will use 428.

Since this is only a simple program that is not architecture-specific, I will just use a ‘common’ ABI.

Also add it to the file syscall_32.tbl

$ vim ~/linux/arch/x86/entry/syscalls/syscall_32.tbl

Then we can add it to the unistd.h file as a micro.

$ sudo vim ~/linux/include/uapi/asm-generic/unistd.h

In the vim editor, you can use the ?__NR_syscalls to search for the pattern. If there are many matches, you can simply hit Enter and then use ‘n’ to search up and ‘N'(Shift + ‘n’) to search down. When you are in the right position use ‘i’ to insert your syscall above this string and then modify the syscall number. For example, it was originally #define __NR_syscalls 436, I modified it to #define __NR_syscalls 437, and added my own syscall number right above it. When the modification is finished, hit ESC key to quit the insert mode. Then use ‘:x’ command to save the changes and exit.

After the modification

Deal with the Page Faults Info Print

Now it’s time to insert our code to the kernel.

Add Global Page Counter

$ vim ~/linux/include/linux/mm.h

Use ‘:$’ vim command to go to the end of the file and insert the following c declaration right above the #endif directive. This will be our variable to count the global page faults.

extern unsigned long pfcount;

Then EXC, ‘:x’ to quit insert mode, save and exit.

Add Counter for task_struct

The task_struct structure is an abstraction for Linux processes. What we will do next is to add a page fault counter for the process structure.

$ vim ~/linux/include/linux/sched.h

Use ‘?task_struct’, hit enter and then use ‘N'(Shift + ‘n’) to search for the structure definition.

WRONG: Now declare our member variable at the beginning of the structure definition. Then save and exit. (Kidding, DO NOT DO THIS!!!) As you can see from the comment below my the #ifdef statement. thread_info should be declared before everything else in the struct.

CORRECT: So the correct thing to do is to scroll down and you can add it at the same place as me.

Initialize the Task Specific Page Fault Counter

Open the file fork.c

$ vim ~/linux/kernel/fork.c

And search for ‘static struct task_struct *dup_task_struct’

Now after the `tsk = alloc_task_struct_node(node)` statement and its judger that detects if tsk is NULL, insert

tsk->pf = 0;

It should be like the following

Increment the Counters Whenever Page Faults Occur

$ vim ~/linux/arch/x86/mm/fault.c

Search for the __do_page_fault function

Perform the forward declaration for the external variable pfcount, and then increment both pfcount and tsk->pf,

unsigned long pfcount;

Implement the System Call Function

$ vim ~/linux/kernel/sys.c

Append the following Syscall function to the end of the file, don’t miss the return 0.


        printk("Number of dirty pages per process: \n");
        struct task_struct *t;
        // Dirty pages of each process
                printk("%s [%d]: %lu\n", t->comm, t->pid, t->nr_dirtied);
        printk("Number of page faults (total):   %lu\n", pfcount);
        printk("Number of page faults (current): %lu\n", current->pf);
        return 0;

Don’t forget the declaration in ~/linux/include/linux/syscalls.h

$ vim ~/linux/include/linux/syscalls.h 

Add the following line at almost the end of the file.

asmlinkage int sys_mysyscall(void);

And for compatibility,

$ vim kernel/sys_ni.c

Add the following to the file


Compile the Kernel

Now it’s time to compile our kernel, simply

$ nohup make -j12 &

nohup will take the program running in the background and the output is kept as a nohup.out log file. The -j12 option means that I use 12 logical processors since I want to make full use of my 6-core 12-process CPU. To see the nohup.out log file

$ cd ~/linux
$ cat nohup.out

You can have a look at it from time to time to make sure it is working correctly.

The log should be updating once in a while.

Install the Kernel

$ sudo make modules_install -j 12
$ sudo make install -j 12 
$ sudo update-grub


To verify that our kernel module is working, you should firstly reboot the system.

$ sudo reboot

Write a Test Program

#define _GNU_SOURCE
#include <linux/unistd.h>
#include <sys/syscall.h>
#include <stdio.h>
#include <string.h>
#define __NR_mysyscall 428

int main()
        syscall(__NR_mysyscall);        /* or syscall(428) */

        return 0;

Then compile the test program

$ gcc -o test test.c

Then run it

$ ./test

To view the output

$ dmesg


In case your compile process fails, every time you retry, the recompile process can be like the following

$ cd ~/linux
$ make mrproper
$ cp /boot/config-$(uname -r) .config
$ yes '' | make oldconfig
$ nohup make -j12 &
$ sudo make modules_install -j 12
$ sudo make install -j 12 
$ sudo update-grub

Wiki: Linux Architecture

What is the difference between i386, x86, x86_64, amd64

  • x86-64 is the 64bit version instruction set of x86
  • x86 is the 32-bit instruction set for Intel CPUs
  • i386 is a former version of x86 in the early ages of Intel
  • AMD64 and Intel 64 are synonyms for x86-64
  • IA-64 is a little-used instruction set, proposed by Intel, a disruptive 64-bit instruction set, while AMD64 is an extension to x86. History turns out that AMD64 wins the market.

But in Linux, the situation is a little bit different, x86 is denoted by i386, while x86-64 is denoted by x86. Since x86_64 is an extension to x86, so it is a general x86 architecture. In this way, it makes sense to use x86 for x86_64.

Ref. 7


  1. How to Compile a Custom Linux Kernel on Unbuntu 19.04:
  2. How to add a new Syscall:
  6. ABI:
  7. More on ABI:
  8. About x86:
  9. Merge i386 and x86_64 into x86:
  10. x86-64 Wiki:

Furthermore, there is a doc provided in kernel version 3.17 for my OS course. Although not update to date, still inspirint

Leave a Reply

Your email address will not be published. Required fields are marked *