Part II: Linux Boot Optimization: Results

Easily replace the init with a shell script based on /etc/rc.d/rc.sysinit and try to

tune it up as a result.

The results are pretty good I think, here is the general time line made

with a wallclock:



00: exit grub; start booting the kernel

04: kernel prints audit()

11: initrd is mounted; Red Hat nash visible

   mount / ro (normal initrd procedure)

13: start bootchart logging; start readahead of approx 193MB files

   sleep until readahead is complete

24: readahead done; now

   create /dev and modprobe (in background)

   mount / rw, enable swap

   start xfs

   startx as user davidz in background

   start messagebus

   start hald

   start acpid

   start NetworkManager

32: X claims the display

34: GNOME desktop banner

40: GNOME desktop is usable (Nautilus desktop, panel fully populated)



Here is a bootchart made with the bootchart software from Ziga Mahkovec:



http://people.redhat.com/davidz/bootchart.png



Thanks to David as I said before, I'm writing from his works here:



"You may notice that you can also start firefox after login and it starts very

very fast that's because readahead loads all files used by Firefox

in earlier experiments. I've also added files from OpenOffice.org to

readahead and that meant I could start up OpenOffice.org Writer in about

three seconds. More below.



I've made the following observations



1. The kernel patch, linux-2.6.3-printopen.patch, wasn't really working

 well for me - it reported far to few files - instead I added a

 printk() to fs/namei.c:link_path_walk()

 (disclaimer: I don't know much about the kernel so there may be a

 better solution than this).



2. The data captured from link_path_walk() was massaged into a list

 of unique files to preload and sorted on sectors.



3. While capturing the data link_path_walk() and before processing

 I went through all the menus in the GNOME desktop (to make sure

 their icon and desktop files would be added to the list) as well as

 loading Firefox. The list contains 5189 unique files - 231 of these

 from my home directory - 103 of these from gconf in my home

 directory and 302 from gconf in /etc. 2267 were .png files and

 814 of them were .desktop files. 488 files had ".so" in their name.

 There was a total of 193MB of files (which says something about

 the footprint of the GNOME desktop on Fedora :-/)



4. Doing the readahead really helped the time from startx till a

 usable desktop - less than 10 seconds!



5. Doing readahead on the 5189 files took about 45 seconds on my

 system, mostly because the files were scattered around the disk.

 Since I had a spare partition 17GB partition, I did this:

  a. format spare partition as ext3

  b. copy all readahead files to spare partition (193MB)

  c. copy rest of files from main partition to spare partition

     (about 9GB)

 Now the readahead is down to 11 seconds which averages out to

 be 18MB/s. On the other hand, I can still see (using fileblock)

 that the files in the readahead is still scattered out and hdparm

 says I should be able to get 33.87 MB/sec with no seeks.



6. I made a hack to cache /dev (a dev.tar file) and the list of modules

 to load. This could be used in production if the kernel could give

 us basically a hash value for the kobject hierarchy representing

 the hardware (perhaps even a 'tree /sys |md5sum' would suffice).

 This shaved some seconds of as well.



7. A number of things was started in parallel - I found that doing

 readahead while running modprobe wasn't helping anything; in fact

 it contributed negatively to performance (a bit to my surprise, I

 guess because the kernel was busy).



8. readahead on the right files is A Good Thing(tm). Booting my system

 without readahead on the partition with the readahead files scattered

 took 58 seconds (compared to 39 with readahead on the optimized

 partition)



  http://people.redhat.com/davidz/bootchart-without-readahead-scattered.png



 and without readahead on on the optimized partition it took 43

 seconds



  http://people.redhat.com/davidz/bootchart-without-readahead-nonscattered.png



 again compared to 39 seconds. As an added bonus, the readahead

 makes sure that e.g Firefox loads fast; all .png and .desktop files

 are in place for when using the menus. As mentioned, one could put

 very big apps like e.g. OO.o in the readahead set.



So, I think these numbers are good and there's still some room for

improvement; e.g. it takes ten seconds from grub to when the initrd is

mounted - surely the kernel can boot faster? It's after all 25% of the

time spent from grub until I have usable desktop.



The bad thing is that this approach is highly specific to my system (and

thus why I'm not posting an RPM with it :-), however I think it clearly

shows where improvements should be made; here are some random thoughts



a. We should keep track of files being loaded and maintain the

  readahead fileset as appropriate. printk() doesn't seem like the

  right solution; perhaps a system daemon using inotify or the

  kernel events layer is the road ahead? This would enable us to

  readahead the KDE stuff if the user is e.g. using KDE a lot.



b. ext3 should support operations for moving blocks around; e.g.

  optimize around the readahead fileset - when idle the system should

  rearrange the files to facilitate faster booting



c. the start_udev and kmodule process could be cached as I did above



d. The whole init(1) procedure seems dated; perhaps something more

  modern built on top of D-BUS is the right choice - SystemServices

  by Seth Nickell comes to mind [1]. Ideally services to be started

  would have dependencies such as 1) don't start the gdm service

  before /usr/bin/gdm is available; 2) the SSH service would only

  be active when NetworkManager says there is a network connection;

  /usr from LABEL=/usr would only be mounted when there is a volume

  with that label and so forth. Also, such a system would of course

  have support for LSB init scripts.

  (This is probably a whole project on it's own so I'm omitting

  detailed thinking on it for now)

"

Thanks a lot to Ziga Mahkovec for the bootchart software - it's been

very useful.