Thursday, December 02, 2004

Part II: Linux Boot Optimization: Results

Easily replace the init with a shell script based on /etc/rc.d/rc.sysinit and try to

tune it up as a result.
The results are pretty good I think, here is the general time line made
with a wallclock:

00: exit grub; start booting the kernel
04: kernel prints audit()
11: initrd is mounted; Red Hat nash visible
mount / ro (normal initrd procedure)
13: start bootchart logging; start readahead of approx 193MB files
sleep until readahead is complete
24: readahead done; now
create /dev and modprobe (in background)
mount / rw, enable swap
start xfs
startx as user davidz in background
start messagebus
start hald
start acpid
start NetworkManager
32: X claims the display
34: GNOME desktop banner
40: GNOME desktop is usable (Nautilus desktop, panel fully populated)

Here is a bootchart made with the bootchart software from Ziga Mahkovec:

http://people.redhat.com/davidz/bootchart.png

Thanks to David as I said before, I'm writing from his works here:

"You may notice that you can also start firefox after login and it starts very
very fast that's because readahead loads all files used by Firefox
in earlier experiments. I've also added files from OpenOffice.org to
readahead and that meant I could start up OpenOffice.org Writer in about
three seconds. More below.

I've made the following observations

1. The kernel patch, linux-2.6.3-printopen.patch, wasn't really working
well for me - it reported far to few files - instead I added a
printk() to fs/namei.c:link_path_walk()
(disclaimer: I don't know much about the kernel so there may be a
better solution than this).

2. The data captured from link_path_walk() was massaged into a list
of unique files to preload and sorted on sectors.

3. While capturing the data link_path_walk() and before processing
I went through all the menus in the GNOME desktop (to make sure
their icon and desktop files would be added to the list) as well as
loading Firefox. The list contains 5189 unique files - 231 of these
from my home directory - 103 of these from gconf in my home
directory and 302 from gconf in /etc. 2267 were .png files and
814 of them were .desktop files. 488 files had ".so" in their name.
There was a total of 193MB of files (which says something about
the footprint of the GNOME desktop on Fedora :-/)

4. Doing the readahead really helped the time from startx till a
usable desktop - less than 10 seconds!

5. Doing readahead on the 5189 files took about 45 seconds on my
system, mostly because the files were scattered around the disk.
Since I had a spare partition 17GB partition, I did this:
a. format spare partition as ext3
b. copy all readahead files to spare partition (193MB)
c. copy rest of files from main partition to spare partition
(about 9GB)
Now the readahead is down to 11 seconds which averages out to
be 18MB/s. On the other hand, I can still see (using fileblock)
that the files in the readahead is still scattered out and hdparm
says I should be able to get 33.87 MB/sec with no seeks.

6. I made a hack to cache /dev (a dev.tar file) and the list of modules
to load. This could be used in production if the kernel could give
us basically a hash value for the kobject hierarchy representing
the hardware (perhaps even a 'tree /sys |md5sum' would suffice).
This shaved some seconds of as well.

7. A number of things was started in parallel - I found that doing
readahead while running modprobe wasn't helping anything; in fact
it contributed negatively to performance (a bit to my surprise, I
guess because the kernel was busy).

8. readahead on the right files is A Good Thing(tm). Booting my system
without readahead on the partition with the readahead files scattered
took 58 seconds (compared to 39 with readahead on the optimized
partition)

http://people.redhat.com/davidz/bootchart-without-readahead-scattered.png

and without readahead on on the optimized partition it took 43
seconds

http://people.redhat.com/davidz/bootchart-without-readahead-nonscattered.png

again compared to 39 seconds. As an added bonus, the readahead
makes sure that e.g Firefox loads fast; all .png and .desktop files
are in place for when using the menus. As mentioned, one could put
very big apps like e.g. OO.o in the readahead set.

So, I think these numbers are good and there's still some room for
improvement; e.g. it takes ten seconds from grub to when the initrd is
mounted - surely the kernel can boot faster? It's after all 25% of the
time spent from grub until I have usable desktop.

The bad thing is that this approach is highly specific to my system (and
thus why I'm not posting an RPM with it :-), however I think it clearly
shows where improvements should be made; here are some random thoughts

a. We should keep track of files being loaded and maintain the
readahead fileset as appropriate. printk() doesn't seem like the
right solution; perhaps a system daemon using inotify or the
kernel events layer is the road ahead? This would enable us to
readahead the KDE stuff if the user is e.g. using KDE a lot.

b. ext3 should support operations for moving blocks around; e.g.
optimize around the readahead fileset - when idle the system should
rearrange the files to facilitate faster booting

c. the start_udev and kmodule process could be cached as I did above

d. The whole init(1) procedure seems dated; perhaps something more
modern built on top of D-BUS is the right choice - SystemServices
by Seth Nickell comes to mind [1]. Ideally services to be started
would have dependencies such as 1) don't start the gdm service
before /usr/bin/gdm is available; 2) the SSH service would only
be active when NetworkManager says there is a network connection;
/usr from LABEL=/usr would only be mounted when there is a volume
with that label and so forth. Also, such a system would of course
have support for LSB init scripts.
(This is probably a whole project on it's own so I'm omitting
detailed thinking on it for now)
"
Thanks a lot to Ziga Mahkovec for the bootchart software - it's been
very useful.

No comments:

There was an error in this gadget