low latency

jerry jdinardo at ix.netcom.com
Tue Oct 16 09:54:14 PDT 2001

Here is an interview with a maintainer of the montavista patch.
It answers a lot of the questions asked in this thread.


   Interview: Robert Love
   Posted on Saturday, October 13 @ 09:48:47 EDT

   Kerneltrap has launched a new series. Once a week we intend to release
   an interview with a member of the kernel hacking community. Though the
   people we interview tend to be quite technical, we hope to keep the
   interviews broad in scope and accessible to all.

   This week's interview is with Robert Love who currently maintains the
   preemptible kernel patches, among other things. He's been using Linux
   now for about 7 years, with numerous contributions in the current
   kernel. All of this is best described in his own words...

   Jeremy Andrews: Please share a little about yourself and your

   Robert Love: I am currently a Computer Science and Mathematics student
   at the University of Florida in Gainesville, although I am originally
   from outside Ft. Lauderdale, FL.

   I am not married, but I have a wonderful girlfriend.

   My programming interests are in operating systems and
   mathematical/scientific computing.

   JA: When do you expect to graduate? Do you have post-graduation plans?

   Robert Love: Expected graduation date is 2004.

   I want to stay in school for a graduate degree, especially if the
   economy remains as-is.

   JA: Where outside of Ft. Lauderdale? That's where I live now, in

   Robert Love: Born and raised in Pembroke Pines.

   JA: How and when did you get started with Linux?

   Robert Love: My first personal computer running Linux was in late
   1994, on a 1.0 kernel. This was my first real experience with a UNIX,
   too. I had a 386SX (thanks Mom!) and switched from a beta of Windows
   95. I used Linux progressively more through 2.0, and finally switched
   completely during 2.2. I've been all Linux ever since.

   I made the switch partly for the obvious reasons: I believe Linux is a
   powerful and robust system with good applications. The greater reason,
   however, was I love the ability to play with the system and the
   community that surrounds that.

   JA: Can you describe a few key differences between the 1.0 kernel
   compared to the current 2.4 kernel?

   Robert Love: I would be surprised if 1.0 and 2.4 shared more than a
   couple pages of code. So, everything is different.

   JA: What was it like to install and use the 1.0 kernel? What software
   was even available?

   Robert Love: Thankfully, come 1994/1995, Linux was evolving nicely. At
   this time SLS and Slackware were both making solid full-featured
   distributions. I first used Slackware 2.0. You had to download the
   thing in disk sets -- maybe you still do. I remember disk set A was
   like the core, and disk set N provided networking, etc.

   By this point Linux had networking, X worked (XFree86 2.0!), and a
   decent amount of hardware was supported.

   The biggest problem from today was the lack of documentation, which
   was compounded by the fact things were fairly complicated to get
   setup. I remember I couldn't get PPP working... I eventually gave up.
   If I could of gotten PPP working, I may of switched to Linux then!

   JA: What are some of the contributions you've made?

   Robert Love: Right now most of my time is going into maintaining the
   preemptible kernel patch. This patch allows a lower priority process
   to be preempted, even if operating in kernel space, resulting in
   improved system response. The patch was originally the work of
   MontaVista - a great company - so I am working closely with them and
   the rest of the community. It is a fun project. We are aiming for
   inclusion in 2.5.

   I'm pretty random, otherwise.. I fix bugs, optimize, and clean cruft
   up. I wrote the i815 and AMD761 AGP GART code. I also maintain some
   other odds and end patches.

   JA: What kinds of changes have to be made to the kernel by your patch
   to allow for kernel preemption?

   Robert Love: The model we use is to allow the kernel to be preempted
   at any time when it is not locked. Under this design, when an event
   occurs that causes a higher priority task to become runnable, the
   system will preempt the current task and run the higher priority task.

   We had to modify the interrupt code in entry.S to prevent some
   situations and to allow preemption on return from an interrupt
   handler. However, we can't preempt within critical regions for the
   same reason we can't allow concurrency within them with SMP -- so we
   prevent preemption while holding a spinlock. The bottom half handler
   and scheduler were also modified to prevent preemption while they are

   JA: How stable do you consider your patch, as it stands today? I do
   all of my work on a single Linux server, which I am continually
   upgrading with new kernels and new software. Improved overall system
   response is certainly attractive, but do you consider the patch stable
   enough for an 'every-day-use' server?

   Robert Love: The patch is very stable. I would feel confident
   suggesting anyone use it. We have a lot of users and are receiving a
   lot of feedback. I think we have the obvious bugs taken care of. I use
   the patch on my main system.

   The good thing is we know the design and implementation is right. With
   our recent patches, I feel we have taken care of any problems.

   JA: How significant an improvement in system response time do you see?

   Robert Love: Very significant.

   A 200% improvement in system latency has been noted. With the
   preempt-kernel patch, average system latency is now around 1ms, with
   spikes up to 10ms. There are still a few long-held locks that kill
   response time, and some of them are 100ms+. Thankfully they occur in
   specific places, like switching VCs.

   JA: There is another patch available (maintained by Andrew Morton)
   with the same end goal. How is your patch different?

   Robert Love: The source of the problem both patches are aiming to
   solve is that kernel code is non-preemptible. Thus, it runs until it
   completes or explicitly yields control. If userspace events happen
   during kernel operations, they must wait until the kernel finishes.
   This waiting is what causes the system latency we are aiming to fix.

   Andrew's low-latency patch places conditional scheduling calls in
   strategic locations throughout the kernel. These calls effectively
   allow long kernel operations to say "I will be nice and let someone
   else run. Any task need to execute? If so, run!" This breaks up the
   operations, reducing latency in those places.

   The preempt-kernel patch works much differently. It modifies the
   kernel itself to allow kernel code to be preempted. So now the above
   problem doesn't even exist - if a task needs to run, it will run - and
   system response benefits.

   JA: In a recent lkml thread you talked about possibly combining the
   two patches. Is this something you're working towards?

   Robert Love: It is certainly something to consider. Since we can't
   preempt during locks, the longer held locks pose a noticeable threat
   to system latency. The duration of the lock effectively becomes the
   latency of the system for that period.

   One solution is to break the locks like in Andrew's patch. This would
   be the purpose of merging parts of his patch with preempt-kernel, or
   more realistically making a preempt-aware version of his low-latency
   patch. Any of his conditional schedules that occur while a lock is
   held may be useful.

   One step in this direction is identifying the long held locks. This is
   what the preempt-stats patch does. It measures periods of disabled
   preemption and reports what lock caused it.

   This is just one solution to the issue of long-held locks.

   JA: What are some sections of the kernel you've identified to have
   these long-held locks?

   Robert Love: The console layer used to hold some locks for a terribly
   long time, but thankfully Andrew Morton fixed a lot of that recently.
   The frame buffer code disables interrupts for a long time -- avoid
   printing a lot to it. If you scroll a lot, you will have latencies
   even up to 500ms. Switching virtual terminals is bad. blkdev_close()
   holds a lock a long time. Some module operations take awhile. If the
   VM starts thrashing, it causes a lot of latency, since it does all
   this thrashing while holding a lock. VFS has a few long-held locks,

   JA: How realistic is it that your patch will get into 2.5? What are
   some of the obstacles?

   Robert Love: It all depends on Linus's opinion of the patch, and what
   the other hackers argue for. Linus before has suggested he is
   interested in the idea, I hope that interest helps get the patch

   We have a very large userbase for an independent patch and that
   obviously helps a lot. We have gotten a lot of feedback including
   benchmarks that show strong improvements.

   The counterargument is that a preemptible kernel lowers system
   throughput. This is a very valid point, and one we will need to
   address. Tests now show a loss in throughput of 0-5% -- I think that
   is worth 200% increase in response! More so, some scenarios show an
   increase in throughput, since we thread better. Despite this, for
   systems where any loss of throughput is unacceptable, the preemptible
   kernel is a config setting.

   JA: What main tools do you use when developing? Please describe your
   environment(s), computer(s) and the methods you use.

   Robert Love: My main machine is a P3-733 with 384MB RAM and U2W SCSI
   disks. I also have an IBM ThinkPad and an assortment of older stuff
   lying around the apartment. I run the latest RedHat Rawhide with
   Ximian GNOME.

   I code in vi because I don't want to learn another OS. :)

   I am typically in X, so I usually have a handful of gnome-terminal and
   gvim windows open with my current project. I tend to grep around,
   research as needed, and then start coding. I rarely do design on
   paper. I rely on lkml and personal email often -- I like bouncing
   ideas off others.

   JA: What other operating systems do you use? What do you like and
   dislike about them, compared to Linux?

   Robert Love: I have a workstation with Windows 2000 and an old SGI
   Indy running Irix. Otherwise everything else is Linux. I keep Windows
   around for Powerpoint and in case AbiWord just doesn't cut it.

   The greatest dislike about the other systems is that they are not open
   source. I am not a free software zealot - although I praise its merits
   - but the open availability of source and the community surrounding
   that is priceless.

   JA: Have you played with any of the open source BSD operating systems?
   Specifically, have you looked much into their kernels?

   Robert Love: I used to have an OpenBSD machine but I thew it into a
   dumpster (not because of OpenBSD, though).

   The rest of my knowledge is academic. I am impressed by Matt Dillon's
   work in FreeBSD (he wrote the VM among other stuff). Their SMPng work
   is coming along nicely (they lag Linux in SMP scalability) and they
   are even considering implementing a preemptible kernel.

   JA: Do you know what form of preemptibility they intend to use? Will
   it be like used in your patch, or Andrew's patch, or perhaps a

   Robert Love: They are considering a fully-preemptible kernel, like
   what my patch does. I don't know what they have done towards this,
   other than it has been discussed by the FreeBSD Core team.

   JA: What reflections can you offer on the current state of the 2.4
   kernel series? Do you consider it to be stable?

   Robert Love: I think we have come an amazing distance since 2.2. The
   work done during 2.3 and now in 2.4 is incredible. Some of the
   advancements that we did, especially some of the work by Linus and
   Ingo, are incredible. 2.4 can play in some very
   high-performance/high-scaling playgrounds now.

   This is not to say I am completely happy with 2.4. At this point, I
   think ripping out Rik's VM was a mistake. If Linus wants to try
   something different in 2.5, by all means he should. In the stable
   series, however, I really think we need to make much smaller measured
   changes. We didn't just throw away a year plus of VM work, but we also
   tossed out all the corresponding documentation and understanding.
   Nonetheless, it seems the VM is shaping up OK. I use Alan's kernel,
   which has stuck to Rik's VM and we are working on improving that. Who
   knows what VM we will have when 2.4 is handed to Alan.

   JA: If you were suddenly in charge of the kernel, how would you handle
   these issues?

   Robert Love: First, the kernel would degenerate into a pile of cat
   phlegm with me at the helm. For whatever complaints I or others have
   against Linus, he is not just an incredible hacker but a superb
   manager. Linux needs him.

   On the issue of the VM, I would follow Alan's lead for 2.4. Rik's VM
   is tested and documented. I think we needed better testing of it. Too
   many of Rik's ideas and patches were ignored. Other patches were
   merged that shouldn't of been. We needed to merge VM work
   little-by-little, alongside no conflicting patches, and test the
   results. We have something close to this in Alan's tree now.

   For 2.5, of course, anything goes. Rip out whatever. I think Andrea's
   VM is performing well. Maybe it is better. It is just a bad time to
   find out, IMHO.

   JA: What do you most look forward to in the upcoming 2.5 kernel? When
   do you expect work to start here?

   Robert Love: Of course, a preemptible kernel :)

   Beyond that, I'd like to continue the quest for finer-grained locking
   and overall cleanup. I think a block device, SCSI layer, and console
   layer rewrite is planned. We should see NAPI and/or some form of IRQ
   throttling. Ben LaHaise's async I/O work should be merged, too.
   Keith's new kbuild is looking great. I look forward to all this.

   I really think we need a total rewrite of the tty system, but I don't
   know who is up to that...I surely am not.

   I don't think anyone, Linus included, has a clue when 2.5 will start.
   I originally figured somewhere in March-May -- a couple months after
   2.4.0, like usual. Now that a year is approaching (2.4 was released 4
   Jan 2001), I can't even guess. I suspect when both the VM proves solid
   and Alan and Linus are fully synced, we will see a 2.5.0. I hope that
   is soon.

   JA: Have you met Linus? Alan? Other kernel hackers?

   Robert Love: I haven't met any of the "famous" hackers. I look forward
   to meeting them, someday. Perhaps at a conference or petting zoo.

   JA: What do you enjoy doing in your non-Linux time?

   Robert Love: Outside of Linux, school, and the girlfriend I like
   mountain biking, tinkering with my car, and going to rock shows.

   JA: What are a few good shows you've seen recently? Gainsville
   certainly has had a lot of famous music come out of it.

   Robert Love: Hm, not much recently. Cadillac Blindside did a show a
   couple weeks ago, I like them. Jane's Addiction played last night. Of
   course, Less than Jake is local, so I see them a lot. They are great.

   JA: What tips and inspiration can you offer aspiring kernel hackers?

   Robert Love: Read the source, play with the source, and bathe
   regularly. For beginners, O'Reilly makes some superb books
   ("Understanding the Linux Kernel" by Bovet and Cesati and "Linux
   Device Drivers, 2ed" by Rubini and Corbet are excellent). But the key
   is to read the source. You have the source to the whole system -- read
   it, change it, learn it.

   I subscribed to lkml years ago, but I didn't make my first serious
   contribution until 2.3. Lurking on lkml is a great way to see The
   Greatest shoot it out and you can learn a lot from that.

   JA: Thanks for taking the time to thoroughly answer all my questions!
   Because of our conversation, I am definitely going to apply your
   preemptible kernel patch on my system.

   Robert Love: Great! Let me know how it goes and thank the great guys
   at MontaVista, too.

   Follow up: After talking with Robert, I dowloaded his patch and gave
   it a test run (applied to 2.4.10-ac12). Wow! The improvement is
   amazing! Programs like VMWare and Mozilla used to freeze up my system
   for a short while during launch and exit. With Robert's patch, I can
   launch them both at the same time and my system remains completely
   responsive. I also noticed that xmms no longer "skips" while these big
   programs launch. There's no question that I'll be using this patch
   always, now. I hope it gets merged with the main source in 2.5.

   Any comments you have about this interview, or this series, can be
   posted here or mailed to me at jeremy at kerneltrap.com

   Related Links

     More about Interviews

     News by jeremy

                     Most read story about Interviews:
                           Interview: Robert Love

                        Last news about Interviews:
                           Interview: Robert Love

   [comment.gif] Printer Friendly Page width=15 height=11    Send this
   Story to a Friend width=15 height=11 

   "Interview: Robert Love" | Login/Create Account | 10 comments | 26229
           Threshold [0.] [Thread.....] [Oldest First........] Refresh
   The comments are owned by the poster. We aren't responsible for their

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Sunday, October 14 @ 14:34:28 EDT
   This is a fantastic interview! Rarely are the questions so appropriate
   and the answers so accurate. This answered all my questions (even
   regarding the Merge of Love's and Morton's patches to eliminate the
   locking issues). The patch itself is superb, and I recommend it to
   anyone. If you're a newbie who's never compiled a kernel, the preempt
   patch is a great excuse to do so. It makes a Linux system fly :-)
   Thanks both to a knowledgable interviewer and an excellent
   interviewee. World domination seems just a little bit closer today. I
   hope we see more interviews like this (and less of the clueless ones).
   -- Johan
   Btw. It's not at all obvious how to post a comment to an article at
   this site. Perhaps the webmaster should make it a little clearer. It
   took me the better part of a day to figure that one out :-)

                                 [ Reply ]

     * Re: Interview: Robert Love by Jeremy on Sunday, October 14 @
       15:23:50 EDT

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Sunday, October 14 @ 18:58:13 EDT
   a guy who has time to write kernel code in college?
   Wow, he must forgo my average day of typical drinking, hanging out,
   and then going to work...
   i envy that....

   [ Reply ]

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Sunday, October 14 @ 21:17:36 EDT
   Well, I did try the patch out with kernel 2.4.12 and with pre13. While
   accurate measurement does indeed report an increase in latency, I can
   for the life of mine not notice it on an empirical basis.
   While I do appreciate Love's work, I do not agree with some of his
   comments. First, as Linus himself said, if latency sucks in the
   kernel, then we should check why it sucks with or without preemptive
   scheduling. If the latency is bad in the stock kernel, then it should
   be fixed anyway.
   Also, I think Love is wrong to say that the old van Riel VM should
   have been continued. Not only is the VM not only Riel's but many
   others contributed (Andrea himself, too, as well as Marcelo Tosatti,
   Ben LaHaise, Linus and others). But it became obvious that the VM up
   to 2.4.10 was a design liability. You can try to fix something that
   was designed badly, but it will never become a beauty. I think Linus
   decision to scrap the old VM and go with the Arcanglei VM was
   couragous and right. Having a functioning and stable Linux box should
   not be deferred to 2.5 when we can do it with 2.4. already.
   Arcangeli's VM is stable, acts predictably (something that the old VM
   really never achieved) and it makes the swap space look again as in
   2.2. days. Additioanlly, the design is much simpler and eaiser to
   understand. People will catch up fast with it.
   So, for me it goes back to using kernels unpatched with the preemptive
   patch. It doesn't co-exist with Mosix anyway, which for me is vital.
   Moshe - www.moshebar.com

   [ Reply ]

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Monday, October 15 @ 04:16:55 EDT
   Looks to me like this is a throughput vs. latency contradiction. Seems
   to me that servers need the througput and workstations need the (lack
   of) latency.
   My suggestion is to make it configurable at boot time.
   Oh, and why wait for 2.5, put in 2.4.x!

   [ Reply ]

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Monday, October 15 @ 05:02:36 EDT
   Coming from an embedded background the latency issue is extremely
   important. If we were to use an embedded Linux we would need a known
   maximum latency and a latency of a couple of hundred milliseconds
   doesn't cut the mustard.
   I agree with Linus that latency needs to be fixed preemptive kernel or
   not. However, looking at the test results so far the preemptive patch
   appears to be a useful stop-gap if nothing else. Even then for systems
   where latency is a major issue the preemptive patch would be useful as
   a kernel configuration.

   [ Reply ]

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Monday, October 15 @ 10:18:14 EDT
   I installed the patch, and it's ... FAST!
   This is ideal for the Linux Desktop. I've noticed no slowdown from
   installing the patch. Things don't stop and chunk like they used too.
   For instance, in mozilla or konqueror, when a page would start to load
   it used to stop everything for half a second and then load the page.
   Now it's just smooth a la Windows.
   Very slick, I'm very happy. Linux is now as fast, if not faster then
   XP for loading and whatnot.

   [ Reply ]

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Tuesday, October 16 @ 01:05:26 EDT
   OK I'ts this is going to look like a pretty silly question but where
   do you get this particular patch from?
   If it isn't in the kernel sources, where does it live?

   [ Reply ]

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Tuesday, October 16 @ 01:40:23 EDT
   It's OK I answered my own comment but it took me a while to find.
   Maybe you could link to it next time (or am I just lazy).

   [ Reply ]

   Re: Interview: Robert Love (Score: 0)
   by Anonymous on Tuesday, October 16 @ 11:37:05 EDT
   [The FreeBSD Project is] even considering implementing a preemptible

   Sorry to rain on your parade, but the FreeBSD-CURRENT kernel has been
   fully preemptive since January or February 2001. Since then we've been
   working on pushing down locks (to avoid holding large-grained locks
   too long).

   If you weren't too narrow-minded to check out the competition once in
   a while, you'd avoid making such a fool of yourself in a public forum.

   des at ofug.org

   [ Reply ]


                       Web site powered by PHP-Nuke 
   All logos and trademarks in this site are property of their respective
    owner. The comments are property of their posters, all the rest is ©
                        2000, 2001 by kerneltrap.com
   This web site is running PHP-Nuke, a web portal system written in PHP.
       PHP-Nuke is Free Software released under the GNU/GPL license.
   You can syndicate our news using the XML link Backend.php or the file
         Palm and Avantgo users can access our news at avantgo.php.
Unsubscribe: send email to listar at linuxfromscratch.org
and put 'unsubscribe blfs-support' in the subject header of the message

More information about the blfs-support mailing list