From: eLinux.org
This page is intended to serve as a collecting point for presentations, documents, results, links and descriptions about testing Realtime performance of Linux systems. In the first section, please upload or place links to presentations or documentsion on the subject of RT testing for linux.
This document uses the definitions for real time terminology found in: Real Time Terms
Here is a list of programs that have been used for realtime testing:
This requires a separate machine to send the signal on the parallel port and receive the response. (Can this be run with a loopback cable? It seems like this would disturb the findings).
Are there any writeups of use of this test?
This program is a very simple test of how well a periodic interrupt is processed. The program programs a periodic interrupt using /dev/rtc to fire at a fixed interval. The program measures the time duration from interrupt to interrupt, and compares this to the expected value for the duration. This simple program just prints a list of variances from the expected value, forever.
This program uses the TSC in user space for timestamps.
This program (latency.c) extends realfeel in several ways:
Trevor Woerner wrote an interesting test which received an interrupt on the serial port, and pushed data through several processes, before sending back out the serial port. This test requires an external machine for triggering the test and measuring the results.
See Trevor Woerner's latency tests
Benno Senoner has a latency test that simulates and audio workload. See http://www.gardena.net/benno/linux/audio/
Used (and extended??) by Takahashi Iwai - see http://www.alsa-project.org/~iwai/latencytest-0.5.6.tar.gz
Feature | Rf-etri | Williams | LRTB |
---|---|---|---|
Is it platform specific (for target)? | yes - i386 | no, but requires serial port on target | no, but requires parallel port on target |
How is interrupt generated? | periodic timer programmed via /dev/rtc | data on serial port | data on parallel port |
What does test measure? | interrupt and scheduling latency | end-to-end response latency | end-to-end response latency |
Here are some things that will kill your RT performance:
This is a list of issues and techniques for dealing with them, having to do with testing realtime performance in Linux.
At one of the sessions at ELC 2007, Nicholas McGuire stated that a pingflood test is actually a poor test of RT performance, since it causes locality in the networking code rather than stressing the system.
Here is a list of issues that have to be dealt with:
Quote about latency-test from Ingo:
I'm seeing roughly half of that worst-case IRQ latency on similar
hardware (2GHz Athlon64), so i believe your system has some hardware
latency that masks the capabilities of the underlying RTOS. It would be
interesting to see IRQSOFF_TIMING + LATENCY_TRACE critical path
information from the -RT tree. Just enable those two options in the
.config (on the host side), and do:
echo 0 > /proc/sys/kernel/preempt_max_latency
and the kernel will begin measuring and tracing worst-case latency
paths. Then put some load on the host when you see a 50+ usec latency
reported to the syslog, send me the /proc/latency_trace. It should be a
matter of a few minutes to capture this information.
Ingo wrote:
also, i'm wondering why you tested with only 1,000,000 samples. I
routinely do 100,000,000 sample tests, and i did one overnight test with
more than 1 billion samples, and the latency difference is quite
significant between say 1,000,000 samples and 100,000,000 samples. All
you need to do is to increase the rate of interrupts generated by the
logger - e.g. my testbox can handle 80,000 irqs/sec with only 15% CPU
overhead.
Another note from Ingo - see here
> First things first, we want to report back that our setup is validated
> before we go onto this one. So we've modified LRTBF to do the
> busy-wait thing.
here's another bug in the way you are testing PREEMPT_RT irq latencies.
Right now you are doing this in lrtbf-0.1a/drivers/par-test.c:
if (request_irq ( PAR_TEST_IRQ,
&par_test_irq_handler,
#if CONFIG_PREEMPT_RT
SA_NODELAY,
#else //!CONFIG_PREEMPT_RT
SA_INTERRUPT,
#endif //PREEMPT_RT
you should set the SA_INTERRUPT flag in the PREEMPT_RT case too! I.e.
the relevant line above should be:
SA_NODELAY | SA_INTERRUPT,
otherwise par_test_irq_handler will run with interrupts enabled, opening
the window for other interrupts to be injected and increasing the
worst-case latency! Take a look at drivers/char/lpptest.c how to do this
properly. Also, double-check that there is no IRQ 7 thread running on
the PREEMPT_RT kernel, to make sure you are measuring irq latencies.
Person | Company | Hardware | Kernel | test method | Measurement method | Results | |
---|---|---|---|---|---|---|---|
Sangbae Lee | Samsung | OSK - OMAP (ARM) 192 MHZ) | 2.4.20 and 2.6.10 | using two machine test | ZI instrumentation - measure interrupt reponse latency | 2.4.20 - 30~35us, 2.6.10 - 30~35us max | |
Sangbae Lee | Samsung | MIPS 264 MHZ | 2.6.10 ?? | ?? | ?? | ?? | |
Katsuya Matsubara | IGEL | SH4 | 2.6.?? | ?? | ?? | ?? | |
YungJoon Jung | ETRI | Via Nehemiah (i386) | 2.6.12 | periodic interrupt | rf-etri - measure scheduling latency minus interrupt latency | 30 us max scheduling latency with RT-preempt | |
Tsutomu Owa | Toshiba | Cell (ppc64) | 2.6.12 | ?? | ?? | ?? |
[Add links here, most recent at top]
[TBD Linux Kernel's performance comparison] by HyoJun Im of LG at RTWG 2nd Face-to-Face Meeting in Korea
Analysis of Interrupt Entry Latency in Linux 2.4 vs 2.6 by !SangBae Lee of Samsung for ELC 2007
Porting and Evaluating the Linux Realtime Preemption on Embedded Platform by Katsuya Matsubara of Igel at ELC 2007
Realtime Preempt Patch Adaptation Experience (and Real Time BOF notes) - !YungJoon Jung of ETRI at ELC 2007
Performance Measurement of PPC64 RT patch (update) (english text)
Porting pre-empt RT patch on SuperH (english text)
Performance Measurement of PPC64 RT Patch (english text)
Linux Realtime Preemption and Its Impact on ULDD by Katsuya Matsubara & Hitomi Takahashi of IGEL, for CELF Jamboree 11
Experience with Realtime Performance
PREEMPT-RT vs I-PIPE: the numbers, take 3
Trevor Woerner's latency tests
Audio Latency on Linux Kernels
Linux Scheduler Latency - by Clark Williams, Red Hat, March 2002
Realfeel Test of the Preemptible Kernel Patch - article in Linux Journal, 2002 by Andrew Webber
Real Time and Linux, Part 3: Sub-Kernels and Benchmarks
[attachment:p-a03_wilshire.pdf Real Time Linux: Testing and Evaluation] - By Phil Wilshire of Lineo at the Second Real Time Linux Workshop, 2000
[FIXTHIS - need to scan for past papers]
Darren Hart wrote:
I have contributed some testing results to Steven Rostedt's OLS RT Internals
paper. That will be available to link to after the conference sometime.
Nicholas said:
There are a number of publications related to both benchmarking and analysis of hardware related artifacts (cache,BTB,TLB,etc.) which were published at the real-time Linux Workshops.
Here is a link to the RTLF events page:
So far, I've scanned 1999-2000 for interesting links.
This section has random stuff I haven't organized yet:
http://eaglet.rain.com/rick/linux/schedstat/
Low-latency HowTo (for audio) - http://lowlatency.linuxaudio.org/
Nicholas McGuire wrote:
The tests noted in the LKML post on this page are very problematic,
ping - -f is not testing RT at all, it keeps the kernel in a very small active
page set thus reducing page related penalties, the while loop using dd
is also not too helpfull as it will de-facto run only in memory and cause
absolutely no disk/mass-storage related interaction (try the same with
mount -o remount,sync / first and it will be devastating ! (limited to ext2/ext3/ufs))
Nicholas McGuire wrote:
The big problem with RT tests published is that they are all looking at the good case,
they are loading the system but assuming successfull operations. The worst cases pop
up when you run in the error paths of the kernel - then a trivial application can
induce very large jitter in the system (run crashme in the background and rerun
the tests...)
Also lmbench can give a statistic view of things (and not even that very precisely in some case i.e. context switch measurements are flawed) so this is not of much help for descision makers which variant to use - it does not help if the average performance is good but the mobile phone or mp3 klicks at 1s intervals "deterministically" - so I guess RT benchmarks need a notion of usage-profile to be of value.