RHEL-RT InstallGuide
From RHEL-RT
DRAFT - MRG Realtime Installation Guide
This chapter contains the installation guide for the realtime component of the MRG product set which consists of the 3 components:
- Realtime kernel - the subject of this chapter - which is not strictly required to run with either the AMQP messaging or grid components. It is possible to utilize just the realtime component.
- AMQP messaging middleware - which is cross platform. When run on RHEL5 Linux, the messaging middleware is optimized for lowest latency when run in conjunction with the realtime kernel. Keep in mind that it is not strictly required to run the realtime kernel. Rather the combination is optional when performance is the objective.
- Grid scheduler - this component does not have dependencies on the realtime kernel.
The candidate class of deployments for the realtime kernel include:
- Combined use with the AMQP messaging middleware for optimal low-latency messaging response times.
- Independent of AMQP messaging for server configurations where kernel low-latency is required for deterministic operation.
Pre-Installation Considerations
For anyone considering evaluating the performance benefits of the realtime kernel, it is crucial to understand 2 points:
- The importance of tuning
- Establishing the right perfornamce expectations
Both of these points are discussed in following Red Hat blog article: http://www.press.redhat.com/2007/09/24/building-an-informed-realtime-customer-base-at-high-performance-on-wall-street/
To quickly recap that info:
- kernel system tuning - gets you the vast majority of the improvement in determinism. For example, in many workloads, through proper system tuning the consistency of results is improved by 90%. This is why we typically recommend that customers first perform the suggested system tuning of standard RHEL to see if it satisfies their objectives.
The importance of system tuning is just as important when using the realtime kernel as standard RHEL. In fact, if you simply took an untuned system running standard RHEL and plopped in the RHEL-RT kernel, you are unlikely to notice any benefit at all. Why? Because as mentioned above, the standard tuning gets you 90% of the determinism improvement, and the realtime kernel yields that last 10% of determinism required by the most demanding workloads.
Establishing the right performance expectations - refers to the fact that the realtime kernel is not a panacea. Its objective is consistent, low-latency determinism = predictable response times. There is some additional kernel overhead associated with the realtime kernel (primarily due to handing interrupts in separately scheduled softirq threads). This increased overhead in some workloads results in some degradation in overall throughput. The exact amount is very workload dependent, ranging from 0% to 30%. It is the cost of determinism.
If your kernel latency requirements are in the millisecond range, then the standard RHEL5 kernel is for you. If your workload has such stringent low-latency determinism requirements for core kernel features such as interrupt handling and process scheduling at the microsecong range, then the realtime kernel is for you.
For information on suggested system tuning to perform to establish a performance
baseline prior to installing the realtime kernel, refer to the General System Tuning section of the online companion document: http://rt.et.redhat.com/page/RHEL-RT_HOWTO
Prerequisite Base Operating System Installation Requirements
A prerequisite to installing RHEL-RT is to do a RHEL5 fresh install. Notes:
- We aren't testing/supporting update installations from RHEL4. Rather, a full system installation is required.
- We certainly do not support a mix of RHEL4/RHEL5 components. It won't work to start with RHEL4 and drop in the realtime kernel and glibc - there are many more dependencies than that.
- RHEL5.1 is the prerequisite baseline install. The originl RHEL5.0 release is not recent enough.
- When doing the initial system installation, you should select the set of packages you normally do to meet the needs of your application environment (ie, nothing realtime specific is required on the initial install).
Post install, you will follow the instructions to incrementally add the realtime specific packages from a separate yum repositiory.
Supplied Realtime Kernel Variants
There are numerous kernel variants provided. The main deployment realtime kernel is identified below as production. Additionally there are several other debug kernels which have successively more diagnostic code compiled in. The reason for doing this is that as the amount of debug code is increased, so does the overhead. The progression from least overhead to most is: kernel-rt-trace, kernel-rt-debug. Lastly the vanilla kernel does not include the realtime features. This is used help distinguish whether bugs were introduced in the realtime features, rather than being an inherent bug in the baseline kernel. The set of kernels is separately provided for x86 (32-bit) and x86-64 (64-bit) systems. Select either set based on whether you intend to run in 32-bit or 64-bit mode.
| Variant | Intended Usage | Notes |
|---|---|---|
| x86 (kernel-rt) | production - Standard 32-bit production realtime kernel | x86 version with PAE enabled (to allow for up to 16G memory) |
| x86 tracing (kernel-rt-trace) | debugging - 32-bit trace kernel | latency tracer enabled - Used to locate latency hotspots |
| x86 debug (kernel-rt-debug) | debugging - 32-bit debugging kernel | latency tracer enabled + other debugging options - Used to debug RT kernel failures |
| x86 vanilla (kernel-rt-vanilla) | debugging - 32-bit base kernel | No RT features, used for comparison |
| x86_64 (kernel-rt) | production - Standard 64-bit production kernel | |
| x86_64 tracing (kernel-rt-tracing) | debugging - 64-bit trace kernel | latency tracer enabled - Used to locate latency hotspots |
| x86_64 debug (kernel-rt-debug) | debugging - 64-bit debugging kernel | latency tracer enabled + other debug options - Used to debug RT kernel failures |
| x86_64 vanilla (kernel-rt-vanilla) | debugging - 64-bit base kernel | No RT features, used for comparison |
Included RPMs
This table lists the RPM packages included in RHEL-RT. It briefly describes why this package is included (replacing a standard RHEL5 version).
Legend:
- Name: package name
- Rationale: why is this included in RHEL-RT
- RT-specific? N = no = this is a generic RHEL5 fix. Y = yes = the package differs from standard RHEL5 and its maintenance stream (or is not applicable on a standard RHEL5 kernel).
- Required? Y = yes = this package is mandatory for correct realtime behavior. N = no = usage is optional.
| Name | Rationale | RT Specific? | Required? |
|---|---|---|---|
| kernel-rt | low latency and preemption functionality | Y | Y |
| rtctl | System startup script used to configure the default realtime scheduling priorities of kernel threads | Y | Y |
| dslimit | shell tool to run a command with a soft CPU limit | N | N |
The following packages contain test and debugging programs associated with RHEL-RT.
| Name | Rationale | RT Specific? | Required? |
|---|---|---|---|
| kernel-rt-devel | headers and libraries for kernel development | Y | N |
| kernel-rt-debuginfo | symbols for profiling and debugging use, such as oprofile or systemtap | Y | N |
| kernel-rt-trace | RT kernel with tracing functions compiled in | Y | N - Provided as a diagnostic tool |
| kernel-rt-trace-devel | headers and libraries for development on trace kernel | Y | N - Provided as a diagnostic tool |
| kernel-rt-trace-debuginfo | symbols for profiling and tracing | Y | N - Provided as a diagnostic tool |
| kernel-rt-debug | RT kernel with debugging functions compiled in (slow) | Y | N - Provided as a diagnostic tool |
| kernel-rt-debug-devel | headers and libraries for development on debug kernel | Y | N - Provided as a diagnostic tool |
| kernel-rt-debug-debuginfo | symbols for profiling and tracing | Y | N - Provided as a diagnostic tool |
| kernel-rt-vanilla | Base kernel for comparisons | Y | N - Provided as a diagnostic tool |
| kernel-rt-vanilla-devel | headers and libraries for development on vanilla kernel | Y | N - Provided as a diagnostic tool |
| kernel-rt-vanilla-debuginfo | symbols for profiling and tracing | Y | N - Provided as a diagnostic tool |
| cyclictest | used to measure timer latency and validate realtime functionality (replaces a similar ancient utility called real-feel) | Y | N - Provided as a diagnostic tool |
| signaltest | used to measure signal passing latency and validate realtime functionality | Y | N - Provided as a diagnostic tool |
| rt-watchdog | when setting kernel and realtime application priorities it is possible to incorrectly elevate application processes too high. This can result in starvation of kernel threads - leading to unexpected or hung systems. This is a tool used to detect such hung systems and provide a means to break out and diagnose | Y | N - Provided as a diagnostic tool |
Download Instructions
Editorial NOTE: these specific download instructions will likely change when the formal beta repository is established.
After you have verified your RHEL5.1 (or later) system installation, obtain the RHEL-RT yum repository description file using the following command:
wget ftp://partners.redhat.com/f227cee111ab00ba82e0e42dc49c63a2/rhel5-rt.repo
Next, as root, move the retrieved file (rhel5-rt.repo) into your local yum repository directory:
# mv rhel5-rt.repo /etc/yum.repos.d
After you've moved the repo description file to where yum can find it, issue the following command (again, as root) to install the RT kernel, rt-setup and the rtctl packages:
# yum install rt-setup
This command line will install the three packages, kernel-rt, rt-setup and rtctl. Basic RT environment setup is performed by rt-setup which is automatically run as part of package installation.
The rtctl package is a startup script that sets the priorities of various kernel threads.
Post-installation Instructions
The following steps are necessary after the realtime packages have been installed:
The realtime kernel is not automatically specified as the default boot kernel. The recommended approach after installing the realtime kernel is to reboot into the grub menu and manually select the realtime kernel. After you have successfully confirmed that the realtime kernel is fully operational on your system, then you can modify grub.conf to make it the default booted kernel.
To confirm that your system is running the realtime kernel, you can run this command:
# uname -a
You can confirm that the realtime kernel is running by seeing the RT designation, for example:
Linux torg 2.6.23-rt1 #2 SMP PREEMPT RT Thu Oct 18 22:57:04 CDT 2007 x86_64 x86_64 x86_64 GNU/Linux
Known Bugs
An up to date listing of known issues can be found here: RHEL-RT KnownBugs
Please review the information on known issues to avoid needlessly wasting time on known problems.
Reporting bugs on RHEL-RT
This page describes how to enter bug reports on RHEL-RT.
Diagnosis
In order to help us most efficiently diagnose problems, it is highly beneficial to differentiate under what scenarios the problem occurs. This primarily consists of triangulating among several kernel variants - with an objective of determining where the problem has been introduced.
- Try reproducing the problem with the standard RHEL5 kernel. If the problem occurs on standard RHEL, bug reports should be entered on the RHEL5 product.
- If the problem does not reproduce when using standard RHEL5 kernels, then the bug is likely the result of changes introduced either:
- The upstream kernel on which RHEL-RT is based. ie, RHEL5 is based on 2.6.18 and RHEL-RT is based on 2.6.21 (or later).
- Realtime specific enhancements Red Hat has applied on top of the baseline (2.6.21) kernel.
- To differential it is highly helpful to see if you can reproduce the problem on unmodified upstream 2.6.21 kernels. For this reason, in addition to providing the realtime kernel, we also provide a -vanilla kernel variant. The vanilla kernel is the unmodified upstream kernel build (without the additional realtime additions).
Bug reporting
Editorial note - This bug reporting info is subject to change based on how we establish the official beta bugzilla product and component tags.
If you have determined that the failure is specific to RHEL-RT follow these instructions to enter a bug report:
- You first need an established bugzilla account: Red Hat bugzilla
- When entering a new bug, the first step is to identify the product the bug occurs in. RHEL-RT appears in the product list as: Red Hat Enterprise Linux RT. Is it imperitave that you use this product identifier to distinguish RHEL-RT issues from standard RHEL5 issues.
- Then continue to enter the bug information as done in standard RHEL, by designating the package and detailed problem description. When entering the detailed problem description be sure to include mention of whether you were able to reproduce the problem on standard RHEL5 or the supplied kernel-rt-vanilla kernel variant.
Realtime Kernel Tuning
The realtime kernel introduces numerous performance tuning parameters which were not otherwise present in standard RHEL5.1. In order to achieve optimal low-latency determinism it is necessary to perform realtime specific system tuning, in addition to the general system tuning described previously. There is detailed online supplementary information describing the primary realtime tuning recommendations. Refer here for that material: http://rt.et.redhat.com/page/RHEL-RT_RealtimeSystemTuningHowto
Latency Tracer Feature
One of the new diagnostic facilities provided with the realtime kernel is the latency tracer. The latency tracer is a peak detector which is used to identify the longest running non-preemptable kernel codepaths. This is particularly useful for identifying whether non-deterministic performance results are attributable to the kernel, vs user space components. The Red Hat kernel developers heavily utilize the latency tracer to prove correctness of low-latency enhancements. In customer deployments, the tool is most useful to differentiate whether delays are in the kernel or application.
It is highly recommended that you review the detailed usage instructions and accompanying case study of the latency tracer in the supplementary online information: http://rt.et.redhat.com/page/RHEL-RT_RealtimeSystemTuningHowto#Realtime_Performance_Monitoring_Tools
Application Development Tips for Realtime
While most of the information in this installation guide pertains primarily to system setup, there are additional realtime recommendations for application developers. Recommendations for application developers to consider to achieve optimal determinism - to best capitalize on realtime features can be found in the following online supplementary material: http://rt.et.redhat.com/page/RHEL-RT_ApplicationTuningHowto
RHEL5 Differences
This section describes ways in which RHEL-RT is different from standard RHEL5.
- NMI Watchdog disabled
- In the x86-64 kernels of standard RHEL5, the kernel's NMI watchdog capability is enabled by default. The NMI watchdog is disabled by default in the RHEL-RT kernels, as a minority of hardware platforms had erroneous triggering.
- To enable: echo 1 >> /proc/sys/kernel/nmi_watchdog, or specify as a boot option nmi_watchdog=1
- PAE is configured in the x86 realtime kernel
- RHEL5 for x86 (32 bit) includes the following kernel variants (using example version numbers which will vary)
- kernel-2.6.18-8.el5 - this kernel does not have PAE (Physical Address Extention) enabled.
- kernel-PAE-2.6.18-8.el5 - this kernel does have PAE enabled - allowing it to access more than 4G of memory.
- In the RHEL-RT x86 kernel PAE is configured as we anticipate that most systems requiring RT capability will have at least 4G of memory. Note: the maximum memory supported on x86 systems is 16G (which is the same limit as standard RHEL5). Larger memory configurations are supported on the x86-64 kernel.
- RHEL5 for x86 (32 bit) includes the following kernel variants (using example version numbers which will vary)
- 3rd party kernel modules are incompatible with standard RHEL5
- Kernel modules are inherently specific to the kernel they are built for. Since the RHEL-RT kernel is substantially different from the standard RHEL5 kernel, kernel modules are incompatible. In other words, you can't simply take 3rd party driver modules from RHEL5 and use them as-is on RHEL-RT.
- The following are some example 3rd party drivers which ship against standard RHEL5 which do not currently have a RHEL-RT build: EMC Powerpath, NVidia graphics, advanced storage adapter configuration utilities from Qlogic & Emulex.
- Note, the user space syscall interface is compatible with standard RHEL5. These compatibility restrictions pertain only to kernel modules not supplied by Red Hat.
Caveats
/dev/rmem feature
Included in the RHEL-RT is a kernel module called rmem.ko. This module is not loaded by default, and is merely provided to meet the realtime Java conformance tests - specifically the TCK test in the JVM's RTSJ conformance suite. It is a RTSJ requirement of this conformance test that Java programs have direct access to physical memory. This /dev/rmem capability allows user applications to map any arbitrary memory region.
WARNING: By enabling direct memory access to Java applications, numerous normal system security mechanisms are bypassed, leaving the system more vulnearable to attacks by malicious users. For this reason, it is strongly suggested that this feature be exclusively used for RTSJ certification purposes only - not for production deployment.
The kernel module implementing this /dev/rmem capability is being provided only to meet the requirements of RTSJ conformance criteria. We have taken extra measures to ensure the capability is only enabled as a result of direct root system administrator action. The kernel module is by default not loaded. To further prevent accidental usage, the capability is disabled unless the following boot option is specified unprotected_address_spaces=1. As a result of explicitly loading the rmem.ko kernel module the kernel will have a tainted flag, suggesting that support will require reproducing any problems without the presence of the rmem.ko module.
There is another related boot option for this feature. the boot option is:
alloc_rtsj_mem.size=<memsize>
Where an example size is 1024 (bytes). The purpose of this boot option is to reserve contiguous physical kernel memory at boot time for later usage by the java runtime. This memory is allocated at boot time prior to the system memory becoming fragmented. The purpose of which is to avoid allocation failures for large contiguous memory.
In summary: don't use it.
