Solaris Internals: Solaris 10 and Open Solaris Kernel Architecture

Welcome to the second edition of Solaris™ Internals and its companion volume, Solaris™ Performance and Tools. It has been almost five years since the release of the first edition, during which time we have had the opportunity to communicate with a great many Solaris users, software developers, system administrators, database administrators, performance analysts, and even the occasional kernel hacker. We are grateful for all the feedback, and we have made specific changes to the format and content of this edition based on reader input. Read on to learn what is different. We look forward to continued communication with the Solaris community.\ About These Books\ These books are about the internals of Sun’s Solaris Operating System—specifically, the SunOS kernel. Other components of Solaris, such as windowing systems for desktops, are not covered. The first edition of Solaris™ Internals covered Solaris releases 2.5.1, 2.6, and Solaris 7. These volumes focus on Solaris 10, with updated information for Solaris 8 and 9.\ In the first edition, we wanted not only to describe the internal components that make the Solaris kernel tick, but also to provide guidance on putting the information to practical use. These same goals apply to this work, with further emphasis on the use of bundled (and in some cases unbundled) tools and utilities that can be used to examine and probe a running system. Our ability to illustrate more of the kernel’s inner workings with observability tools is facilitated in no small part by the inclusion of some revolutionary and innovative technology in Solaris 10—DTrace, a dynamic kernel tracing framework. DTrace is one of many new technologies in Solaris 10, and is used extensively throughout this text.\ In working on the second edition, we enlisted the help of several friends and colleagues, many of whom are part of Solaris kernel engineering. Their expertise and guidance contributed significantly to the quality and content of these books. We also found ourselves expanding topics along the way, demonstrating the use of dtrace(1), mdb(1), kstat(1), and other bundled tools. So much so that we decided early on that some specific coverage of these tools was necessary, and chapters were written to provide readers with the required background information on the tools and utilities. From this, an entire chapter on using the tools for performance and behavior analysis evolved.\ As we neared completion of the work, and began building the entire manuscript, we ran into a bit of a problem—the size. The book had grown to over 1,500 pages. This, we discovered, presented some problems in the publishing and production of the book. After some discussion with the publisher, it was decided we should break the work up into two volumes.\ Solaris™ Internals. This represents an update to the first edition, including a significant amount of new material. All major kernel subsystems are included: the virtual memory (VM) system, processes and threads, the kernel dispatcher and scheduling classes, file systems and the virtual file system (VFS) framework, and core kernel facilities. New Solaris facilities for resource management are covered as well, along with a new chapter on networking. New features in Solaris 8 and Solaris 9 are called out as appropriate throughout the text. Examples of Solaris utilities and tools for performance and analysis work, described in the companion volume, are used throughout the text.\ Solaris™ Performance and Tools. This book contains chapters on the tools and utilities bundled with Solaris 10: dtrace(1), mdb(1), kstat(1), etc. There are also extensive chapters on using the tools to analyze the performance andbehavior of a Solaris system.\ The two texts are designed as companion volumes, and can be used in conjunction with access to the Solaris source code on\ http://www.opensolaris.org\ Readers interested in specific releases before Solaris 8 should continue to use the first edition as a reference.\ Intended AudienceWe believe that these books will serve as a useful reference for a variety of technical staff members working with the Solaris Operating System.Application developers can find information in these books about how Solaris OS implements functions behind the application programming interfaces. This information helps developers understand performance, scalability, and implementation specifics of each interface when they develop Solaris applications. The system overview section and sections on scheduling, interprocess communication, and file system behavior should be the most useful sections.Device driver and kernel module developers of drivers, STREAMS modules, loadable system calls, etc., can find herein the general architecture and implementation theory of the Solaris OS. The Solaris kernel framework and facilities portions of the books (especially the locking and synchronization primitives chapters) are particularly relevant.Systems administrators, systems analysts, database administrators, and Enterprise Resource Planning (ERP) managers responsible for performance tuning and capacity planning can learn about the behavioral characteristics of the major Solaris subsystems. The file system caching and memory management chapters provide a great deal of information about how Solaris behaves in real-world environments. The algorithms behind Solaris tunable parameters are covered in depth throughout the books.Technical support staff responsible for the diagnosis, debugging, and support of Solaris will find a wealth of information about implementation details of Solaris. Major data structures and data flow diagrams are provided in each chapter to aid debugging and navigation of Solaris systems.System users who just want to know more about how the Solaris kernel works will find high-level overviews at the start of each chapter.\ Beyond the technical user community, those in academia studying operating systems will find that this text will work well as a reference. Solaris OS is a robust, feature-rich, volume production operating system, well suited to a variety of workloads, ranging from uniprocessor desktops to very large multiprocessor systems with large memory and input/output (I/O) configurations. The robustness and scalability of Solaris OS for commercial data processing, Web services, network applications, and scientific workloads is without peer in the industry. Much can be learned from studying such an operating system.\ OpenSolaris\ In June 2005, Sun Microsystems introduced OpenSolaris, a fully functional Solaris operating system release built from open source. As part of the OpenSolaris initiative, the Solaris kernel source was made generally available through an open license offering. This has some obvious benefits to this text. We can now include Solaris source directly in the text where appropriate, as well as refer to full source listings made available through the OpenSolaris initiative.\ With OpenSolaris, a worldwide community of developers now has access to Solaris source code, and developers can contribute to whatever component of the operating system they find interesting. Source code accessibility allows us to structure the books such that we can cross-reference specific source files, right down to line numbers in the source tree.\ OpenSolaris represents a significant milestone for technologists worldwide; a world-class, mature, robust, and feature-rich operating system is now easily accessible to anyone wishing to use Solaris, explore it, and contribute to its development.\ Visit the Open Solaris Website to learn more about OpenSolaris:\ http://www.opensolaris.org\ The OpenSolaris source code is available at:\ http://cvs.opensolaris.org/source\ Source code references used throughout this text are relative to that starting location.\ How the Books Are Organized\ We organized the Solaris™ Internals volumes into several logical parts, each part grouping several chapters containing related information. Our goal was to provide a building block approach to the material by which later sections could build on information provided in earlier chapters. However, for readers familiar with particular aspects of operating systems design and implementation, the individual parts and chapters can stand on their own in terms of the subject matter they cover.\ Volume 1: Solaris™ Internals\ Part One: Introduction to Solaris Internals\ Chapter 1 — Introduction\ Part Two: The Process Model\ Chapter 2 — The Solaris Process Model\ Chapter 3 — Scheduling Classes and the Dispatcher\ Chapter 4 — Interprocess Communication\ Chapter 5 — Process Rights Management\ Part Three: Resource Management\ Chapter 6 — Zones\ Chapter 7 — Projects, Tasks, and Resource Controls\ Part Four: Memory\ Chapter 8 — Introduction to Solaris Memory\ Chapter 9 — Virtual Memory\ Chapter 10 — Physical Memory\ Chapter 11 — Kernel Memory\ Chapter 12 — Hardware Address Translation\ Chapter 13 — Working with Multiple Page Sizes in Solaris\ Part Five: File Systems\ Chapter 14 — File System Framework\ Chapter 15 — The UFS File System\ Part Six: Platform Specifics\ Chapter 16 — Support for NUMA and CMT Hardware\ Chapter 17 — Locking and Synchronization\ Part Seven: Networking\ Chapter 18 — The Solaris Network Stack\ Part Eight: Kernel Services\ Chapter 19 — Clocks and Timers\ Chapter 20 — Task Queues\ Chapter 21 — kmdb Implementation\ Volume 2: Solaris™ Performance and Tools\ Part One: Observability Methods\ Chapter 1 — Introduction to Observability Tools\ Chapter 2 — CPUs\ Chapter 3 — Processes\ Chapter 4 — Disk Behavior and Analysis\ Chapter 5 — File Systems\ Chapter 6 — Memory\ Chapter 7 — Networks\ Chapter 8 — Performance Counters\ Chapter 9 — Kernel Monitoring\ Part Two: Observability Infrastructure\ Chapter 10 — Dynamic Tracing\ Chapter 11 — Kernel Statistics\ Part Three: Debugging\ Chapter 12 — The Modular Debugger\ Chapter 13 — An MDB Tutorial\ Chapter 14 — Debugging Kernels\ Updates and Related Material\ To complement these books, we created a Web site at which we will place updated material, tools we refer to, and links to related material on the topics covered. We will regularly update the Web site (http://www.solarisinternals.com) with information about this text and future work on Solaris™ Internals. The Web site will be enhanced to provide a forum for Frequently Asked Questions (FAQs) related to the text, as well as general questions about Solaris internals, performance, and behavior. If bugs are discovered in the text, we will post errata on the Web site as well.\ A Note from the Authors\ Once again, a large investment in time and energy proved enormously rewarding for the authors. The support from Sun’s Solaris kernel development group, the Solaris user community, and readers of the first edition has been extremely gratifying. We believe we have been able to achieve more with the second edition in terms of providing Solaris users with a valuable reference text. We certainly extended our knowledge in writing it, and we look forward to hearing from readers.

Ch. 1Introduction3Ch. 2The solaris process model43Ch. 3Scheduling classes and the dispatcher157Ch. 4Interprocess communication273Ch. 5Process rights management323Ch. 6Zones367Ch. 7Projects, tasks, and resource controls415Ch. 8Introduction to solaris memory447Ch. 9Virtual memory455Ch. 10Physical memory503Ch. 11Kernel memory527Ch. 12Hardware address translation581Ch. 13Working with multiple page sizes in solaris639Ch. 14File system framework657Ch. 15The UFS file system737Ch. 16Support for NUMA and CMT hardware795Ch. 17Locking and synchronization815Ch. 18The solaris network stack855Ch. 19Clocks and timers901Ch. 20Task queues927Ch. 21kmdb Implementation943App. AKernel virtual address maps965App. BAdding a system call to solaris971App. CA sample procfs utility975