Contemporary Software Systems - Case Study Example

Contemporary software systems are beset by problems that create challenges and opportunities for broad new OS research. The areas, where broad OS research could significantly improve the current user experience, are dependability, security, system configuration, system extension, and multi-processor programming. Singularity is a Microsoft Research project to develop techniques and tools for building dependable systems that address the challenges faced by contemporary software systems. Singularity is approaching these challenges by simultaneously pushing the state of the art in operating systems, run-time systems, programming languages, and programming tools—the foundation on which software is built.  The Singularity OS is first and foremost a research system.  Singularity strives for minimalism and design clarity, and makes extensive use of modern languages and tools. Microsoft is making prototype Operating system available free to academic and research communities so that they can use it to develop new computer architectures. The design of Singularity makes it easy to test how operating system and other application interact with each other. Singularity will improve software reliability and boost research in programming language and tools. Singularity is based on advances in programming languages and programming tools to develop and build a new system architecture and operating system (named Singularity), with the aim of producing a more robust and dependable software platform. Although dependability is difficult to measure in a research prototype, Singularity shows the practicality of new technologies and architectural decisions, which should lead to more robust and dependable systems in the future. The Singularity system consists of three key architectural features: software-isolated processes, contract-based channels, and manifest-based programs. A key aspect of Singularity is an extension model based on Software-Isolated Processes (SIPs), which encapsulate pieces of an application or a system and provide information hiding, failure isolation, and strong interfaces. SIPs are the OS processes on Singularity. All code outside the kernel executes in a SIP. SIPs differ from conventional operating system processes in a number of ways: SIPs are closed object spaces, not address spaces. Two Singularity processes cannot simultaneously access an object. Communications between processes transfers exclusive ownership of data. SIPs are closed code spaces. A process cannot dynamically load or generate code. SIPs do not rely on memory management hardware for isolation. Multiple SIPs can reside in a physical or virtual address space. Communications between SIPs is through bidirectional, strongly typed, higher-order channels. A channel specifies its communications protocol as well as the values transferred, and both aspects are verified. SIPs are inexpensive to create and communication between SIPs incurs low overhead. Low cost makes it practical to use SIPs as a fine-grain isolation and extension mechanism. SIPs are created and terminated by the operating system, so that on termination, a SIP’s resources can be efficiently reclaimed. SIPs executed independently, even to the extent of having different data layouts, run-time systems, and garbage collectors. A key focus of Singularity research is improving system dependability.  Singularity improves dependability by dramatically increasing the scope of sound verification techniques to detect sources of unexpected system behaviour.  To broaden the scope of sound verification techniques, Singularity fixes the behaviour of system components as early as possible in lifetime of their code.  To lengthen the scope of sound verification techniques, Singularity constrains system organization and preserves metadata so that verification results can be applied even to late-bound composites. Singularity incorporates three key design choices to improve system dependability.  These design choices are: an abstract instruction set as part of the system’s application binary interface (ABI), a unified extension architecture, and a first-class application abstraction.  The abstract instruction set provides the OS with a flexible layer of indirection between application code and a processor’s instruction stream. The unified extension architecture enables rich, inexpensive, and safe interaction between system components.   The application abstraction enables OS management of applications and integration of applications into the security model as security principles. Early indications are that these design choices also have a positive impact on the challenges of system security, configuration, and extension. Singularity executables represent executable code in an abstract instruction set, called MSIL.  MSIL is Microsoft’s implementation of the ECMA Common Intermediate Language. All third-party executables, including applications and device drivers, are delivered to Singularity as type-safe MSIL binaries. Singularity requires that all users MSIL be type safe, which eliminates an entire class of programmer errors due to erroneous or malicious pointer arithmetic.  Because Singularity controls the translation of MSIL into processor instructions, the OS retains the opportunity to insert trusted instruction sequences into the unprivileged, but verified, instruction stream.  The abstract instruction set also opens new opportunities to dynamically adjust the trade-offs between security and performance, and it allows rigorous analysis and instrumentation of application code. Singularity provides extension architecture for the operating system and applications.  Like previous micro-kernels. Singularity incorporates a process-based extension model.  Singularity, however, assumes more aggressive closed-process architecture for both OS and application extensions. Singularity processes are closed worlds in two regards.  First, Singularity disallows shared memory between processes; Singularity processes exchange data exclusively through messages, which are visible to only one process at a time.  Second, once execution begins within a process, no new code may be added to the process.  Singularity disallows both loading of new code modules and generation of new code into an existing process. Any OS or application extension code can be loaded only into a child process, separated by a strong isolation boundary.  Communication between host and extension across the process isolation boundary is restricted to verified message-passing channels.  Channels are strongly typed with contracts.  All cross-channel interactions and contracts are statically verified using a technique called conformance checking. Conformance checking guarantees that a contract is fully specified, that two parties communicating through a contract will not deadlock, and that neither party will receive an unexpected message. By disallowing dynamic loading of new code into a process, Singularity processes become a closed world in which analysis tools can make sound assumptions about process states, invariants, and valid state transitions.  The closed-world extension architecture opens new opportunities for static analysis and optimization.  Singularity raises the notion of an application to a first-class OS abstraction.  Applications have security identities and signed manifests declaring their constituent components.  Installation, maintenance, and removal of applications are all operations controlled by the OS.  Applications are strongly isolated.   Access to shared resources—including other applications—is mediated through the Singularity security model.  The security model uses code identity and component relationships in access control checks. The application abstraction is recursively applied to the OS itself, with the kernel and other OS components described by manifests.  Manifests form the roots of a metadata infrastructure that enables introspection across the entire system –both applications and operating system.  Through this metadata it should be possible, for example, to examine an offline Singularity system image and determine if it has the necessary components and configuration to run on a specific hardware configuration or host a specific application. A specific Singularity system as represented by an installation image then becomes a self-describing artefact, not just a collection of bits accumulated with at best an anecdotal history. Most operating systems install and uninstall applications through imperative updates to mutable configuration information held in the file system and in configuration registries. Singularity’s application abstraction is extended to support a declarative form of configuration for a whole system, which will eliminate whole classes of system miss - configuration Supporting the architectural foundation of Singularity is the Singularity kernel. The kernel provides the base abstractions of software-isolated processes, contract-based channels, and manifest-based programs. The kernel performs the crucial role of dividing systems resources among competing programs and abstracting the complexities of system hardware. To each SIP, the kernel provides a pure execution environment with threads, memory, and access to other MBPs via channels. The Singularity project enjoys the safety and productivity benefits of writing a kernel in a type-safe, garbage-collected language. Counting lines of code, over 90% of the Singularity kernel is written in Sing#. While most of the kernel is type-safe Sing#, a significant portion of the kernel code is written in the unsafe variant of the language. The most significant unsafe code is the garbage collector, which accounts for 48% of the unsafe code in Singularity. Other major sources of unsafe Sing# code include the memory management and I/O access subsystems. Singularity includes small pockets of assembly language code in the same places it would be used in a kernel written in C or C++, for example, the thread context switch, interrupt vectors, etc. Approximately 6% of the Singularity kernel is written in C++, consisting primarily of the kernel debugger and low-level system initialization code. Following is the architecture design of Singularity Operating System. Singularity Architecture In addition to the usual mechanism of message-passing channels, processes communicate with the kernel through a strongly versioned application binary interface (ABI) that invokes static methods in kernel code. This interface follows the design of the rest of the system and isolates the kernel and process object spaces. All parameters to this ABI are values, not pointers, so the kernel and process’s garbage collectors need not coordinate. The only exception is the location of the ABI methods. Our garbage collectors currently do not relocate code, but if they did, they would need to maintain the invariant that these methods remain at known addresses. The ABI maintains the system-wide state isolation invariant: a process cannot alter the state of another process using the ABI. With only two exceptions, an ABI call affects only the state of its calling process. The two exceptions alter the state of a child process before or after it executes, but not during execution. The first is a call to create a child process, which specifies the code loaded for the child before it begins execution. The second is a call to stop a child process, which reclaims its resources after all threads cease execution. State isolation ensures that a Singularity process has sole control over its state. Singularity supports a compile-time replaceable scheduler. It is based on four type of scheduler-the Rialto scheduler, a multi-resource laxity-based scheduler, a round-robin scheduler, and a minimum latency round-robin scheduler. The minimum latency round-robin scheduler is optimized for a large number of threads that communicate frequently. The scheduler maintains two lists of run able threads. The first, called the unblocked list, contains threads that have recently become unable. The second, called the pre-empted list, contains unable threads that have been pre-empted. When choosing the next thread to run, the scheduler removes threads from the unblocked list in FIFO order. When the unblocked list is empty, the scheduler removes the next thread from the pre-empted list. Whenever a scheduling timer interrupt occurs, all threads in the unblocked list are moved to the end of the pre-empted list, followed by the thread that was running when the timer fired. The first thread from the unblocked list is scheduled and the scheduling timer is reset. A Singularity system lives in a single virtual address space. Virtual memory hardware is used to protect pages, for example by mapping out the first 16K of address space to trap null pointer references. Within a Singularity system, the address space is logically partitioned into: a kernel object space, an object space for each process, and the Exchange Heap for channel data. A pervasive design decision is the memory independence invariant: cross-object space pointers only 10 point into the Exchange Heap. In particular, the kernel does not have pointers into a process’s object space, nor does one process have a pointer to another process objects. This invariant ensures that each process can be garbage collected and terminated without the cooperation of other processes. Singularity uses linked stacks to reduce the memory overhead of a thread. These stacks grow on demand by adding non-contiguous segments of 4K or more. Singularity’s compiler performs static inter procedural analysis to optimize placement of overflow tests. Each of these compiler-inserted checks is trusted code that accesses system data structures, residing in the process’s object space, to determine the amount of space remaining in the current stack segment. Before the running thread pushes a new stack frame which would potentially allow overflow of the current stack segment, the trusted code calls a kernel method, which disable interrupts and invokes the page manager to allocate a new stack segment Built on the kernel, SIPs, channels, and language model described above, Singularity supports a number of conventional operating system services. The Singularity kernel uses a manifest to create and bind device drivers. On start-up, the kernel does a plug and play configuration of the system. The kernel use information acquired from the BIOS by the boot loader and from buses, such as the PCI bus, to enumerate devices, start the appropriate device drivers, and pass these drivers objects that encapsulate access to device hardware. Each driver is written in safe code and runs in its own process. Drivers communicate with other parts of the system, including the network stack and file system, exclusively through channels. When a driver starts, the kernel provides it with four types of initialized objects that enable the driver to communicate with its device. All these objects provide a safe interface that checks each reference before directly accessing the hardware’s memory mapped locations. The only unsafe aspect of the driver-device interface is DMA. Existing DMA architectures provide no memory protection, so a misbehaving or malicious driver can program a DMA capable device to overwrite any part of memory. An interrupt from a device is serviced by the kernel, which masks the interrupt, and then signals the appropriate driver’s IoIrq. Each driver process has a thread waiting on its Irq event, which starts processing the interrupt and re-enables the interrupt line through a kernel ABI. The scheduler runs immediately after the interrupt handler and signals all events in this queue. The Singularity system makes extensive use of metadata to describe pieces of the system, explain how they fit together, and specify their behaviour. The metadata in Singularity declaratively labels each Singularity component, system, or application with its dependencies, exports, and resources. Tools in Singularity use this metadata to verify and configure application and system code, both before and during system execution. A Singularity system image contains metadata describing the device drivers. Through the metadata, Singularity maintains three invariants. First, Singularity will never install a device driver that cannot start successfully due to resources conflicts with another driver or portion of the system. Second, Singularity will never start a device driver that cannot run successfully due to either a conflicting or missing resource. Third, a device driver cannot access resources at runtime that were not declared in its metadata. "Singularity is not the next Windows," Rick Rashid, senior vice president of Microsoft Research, said in a statement. "Think of it like a concept car. It is a prototype operating system designed from the ground up to test-drive a new paradigm for how operating systems and applications interact with one another. We are making it available to the community in the hope that it will enable researchers to try out new ideas quickly." The roots of the project stem from research into what a modern operating system would look like and how it would behave. Microsoft points out that Windows, like UNIX, Linux and the Maces, all trace their origins back to Multics, an operating system that originated in the mid-1960s. In essence, the operating systems we use today are built on foundations that are more than 40 years old. References: Allen, D.H., Dhong, S.H., Hofstee, H.P., Leenstra, J., Nowka, K.J., Stasiak, D.L. and Wendel, D.F. Custom Circuit Design as a Driver of Microprocessor Performance. IBM Journal of Research and Development, 44 (6) 2000. Hawblitzel, C. and Eicken, T.v. Luna: A Flexible Java Protection System. in Proceedings of the Fifth ACM Symposium on Operating System Design and Implementation (OSDI 02), Boston, MA, 2002, 391-402. Hunt, G.C., Larus, J.R., Tarditi, D. and Wobber, T. Broad New OS Research: Challenges and Opportunities. in Proceedings of the 10th Workshop on Hot Topics in Operating Systems (HotOS X), Santa Fe, NM, 2005 Microsoft Corporation, Scalable Networking: Network Protocol Offload - Introducing TCP Chimney. Technical Report 2004. Wobber, T., Abadi, M., Birrell, A., Simon, D.R. and Yumerefendi, A., Authorizing Applications in Singularity. In Proceedings of the EuroSys 2007 Conference, pp. 355- 368, Lisbon, Portugal, March 2007. Read More