Transcript Config OS - University of New Mexico

A Framework for Adaptable Operating and
Runtime Systems
Ron Brightwell
Scalable Computing Systems
Sandia National Laboratories
Albuquerque, New Mexico, USA
FAST-OS PI Meeting
June 9-10, 2005
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,
for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Project Details
• Sandia National Laboratories
– Neil Pundit (Project Director)
– Ron Brightwell (Coordinating PI)
– Rolf Riesen, Trammell Hudson, Zaid Adubayeh
• University of New Mexico
– Barney Maccabe (PI)
– Patrick Bridges
• California Institute of Technology
– Thomas Sterling (PI)
What’s wrong with current operating
systems?
Cluster Network Hardware
• DMA between NIC and
host memory
– Physical addresses on
NIC
– Can be user- or kernelspace
– Memory descriptors on
NIC
• Benefits associated with
offloading
– Reduced overhead
– Increased bandwidth
– Reduced latency
OS Bypass Versus Splintering
Cluster Architecture
Distributing little bits
of the OS
Other Issues
• General-purpose operating systems
– Generality comes at the cost of performance for all
applications
– Assume a generic architectural model
• Difficult to expose novel features
• Lightweight operating systems
– Limited functionality
– Difficult to add new features
– Designed to be used in the context of a specific usage
model
• Operating system is an impediment to new architectures
and programming models
Factors Impacting OS Design
LWK Influences
• Lightweight OS
– Small collection of apps
• Single programming
model
– Single architecture
– Single usage model
– Small set of shared
services
– No history
• Puma/Cougar
– MPI
– Distributed memory
– Space-shared
– Parallel file system
– Batch scheduler
Programming Models
Usage Models
Current and Future System Demands
• Architecture
– Modern ultrascale machines have widely varying system-level
and node-level architectures
– Future systems will have further hardware advances (e.g.,
multi-core chips, PIMs)
• Programming model
– MPI, Thread, OpenMP, PGAS, …
• External services
– Parallel file systems, dynamic libraries, checkpoint/restart, …
• Usage model
– Single, large, long-running simulation
– Parameter studies with thousands of single-processor, shortrunning jobs
Project Goals
• Realize a new generation of scalable, efficient, reliable, easy
to use operating systems for a broad range of future
ultrascale high-end computing systems based on both
conventional and advanced hardware architectures and in
support of diverse, current and emerging parallel
programming models.
• Devise and implement a prototype system that provides a
framework for automatically configuring and building
lightweight operating and runtime system based on the
requirements presented by an application, system usage
model, system architecture, and the combined needs for
shared services.
Approach
• Define and build a collection of micro-services
– Small components with well-defined interfaces
– Implement an indivisible portion of service
semantics
– Fundamental elements of composition and re-use
• Combine micro-services specifically for an
application and a target platform
• Develop tools to facilitate the synthesis of
required micro-services
Tools for Combining Micro-Services
• Need to insure that required micro-services are
available
• Need to insure that applications are isolated from
one another within the context of a given usage
model
• Verifying that a set of constraints are met
• Further work will allow for reasoning about
additional system properties, such as
performance based on feedback from previous
runs
Building Custom Operating/Runtime Systems
Timetable
• 12 months
– Define basic framework and micro-service APIs
– Define initial micro-services for supporting a lightweight
kernel equivalent
– Identify applications and related metrics for evaluating
resulting systems
• 24 months
– Demonstrate configuration and linking tools with
multiple lightweight kernel configurations
– Define application-specific micro-services for optimizing
application performance
– Define shared-service micro-services for common
application services (e.g. TCP/IP)
Timetable (cont’d)
• 36 months
– Demonstrate instance of framework for PIM-based
system on base-level PIM architecture simulator
– Demonstrate application/kernel configurability
using application-specific and shared-service
micro-services
– Release complete software package as open
source
– Provide detailed report summarizing completed and
future work
Related Work
• Microkernels
– K42, L4, Pebble, Mach, …
– Exo-kernel
• Extensible operating systems
– Spin, Vino, sandboxing, …
– Modules
• Configurable OS/Runtime
– Scout, Think, Flux OSKit, eCos, TinyOS
– SREAMS, x-kernel, CORDS
More Info
• “Highly Configurable Operating Systems for
Ultrascale Systems,” Maccabe et al. In
Proceedings of the First International Workshop
on Operating Systems, Programming
Environments and Management Tools for HighPerformance Computing on Clusters (COSET-1),
June 2004. (http://coset.irisa.fr/)
Current Thinking
HPC Architecture Framework
• Next generation operating system must serve
conventional architectures of today and future
architectures towards nanoscale technology
– Today: commodity clusters, MPPs, vector,
multicore
– Tomorrow: Heterogeneous, Stream, PIM,
Reconfigurable
Some Basic Assumptions
• Different programming and execution
models will demand different
optimizations in OS services for best
performance
• Micro-services may require more than a
single node resource to accomplish
function
• Micro-services may perform management
function on more than one node resource
at a time
Seven-Dimensional Space of Architectures
• Degree of Coupling
– Relative bandwidth between external channel and local
memory
– Latency and latency hiding strategy support (e.g.
multithreading)
• Local state capacity
• Overhead of context instantiation, switching, and synchronization
• Namespace semantics and management towards single system
image and virtualization
• Resource allocation for memory objects (static versus dynamic)
• Intrinsic model of execution
• What to do when the application flatlines – exceptions and
booting
Programming Models
• Distributed namespace
– Shared nothing
• Shared namespace
– Shared addresses, heavyweight threads
• Highly dynamic and fine-grained
– Lightweight threads
– In-memory synchronization
• Need to keep an eye on new language
developments ( Chpl, Fortress, X10)
Additional Considerations
• OS services may consume parallel resources
• Different applications may vary dramatically in
terms of balance of resource usage and OS
service request functionality
• Architecture support may vary dramatically for
efficient OS service functionality
• “local” means all references and resources for an
action are immediately available within a single
cycle
Initial Focus
• Programming models
– Distributed namespace (MPI)
– Shared namespace (UPC)
– Dynamic, fine-grained (ParalleX)
• Architectures
– Distributed memory (Red Storm)
– Shared memory (Columbia)
– PIM (MIND simulator)
Currently Brainstorming on Micro-services
•
•
•
•
•
•
Definition of a micro-service
Programming model for micro-services
What does a micro-service need?
What does a micro-service provide?
How does a micro-service behave?
How to support multiple implementations of the same microservice?
• What’s the smallest unit of activity that can be distributed?
• Some micro-services may not be useful until combined with
other micro-services
• Can dynamic compilation serve as a possible
implementation strategy for composing services
Composability Framework
• Model for composability of micro-services
• Need more brainstorming to design the
framework
• Leverage OpenCatamount’s implementation
details when the framework is defined
OpenCatamount
• Open source version of Catamount
• All Cray, Intel, and nCUBE proprietary code
removed
• Export control restriction lifted
• Boots on a Dell box
• Working on configuration and build environment
• Likely will be released under Mozilla-style license
• Should be up on Sandia download site real soon®
• Port to a virtualization environment?
Timeline
• Programming model for micro-services
– Q1FY06
• Initial draft of the framework design and definition
of micro-services
– Q2FY06
• Description of tools to do composition
– Q3FY06
Most Pressing Issue
• What is the name of this project? 
• Potential candidates:
– ConfigOS, ANOS, ACRONYM, ASCENT, OPUS….
Barney “Working”
FAST-OS Funding Comes Through