1   Why Cpusets?

The essential purpose of cpusets is to provide CPU and memory containers or "soft partitions" within which to run sets of related tasks.

On an SMP (multiple CPU) system without some means of CPU placement, any task can run on any CPU. On a NUMA (multiple memory node) system, any memory page can be allocated on any node. This can cause both poor cache locality and poor memory access times, substantially reducing performance and run-time repeatability. By restraining all other jobs from using any of the CPUs or memory nodes assigned to critical jobs, interference with critical jobs can be minimized.

For example, some multi-threaded high performance computing (HPC) jobs consist of a number of threads that communicate via message passing interfaces (MPI) and other such jobs rely on multi-platform shared-memory parallel programming (OpenMP) that can tightly couple parallel computation threads using special language directives. It is common that threads in such jobs need to be executing at the same time to make optimum progress. In such cases, if a single thread loses a CPU, all threads stop making forward progress and spin at a barrier. Cpusets can eliminate the need for a gang scheduler, provide isolation of one such job from other tasks on a system, and facilitate providing equal resources to each thread in such a job. This results in both optimum and repeatable performance.

This document focuses on the 'C' API provided by the user level libcpuset library. This library depends on the following Linux 2.6 kernel facilities:

• sched_setaffinity (for CPU binding),
• mbind and set_mempolicy (for memory node binding), and
• kernel cpusets support.

The sched_setaffinity, mbind and set_mempolicy calls enable specifying the CPU and memory placement for individual tasks. On smaller or limited use systems, these calls may be sufficient.

The kernel cpuset facility provides additional support for system wide management of CPU and memory resources, by related sets of tasks. It provides a hierarchical structure to the resources, with file system like name-space and permissions, and support for guaranteed exclusive use of resources.

The Linux kernel provides the following support for cpusets:

• Each task has a link to a cpuset structure that specifies the CPUs and memory nodes available for its use.
• A hook in the sched_setaffinity and mbind system calls ensures that any requested CPU or memory node is available in that tasks cpuset.
• Tasks sharing the same placement constraints reference the same cpuset.
• These kernel cpusets are arranged in a hierarchical virtual file system, reflecting the possible nesting of "soft partitions".
• The kernel task scheduler is constrained to only schedule a task on the CPUs in that task's cpuset.
• The kernel memory allocator is constrained to only allocate physical memory to a task from the memory nodes in that tasks cpuset.
• The kernel memory allocator provides an economical per-cpuset metric of the aggregate memory pressure (frequency of requests for a free memory page not easily satisfied by an available free page) of the tasks in a cpuset (see the per-cpuset 'memory_pressure' file.)
• The kernel memory allocator provides the option to request that memory pages used for file I/O (the kernel page cache) and associated kernel data structures for file inodes and directories be evenly spread across all the memory nodes in a cpuset, rather than preferentially allocated on whatever memory node the task that first accessed the page was first running (see the per-cpuset 'memory_spread_page' and 'memory_spread_slab' files).
• The memory migration facility in the kernel can be controlled using per-cpuset files, so that when the memory nodes allowed to a task by cpusets changes, any pages it had on no longer allowed nodes are migrated to nodes now allowed.

A cpuset constrains the jobs (set of related tasks) running in it to a subset of the systems memory and CPUs. They enable administrators and system service software to:

• Create and delete named cpusets.
• Decide which CPUs and memory nodes are available to a cpuset.
• Attach a task to a particular cpuset.
• Identify all tasks sharing the same cpuset.
• Exclude any other cpuset from overlapping a given cpuset, giving the tasks running in that cpuset exclusive use of those CPUs and memory nodes.
• Perform bulk operations on all tasks associated with a cpuset, such as varying the resources available to that cpuset, or hibernating those tasks in temporary favor of some other job.
• Perform sub-partitioning with hierarchical permissions and resource management.

Cpusets are exposed by the kernel to user space by mounting the cpuset virtual file system (VFS) at /dev/cpuset, rather than by additional system calls. Such a VFS is a natural way to represent nested resource allocations and the associated hierarchical permission model.

Within a single cpuset, other facilities such as dplace, first-touch memory placement, pthreads, sched_setaffinity and mbind can be used to manage processor and memory placement to a more fine-grained level.

There is a single set of kernel mechanisms that supports all these facilities and provides a consistent processor and memory model regardless of what mix of utilities and API's you use to manage it. This provides a consistent execution model for all users.

2   Linux Cpuset Kernel Support

Several developers in the Open Source community provided the Linux 2.6 kernel support for CPU and memory placement, including the following:

• Robert Love - Tech9, Novell (USA) - Scheduler attributes such as CPU affinity
• Simon Derr - Bull (France) - CPU placement, core architecture and file system interface to cpusets
• Andi Kleen - SUSE (Germany) - NUMA memory placement, mbind and mempolicy
• Paul Jackson - SGI (USA) - kernel bitmask improvements, kernel cpuset integration, libbitmask, libcpuset

At least a couple of command line utilities have been developed that use these affinity calls to allow placing a process on a specific CPU. Robert Love has a package schedutils with a command taskset. The numactl command in Andi Kleen's work has options to run a specified command on specified CPUs and memory nodes.

Andi Kleen led a session at the 2003 Kernel Summit in Ottawa in NUMA memory management, and has been developing a NUMA library and kernel support for memory placement. Minutes from that session are available at: http://lwn.net/Articles/40626/.

The cpuset kernel facility and file system for Linux 2.6 kernels is based on the work of Simon Derr, with integration and refinements by Paul Jackson. In addition this document of the libcpuset user library, the kernel cpuset facility is documented in the kernel source file Documentation/cpusets.txt, and Simon Derr has provided a document of cpusets at
http://www.bullopensource.org/cpuset/.

The user level bitmask library supports convenient manipulation of multi-word bitmasks useful for CPUs and memory nodes. This bitmask library is required by and designed to work with the cpuset library. The design notes of libbitmask are available in a separate document, Bitmask_Library.html or Bitmask_Library.pdf.

Unlike sched_setaffinity() and mbind(), which are implemented as additional kernel system calls, the primary kernel interface for accessing the cpuset facility is The Cpuset File System, usually mounted at /dev/cpuset. The cpuset library libcpuset provides convenient access to these facilities from 'C' programs.

Cpusets extend the usefulness of the Linux 2.6 kernel mechanisms sched_setaffinity() for CPU placement, and mbind() and set_mempolicy() for memory placement. On smaller or dedicated use systems, these other mechanisms are often sufficient. The libcpuset library provides a convenient API to these other mechanisms that has the added advantage of being robustly adapting to memory migration.

On larger NUMA systems, running more than one, performance critical, job, it is necessary to be able to manage jobs in their entirety. This includes providing a job with exclusive CPU and memory that no other job can use and being able to list all tasks currently in a cpuset.

You can use both these other placement mechanisms and cpusets together, using the Advanced Cpuset Library Functions to manage overall job placement, and using the other mechanisms, perhaps via the Basic Cpuset Library Functions within each given job to manage the details of thread and memory page placement.

#!/bin/sh
rmdir /dev/cpuset/$1

First, migrate pages on node 7 to node 8
Second, migrate pages on node 6 to node 7
Third, migrate pages on node 5 to node 6
Fourth, migrate pages on node 4 to node 5

3   Using Cpusets at the Shell Prompt

There are multiple ways to use cpusets, including:

• They can be queried and changed from a shell prompt, using such command line utilities as echo, cat, mkdir and ls.
• They can be queried and changed via the libcpuset 'C' programming API. The primary emphasis of this document is on the 'C' API.

This section describes the use of cpusets using shell commands.

One convenient way to learn how cpusets work is to experiment with them at the shell prompt, before doing extensive 'C' coding.

Note that there is one significant difference between these two interfaces.

Modifying the CPUs in a cpuset the shell prompt requires an additional step, due to intentional limitations in the kernel support for cpusets. The cpuset_reattach routine can be used to perform this step when using libcpuset. The extra step consists of writing the pid of each task attached to that cpuset back into the cpusets tasks file:

In order to minimize the impact of cpusets on critical kernel code, such as the scheduler, and due to the fact that the kernel does not support one task updating the memory placement of another task directly, the impact on a task of changing its cpuset CPU or memory node placement, or of changing to which cpuset a task is attached, is subtle.

If a cpuset has its memory nodes modified, then for each task attached to that cpuset, the next time that the kernel attempts to allocate a page of memory for that task, the kernel will notice the change in the tasks cpuset, and update its per-task memory placement to remain within the new cpusets memory placement. If the task was using mempolicy MPOL_BIND, and the nodes to which it was bound overlap with its new cpuset, then the task will continue to use whatever subset of MPOL_BIND nodes are still allowed in the new cpuset. If the task was using MPOL_BIND and now none of its MPOL_BIND nodes are allowed in the new cpuset, then the task will be essentially treated as if it was MPOL_BIND bound to the new cpuset (even though its NUMA placement, as queried by get_mempolicy(), doesn't change). If a task is moved from one cpuset to another, then the kernel will adjust the tasks memory placement, as above, the next time that the kernel attempts to allocate a page of memory for that task.

If a cpuset has its CPUs modified, then each task using that cpuset does _not_ change its behavior automatically. In order to minimize the impact on the critical scheduling code in the kernel, tasks will continue to use their prior CPU placement until they are rebound to their cpuset, by rewriting their pid to the 'tasks' file of their cpuset. If a task had been bound to some subset of its cpuset using the sched_setaffinity() call, the effect of this is lost on the rebinding. The rebound tasks cpus_allowed is set to include all cpus in the tasks new cpuset. If a task is moved from one cpuset to another, its CPU placement is updated in the same way as if the tasks pid is rewritten to the 'tasks' file of its current cpuset.

In summary, the memory placement of a task whose cpuset is changed is automatically updated by the kernel, on the next allocation of a page for that task, but the processor placement is not updated, until that tasks pid is rewritten to the 'tasks' file of its cpuset. The delay in rebinding a tasks memory placement is necessary because the kernel does not support one task changing another tasks memory placement. The added user level step in rebinding a tasks CPU placement is necessary to avoid impacting the scheduler code in the kernel with a check for changes in a tasks processor placement.

To create a new cpuset and attach the current command shell to it, the steps are:

1. mkdir /dev/cpuset (if not already done)
2. mount -t cpuset cpuset /dev/cpuset (if not already done)
3. Create the new cpuset using mkdir(1).
4. Assign CPUs and memory nodes to the new cpuset.
5. Attach the shell to the new cpuset.

For example, the following sequence of commands will setup a cpuset named "Charlie", containing just CPUs 2 and 3, and memory node 1, and then attach the current shell to that cpuset:

mkdir /dev/cpuset
mount -t cpuset cpuset /dev/cpuset
cd /dev/cpuset
mkdir Charlie
cd Charlie
/bin/echo 2-3 > cpus
/bin/echo 1 > mems
/bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
cat /proc/self/cpuset

Migrating a job (the set of tasks attached to a cpuset) to different CPUs and memory nodes in the system, including moving the memory pages currently allocated to that job, can be done as follows. Lets say you want to move the job in cpuset alpha (CPUs 4-7 and memory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory nodes 8-9).

1. First create the new cpuset beta.
2. Then allow CPUs 16-19 and memory nodes 8-9 in beta.
3. Then enable memory_migration in beta.
4. Then move each task from alpha to beta.

The following sequence of commands accomplishes this:

cd /dev/cpuset
mkdir beta
cd beta
/bin/echo 16-19 > cpus
/bin/echo 8-9 > mems
/bin/echo 1 > memory_migrate
while read i; do /bin/echo $i; done < ../alpha/tasks > tasks

The above should move any tasks in alpha to beta, and any memory held by these tasks on memory nodes 2-3 to memory nodes 8-9, respectively.

Notice that the last step of the above sequence did not do:

cp ../alpha/tasks tasks     # Doesn't work (ignores all but first task)

The while loop, rather than the seemingly easier use of the cp(1) command, is necessary because only one task PID at a time may be written to the tasks file.

The same affect (writing one pid at a time) as the while loop can be accomplished more efficiently, in fewer keystrokes and in syntax that works in any shell, but alas more obscurely, by using the sed -u [unbuffered] option:

sed -un p < ../alpha/tasks > tasks

4   Cpuset Programming Model

The libcpuset programming model for cpusets provides a hierarchical cpuset name space that integrates smoothly with Linux 2.6 kernel support for simple processor and memory placement. As systems become larger, with more complex memory, processor and bus architectures, the hierarchical cpuset model for managing processor and memory resources will become increasingly important.

The cpuset name space remains visible to all tasks on a system. Once created, a cpuset remains in existence until it is deleted or until the system is rebooted, even if no tasks are currently running in that cpuset.

The key properties of a cpuset are its pathname, the list of which CPUs and memory nodes it contains, and whether the cpuset has exclusive rights to these resources.

Every task (process) in the system is attached to (running inside) a cpuset. Tasks inherit their parents cpuset attachment when forked. This binding of task to a cpuset can subsequently be changed, either by the task itself, or externally from another task, given sufficient authority.

Tasks have their CPU and memory placement constrained to whatever their containing cpuset allows. A cpuset may have exclusive rights to its CPUs and memory, which provides certain guarantees that other cpusets will not overlap.

At system boot, a top level root cpuset is created, which includes all CPUs and memory nodes on the system. The usual mount point of the cpuset file system, and hence the usual file system path to this root cpuset, is /dev/cpuset.

Changing the cpuset binding of a task does not by default move the memory the tasks might have currently allocated, even if that memory is on memory nodes no longer allowed in the tasks cpuset. On kernels that support such memory migration, use the [optional] cpuset_migrate function to move allocated memory as well.

To create a cpuset from 'C' code, one obtains a handle to a new struct cpuset, sets the desired attributes via that handle, and issues a cpuset_create() to actually create the desired cpuset and bind it to the specified name. One can also list by name what cpusets exist, query and modify their properties, move tasks between cpusets, list what tasks are currently attached to a cpuset, and delete cpusets.

The cpuset_alloc() call applies a hidden undefined mark to each attribute of the allocated struct cpuset. Calls to the various cpuset_set*() routines mark the attribute being set as defined. Calls to cpuset_create() and cpuset_modify() only set the attributes of the cpuset marked defined. This is primarily noticable when creating a cpuset. Code in the kernel sets some attributes of new cpusets, such as memory_spread_page, memory_spread_slab and notify_on_release, by default to the value inherited from their parent. Unless the application using libcpuset explicitly overrides the setting of these attributes in the struct cpuset, between the calls to cpuset_alloc() and cpuset_create(), the kernel default settings will prevail. These hidden marks have no noticeable affect when modifying an existing cpuset using the sequence of calls cpuset_alloc(), cpuset_query(), and cpuset_modify(), because the cpuset_query() call sets all attributes and marks them defined, while reading the attributes from the cpuset.

The names of cpusets in this 'C' library are either relative to the root cpuset mount point (typically /dev/cpuset) if the name starts with a slash '/' character or else relative to the current tasks cpuset.

Cpusets can be renamed using the rename(2) system call. The per-cpuset files within each cpuset directory cannot be renamed, and the rename(2) system call cannot be used to modify the cpuset hierarchy. You cannot change the parent cpuset of a cpuset using rename(2).

Despite its name, the pid parameter to various libcpuset routines is actually a thread id, and each thread in a threaded group can be attached to a different cpuset. The value returned from a call to gettid(2) can be passed in the argument pid.

5   CPUs and Memory Nodes

As of this writing, the Linux kernel NUMA support presumes that there are some memory nodes and some CPUs, and that for each CPU, there is exactly one preferred or local memory node. Frequently, multiple (two or four perhaps) CPUs will be local to the same memory node. Some memory nodes have no local CPUs - these are called headless nodes.

However, this is not the only possible architecture, and architectures are constantly changing, usually toward the more complex. Driven by nonstop increases in logic density for a half century now, the bus, cache, CPU, memory and storage hierarchy of large systems continues to evolve.

This cpuset interface should remain stable over a long period of time, and be usable over a variety of system architectures. So this interface avoids presuming that there is exactly one memory node local to each CPU. Rather it just presumes that there are some CPUs and memory nodes, that some zero or more memory nodes are local to each CPU, and that some zero or more CPUs are local to each memory node.

This interface provides the functions cpuset_localmems to identify, for a CPU, which memory node(s) are local to that CPU, and cpuset_localcpus to identify for a memory node, which CPU(s) are local. Applications using this interface can explicitly specify just one of CPU or memory placement, and then use these functions to determine the corresponding memory or CPU placement.

In some situations, applications will want to explicitly place both CPUs and memory nodes, not necessarily according to the default relation between a CPU and its local memory node. Such situations include the following.

• You can control specific CPU and memory node placement when precise placement control is required, such as when optimizing performance for a specific important application.
• A higher level system service such as a batch scheduler can control specific CPU and memory node placement.
• On systems having some memory on headless nodes (memory nodes with no associated local CPU), you can explicitly use a particular headless node with a particular CPU.
• On systems supporting Hyper-Threadthreading, one could affectively disable Hyper-Threading, by using just one out of the two or more execution engines (CPUs) available on a processor die.

6   Extensible API

In order to provide for the convenient and robust extensibility of this cpuset API over time, the following function enables dynamically obtaining pointers for optional functions by name, at run-time:

void *cpuset_function(const char * function_name) - returns function pointer, or NULL if function name unrecognized

For maximum portability, you should not reference any optional cpuset function by name by explicit name.

However, if you are willing to presume that an optional function will always be available on the target systems of interest, you might decide to explicitly reference it by name, in order to improve the clarity and simplicity of the software in question.

Also to support robust extensibility, flags and integer option values have names dynamically resolved at run-time, not via preprocessor macros.

All functions use only the primitive types of int, char *, pid_t (for process id of a task), size_t (for buffer sizes), pointers to opaque structures, functions whose signatures use these types, and pointers to such functions. They use no structure members, special types or magic define constants.

Some functions in Advanced Cpuset Library Functions are marked [optional]. They are not available in all implementations of libcpuset. Additional [optional] cpuset_* functions may also be added in the future. Functions that are not marked [optional] are available on all implementations of libcpuset.so, and can be called directly without using cpuset_function(). However, any of them can also be called indirectly via cpuset_function().

To safely invoke an optional function, such as for example cpuset_migrate(), use the following call sequence:

/* fp points to function of the type of cpuset_migrate() */
int (*fp)(struct cpuset *fromcp, struct cpuset *tocp, pid_t pid);
fp = cpuset_function("cpuset_migrate");
if (fp) {
        fp( ... );
} else {
        puts ("cpuset migration not supported");
}

If you invoke an [optional] function directly, then your resulting program will not be able to link with any version of libcpuset.so that does not define that particular function.

7   Cpuset Text Format

Cpuset settings may be exported to, and imported from, text files using a text format representation of cpusets.

The permissions of files holding these text representations have no special significance to the implementation of cpusets. Rather, the permissions of the special cpuset files in the cpuset file system, normally mounted at /dev/cpuset, control reading and writing of and attaching to cpusets.

The text representation of cpusets is not essential to the use of cpusets. One can directly manipulate the special files in the cpuset file system. This text representation provides an alternative that may be convenient for some uses.

The cpuset text format supports one directive per line. Comments begin with the '#' character and extend to the end of line.

After stripping comments, the first white space separated token on each remaining line selects from the following possible directives:

cpus

Specify which CPUs are in this cpuset. The second token on the line must be a comma-separated list of CPU numbers and ranges of numbers, optionally modified by a stride operator (see below).

mems

Specify which memory nodes are in this cpuset. The second token on the line must be a comma-separated list of memory node numbers and ranges of numbers, optionally modified by a stride operator (see below).

cpu_exclusive

The cpu_exclusive flag is set.

mem_exclusive

The mem_exclusive flag is set.

notify_on_release

The notify_on_release flag is set

Additional unnecessary tokens on a line are quietly ignored. Lines containing only comments and white space are ignored.

The token "cpu" is allowed for "cpus", and "mem" for "mems". Matching is case insensitive.

The stride operator for "cpus" and "mems" values is used to designate every N-th CPU or memory node in a range of numbers. It is written as a colon ":" followed by the number N, with no spaces on either side of the colon. For example, the range 0-31:2 designates the 16 even numbers 0, 2, 4, ... 30.

The libcpuset routines cpuset_import and cpuset_export to handle converting the internal 'struct cpuset' representation of cpusets to (export) and from (import) this text representation.

8   Basic Cpuset Library Functions

The basic cpuset API provides functions usable from 'C' for processor and memory placement within a cpuset.

These functions enable an application to place various threads of its execution on specific CPUs within its current cpuset, and perform related functions such as asking how large the current cpuset is, and on which CPU within the current cpuset a thread is currently executing.

These functions do not provide the full power of the advanced cpuset API, but they are easier to use, and provide some common needs of multi-threaded applications.

Unlike the rest of this document, the functions described in this section use cpuset relative numbering. In a cpuset of N CPUs, the relative cpu numbers range from zero to N - 1.

Unlike the underlying system calls sched_setaffinity, mbind or set_mempolicy, these basic cpuset API functions are robust in the presence of cpuset migration. If you pin a thread in your job to one of the CPUs in your jobs cpuset, it will stay properly pinned even if your jobs cpuset is migrated later to another set of CPUs and memory nodes, or even if the migration is occurring at the same time as your calls.

memory placement is done automatically, preferring the node local to the requested CPU. Threads may only be placed on a single CPU. This avoids the need to allocate and free the bitmasks required to specify a set of several CPUs. These functions do not support creating or removing cpusets, only the placement of threads within an existing cpuset. This avoids the need to explicitly allocate and free cpuset structures. Operations only apply to the current thread, avoiding the need to pass the process id of the thread to be affected.

If more powerful capabilities are needed, use the Advanced Cpuset Library Functions. These basic functions do not provide any essential new capability. They are implemented using the advanced functions, and are fully interoperable with them.

On error, these routines return -1 and set errno. If invoked on an operating system kernel that does not support cpusets, errno is set to ENOSYS. If invoked on a system that supports cpusets, but when the cpuset file system is not currently mounted at /dev/cpuset, then errno is set to ENODEV.

The following inclusion and linkage provides access to the cpuset API from 'C' code:

#include <cpuset.h>
/* link with -lcpuset */

The following functions are supported in the basic cpuset 'C' API:

• cpuset_pin - Pin the current thread to a CPU, preferring local memory
• cpuset_size - Return the number of CPUs are in the current tasks cpuset
• cpuset_where - On which CPU in current tasks cpuset did the task most recently execute
• cpuset_unpin - Remove affect of cpuset_pin, let task have run of its entire cpuset

9   Using Cpusets with Hyper-Threads

Threading in a software application splits instructions into multiple streams so that multiple processors can act on them.

Hyper-Threading (HT) Technology, developed by Intel Corporation, provides thread-level parallelism on each processor, resulting in more efficient use of processor resources, higher processing throughput, and improved performance. One physical CPU can appear as two logical CPUs by having additional registers to overlap two instruction streams or a single processor can have dual-cores executing instructions in parallel.

In addition to their traditional use to control the placement of jobs on the CPUs and memory nodes of a system, cpusets also provides a convenient mechanism to control the use of Hyper-Threading.

Some jobs achieve better performance using both of the Hyper-Thread sides, A and B, of a processor core, and some run better using just one of the sides, allowing the other side to idle.

Since each logical (Hyper-Threaded) processor in a core has a distinct CPU number, one can easily specify a cpuset that contains both sides, or contains just one side from each of the processor cores in the cpuset.

Cpusets can be configured to include any combination of the logical CPUs in a system.

For example, the following cpuset configuration file called cpuset.cfg includes the A sides of an HT enabled system, along with all the memory, on the first 32 nodes (assuming 2 cores per node). The colon ':' prefixes the stride. The stride of '2' in this example means use every other logical CPU.

cpus 0-127:2    # even numbered CPUs 0, 2, 4, ... 126
mems 0-31       # memory nodes 0, 1, 2, ... 31

To create a cpuset called foo and run a command called bar in that cpuset, defined by the cpuset configuration file cpuset.cfg shown above, use the following commands:

cpuset -c /foo < cpuset.cfg
cpuset -i /foo -I bar

To specify both sides of the first 64 cores, use the following entry in your cpuset configuration file:

cpus 0-127

To specify just the B sides, use the following entry in your cpuset configuration file:

cpus 1-127:2

The examples above assume that CPUs are uniformly numbered, with the even numbers for the A side and odd numbers for the B side. This is usually the case, but not guaranteed. One could still place a job on a system that was not uniformly numbered, but currently it would involve a longer argument list to the cpus option, explicitly listing the desired CPUs.

Typically, the logical numbering of CPUs puts the even numbered CPUs on the A sides, and the odd numbered CPUs on the B side. The stride operator (":2", above) makes it easy to specify that only every other side will be used. If the CPU number range starts with an even number, this will be the A sides, and if the range starts with an odd number, this will be the B sides.

A ps(1) or top(1) invocation will show a handful of threads on unused CPUs, but these are kernel threads assigned to every CPU in support of user applications running on those CPUs, to handle tasks such as asynchronous file system writes and task migration between CPUs. If no application is actually using a CPU, then the kernel threads on that CPU will be almost always idle and will consume very little memory.

10   Advanced Cpuset Library Functions

The advanced cpuset API provides functions usable from 'C' for managing cpusets on a system-wide basis.

These functions primarily deal with the three entities (1) struct cpuset *, (2) system cpusets and (3) tasks.

The struct cpuset * provides a transient in-memory structure used to build up a description of an existing or desired cpuset. These structs can be allocated, freed, queried and modified.

Actual kernel cpusets are created under /dev/cpuset, which is the usual mount point of the kernels virtual cpuset filesystem. These cpusets are visible to all tasks (with sufficient authority) in the system, and persist until the system is rebooted or until the cpuset is explicitly deleted. These cpusets can be created, deleted, queried, modified, listed and examined.

Every task (also known as a process) is bound to exactly one cpuset at a time. You can list which tasks are bound to a given cpuset, and to which cpuset a given task is bound. You can change to which cpuset a task is bound.

The primary attributes of a cpuset are its lists of CPUs and memory nodes. The scheduling affinity for each task, whether set by default or explicitly by the sched_setaffinity() system call, is constrained to those CPUs that are available in that tasks cpuset. The NUMA memory placement for each task, whether set by default or explicitly by the mbind() system call, is constrained to those memory nodes that are available in that tasks cpuset. This provides the essential purpose of cpusets - to constrain the CPU and memory usage of tasks to specified subsets of the system.

The other essential attribute of a cpuset is its pathname beneath /dev/cpuset. All tasks bound to the same cpuset pathname can be managed as a unit, and this hierarchical name space describes the nested resource management and hierarchical permission space supported by cpusets. Also, this hierarchy is used to enforce strict exclusion, using the following rules:

• A cpuset may only be marked strictly exclusive for CPU or memory if its parent is also.
• A cpuset may not make any CPUs or memory nodes available that are not also available in its parent.
• If a cpuset is exclusive for CPU or memory, then it may not overlap CPUs or memory with any of its siblings.

The combination of these rules enables checking for strict exclusion just by making various checks on the parent, siblings and existing child cpusets of the cpuset being changed, without having to check all cpusets in the system.

On error, some of these routines return -1 or NULL and set errno. If one of the routines below that requires cpuset kernel support is invoked on an operating system kernel that does not support cpusets, then that routine returns failure and errno is set to ENOSYS. If invoked on a system that supports cpusets, but when the cpuset file system is not currently mounted at /dev/cpuset, then it returns failure and errno is set to ENODEV.

The following inclusion and linkage provides access to the cpuset API from 'C' code:

#include <bitmask.h>
#include <cpuset.h>
/* link with -lcpuset */

The following functions are supported in the advanced cpuset 'C' API:

• Cpuset library (libcpuset) version
  • cpuset_version - [optional] Version (simple integer) of the library
• Allocate and free struct cpuset *
  • cpuset_alloc - Return handle to newly allocated struct cpuset *
  • cpuset_free - Discard no longer needed struct cpuset *
• Lengths of CPUs and memory nodes bitmasks - needed to allocate bitmasks
  • cpuset_cpus_nbits - Number of bits needed for a CPU bitmask on current system
  • cpuset_mems_nbits - Number of bits needed for a memory bitmask on current system
• Set various attributes of a struct cpuset *
  • cpuset_setcpus - Specify CPUs in cpuset
  • cpuset_setmems - Specify memory nodes in cpuset
  • cpuset_set_iopt - Specify an integer value option of cpuset
  • cpuset_set_sopt - [optional] Specify a string value option of cpuset
• Query various attributes of a struct cpuset *
  • cpuset_getcpus - Query CPUs in cpuset
  • cpuset_getmems - Query memory nodes in cpuset
  • cpuset_cpus_weight - Number of CPUs in a cpuset
  • cpuset_mems_weight - Number of memory nodes in a cpuset
  • cpuset_get_iopt - Query an integer value option of cpuset
  • cpuset_get_sopt - [optional] Query a string value option of cpuset
• Local CPUs and memory nodes
  • cpuset_localcpus - Query the CPUs local to specified memory nodes
  • cpuset_localmems - Query the memory nodes local to specified CPUs
  • cpuset_cpumemdist - [optional] Hardware distance from CPU to memory node
  • cpuset_cpu2node - Return system number of memory node closest to specified CPU.
  • cpuset_addr2node - Return sytem number of memory node holding page at specified address.
• Create, delete, query, modify, list and examine cpusets.
  • cpuset_create - Create named cpuset as specified by struct cpuset *
  • cpuset_delete - Delete the specified cpuset (if empty)
  • cpuset_query - Set struct cpuset to settings of specified cpuset
  • cpuset_modify - Modify a cpuset's settings to those specified in a struct cpuset
  • cpuset_getcpusetpath - Get path of a tasks (0 for current) cpuset.
  • cpuset_cpusetofpid - Set struct cpuset to settings of cpuset of specified task
  • cpuset_mountpoint - Return path at which cpuset filesystem is mounted
  • cpuset_collides_exclusive - [optional] True if would collide exclusive
  • cpuset_nuke - [optional] Remove cpuset anyway possible
• List tasks currently attached to a cpuset
  • cpuset_init_pidlist - Initialize a list of tasks attached to a cpuset
  • cpuset_pidlist_length - Return the number of tasks in such a list
  • cpuset_get_pidlist - Return a specific task from such a list
  • cpuset_freepidlist - Deallocate such a list
• Attach tasks to cpusets.
  • cpuset_move - Move task (0 for current) to a cpuset
  • cpuset_move_all - Move all tasks in a list of pids to a cpuset
  • cpuset_move_cpuset_tasks - [optional] Move all tasks in a cpuset to another cpuset
  • cpuset_migrate - [optional] Move a task and its memory to a cpuset
  • cpuset_migrate_all - [optional] Move all tasks with memory in a list of pids to a cpuset
  • cpuset_reattach - Rebind cpus_allowed of each task in a cpuset after changing its cpus
• Determine memory pressure
  • cpuset_open_memory_pressure - [optional] Open handle to read memory_pressure
  • cpuset_read_memory_pressure - [optional] Read cpuset current memory_pressure
  • cpuset_close_memory_pressure - [optional] Close handle to read memory pressure
• Map between cpuset relative and system-wide CPU and memory node numbers
  • cpuset_c_rel_to_sys_cpu - Map cpuset relative CPU number to system wide number
  • cpuset_c_sys_to_rel_cpu - Map system wide CPU number to cpuset relative number
  • cpuset_c_rel_to_sys_mem - Map cpuset relative memory node number to system wide number
  • cpuset_c_sys_to_rel_mem - Map system wide memory node number to cpuset relative number
  • cpuset_p_rel_to_sys_cpu - Map task cpuset relative CPU number to system wide number
  • cpuset_p_sys_to_rel_cpu - Map system wide CPU number to task cpuset relative number
  • cpuset_p_rel_to_sys_mem - Map task cpuset relative memory node number to system wide number
  • cpuset_p_sys_to_rel_mem - Map system wide memory node number to task cpuset relative number
• Placement operations - for detecting cpuset migration
  • cpuset_get_placement - [optional] Return current placement of task pid
  • cpuset_equal_placement - [optional] True if two placements equal
  • cpuset_free_placement - [optional] Free placement
• Traverse a cpuset hierarchy.
  • cpuset_fts_open - [optional] Open cpuset hierarchy
  • cpuset_fts_read - [optional] Next entry in hierarchy
  • cpuset_fts_reverse - [optional] Reverse order of cpusets
  • cpuset_fts_rewind - [optional] Rewind to first cpuset in list
  • cpuset_fts_get_path - [optional] Get entry's cpuset path
  • cpuset_fts_get_stat - [optional] Get entry's stat(2) pointer
  • cpuset_fts_get_cpuset - [optional] Get entry's cpuset pointer
  • cpuset_fts_get_errno - [optional] Get entry's errno
  • cpuset_fts_get_info - [optional] Get operation causing error
  • cpuset_fts_close - [optional] Close cpuset hierarchy
• Bind to a CPU or memory node within the current cpuset
  • cpuset_cpubind - Bind to a single CPU within a cpuset (uses sched_setaffinity(2))
  • cpuset_latestcpu - Most recent CPU on which a task has executed
  • cpuset_membind - Bind to a single memory node within a cpuset (uses set_mempolicy(2))
• Export cpuset settings to a regular file, and import them from a regular file
  • cpuset_export - Export cpuset settings to a text file
  • cpuset_import - Import cpuset settings from a text file
• Support calls to [optional] cpuset_* API routines
  • cpuset_function - Return pointer to a libcpuset.so function, or NULL

A typical calling sequence would use the above functions in the following order to create a new cpuset named "xyz" and attach itself to it.

struct cpuset *cp = cpuset_alloc();
various cpuset_set*(cp, ...) calls
cpuset_create(cp, "xyz");
cpuset_free(cp);    
cpuset_move(0, "xyz");

Some functions above are marked [optional]. For more information, see the Extensible API section, above, for an explanation of this marking and how to invoke such functions in a portable manner.

11   System Error Numbers

The Linux kernel implementation of cpusets sets errno to specify the reason for a failed system call affecting cpusets. These errno values are available when a cpuset library call fails. Most of these values can also be displayed by shell commands used to directly manipulate files below /dev/cpuset.

The possible errno settings and their meaning when set on a failed cpuset call are as listed below.

ENOSYS

Invoked on an operating system kernel that does not support cpusets.

ENODEV

Invoked on a system that supports cpusets, but when the cpuset file system is not currently mounted at /dev/cpuset.

ENOMEM

Insufficient memory is available.

EBUSY

Attempted cpuset_delete() on a cpuset with attached tasks.

EBUSY

Attempted cpuset_delete() on a cpuset with child cpusets.

ENOENT

Attempted cpuset_create() in a parent cpuset that doesn't exist.

EEXIST

Attempted cpuset_create() for a cpuset that already exists.

EEXIST

Attempted to rename(2) a cpuset to a name that already exists.

ENOTDIR

Attempted to rename(2) a non-existent cpuset.

E2BIG

Attempted a write(2) system call on a special cpuset file with a length larger than some kernel determined upper limit on the length of such writes.

ESRCH

Attempted to cpuset_move() a non-existent task.

EACCES

Attempted to cpuset_move() a task which one lacks permission to move.

EACCES

Attempted to write(2) a memory_pressure file.

ENOSPC

Attempted to cpuset_move() a task to an empty cpuset.

EINVAL

The relcpu argument to cpuset_pin() is out of range (not between "zero" and "cpuset_size() - 1").

EINVAL

Attempted to change a cpuset in a way that would violate a cpu_exclusive or mem_exclusive attribute of that cpuset or any of its siblings.

EINVAL

Attempted to write an empty cpus or mems bitmask to the kernel. The kernel creates new cpusets (via mkdir) with empty cpus and mems, and the user level cpuset and bitmask code works with empty masks. But the kernel will not allow an empty bitmask (no bits set) to be written to the special cpus or mems files of a cpuset.

EIO

Attempted to write(2) a string to a cpuset tasks file that does not begin with an ASCII decimal integer.

EIO

Attempted to rename(2) a cpuset outside of its current directory.

ENOSPC

Attempted to write(2) a list to a cpus file that did not include any online cpus.

ENOSPC

Attempted to write(2) a list to a mems file that did not include any online memory nodes.

EACCES

Attempted to add a cpu or mem to a cpuset that is not already in its parent.

EACCES

Attempted to set cpu_exclusive or mem_exclusive on a cpuset whose parent lacks the same setting.

ENODEV

The cpuset was removed by another task at the same time as a write(2) was attempted on one of the special files in the cpuset directory.

EBUSY

Attempted to remove a cpu or mem from a cpuset that is also in a child of that cpuset.

EFAULT

Attempted to read or write a cpuset file using a buffer that was outside your accessible address space.

ENAMETOOLONG

Attempted to read a /proc/<pid>/cpuset file for a cpuset path that was longer than the kernel page size.

ENAMETOOLONG

Attempted to create a cpuset whose base directory name was longer than 255 characters.

ENAMETOOLONG

Attempted to create a cpuset whose full pathname including the "/dev/cpuset" is longer than 4095 characters.

ENXIO

Attempted to create a cpu_exclusive cpuset whose cpus covered just part of one or more physical processor packages, such as including just one of the two Cores on a package. For Linux kernel version 2.6.16 on i386 and x86_64, this operation is rejected with this error to avoid a fatal kernel bug. Otherwise, this is a normal and supported operation.

EINVAL

Specified a cpus or mems list to the kernel which included a range with the second number smaller than the first number.

EINVAL

Specified a cpus or mems list to the kernel which included an invalid character in the string.

ERANGE

Specified a cpus or mems list to the kernel which included a number too large for the kernel to set in its bitmasks.

ETIME

Time limit for cpuset_nuke operation reached without successful completion of operation.

ENOTEMPTY

Tasks remain after multiple attempts by cpuset_move_cpuset_tasks to move them to a different cpuset.

EPERM

Lacked permission to kill (send a signal to) a task.

EPERM

Lacked permission to read a cpuset or its files.

EPERM

Attempted to unlink a per-cpuset file. Such files can not be unlinked. They can only be removed by removing (rmdir) the directory representing the cpuset that contains these files.

12   Change History

Here is the history of changes to this document and associated software.

• 2006-04-26 Paul Jackson <pj@sgi.com>

  • First published.

• 2006-11-14 Paul Jackson <pj@sgi.com> Version 1 (cpuset_version())

  • Added cpuset_nuke routine.
  • Added various cpuset_fts_* routines.
  • Added various cpuset_*_affinity routines.
  • Added cpuset_move_cpuset_tasks routine.
  • Converted from using ftw(3) to fts(3).
  • Various minor documentation errors fixed.
  • Added cpuset_version routine.
  • Improved error checking by cpuset_move_all.
  • Correct system call numbering for sched_setaffinity on i386 arch.
  • Correct cpuset_latestcpu result if command basename has space character.
  • Remove 256 byte limit on cpuset_export output.

• 2007-01-14 Paul Jackson <pj@sgi.com> Version 2 (cpuset_version())

  • Fix cpuset_create, cpuset_modify to not zero undefined attributes such as memory_spread_page, memory_spread_slab and notify_on_release. See further the explanation of an "undefined mark", in Cpuset Programming Model. cpuset_create now respects default kernel cpuset creation settings, and cpuset_modify now respects the existing settings of the cpuset, unless explicitly set.

• 2007-04-01 Paul Jackson <pj@sgi.com> Version 3 (cpuset_version())

  • Fix cpuset_setcpus() and cpuset_setmems() to mark the cpus and mems fields as defined, so that setting them before doing a cpuset_create() has affect.