Binding tasks and data buffers is hwloc's second main goal after discovering and exposing the hardware topology. hwloc defines APIs to bind threads and processes to cores and processing units (see CPU binding), and to bind memory buffers to NUMA nodes (see Memory binding). Some examples are available under doc/examples/ in the source tree.
Sections below provide high-level insights on how these APIs work.
Binding Policies and Portability
hwloc binding APIs are portable to multiple operating systems. However operating systems sometimes define slightly different policies, which means hwloc's behavior might slightly differ.
On the CPU binding side, OSes have different constraints of which sets of PUs can be used for binding (only full cores, random sets of PUs, etc.). Moreover the HWLOC_CPUBIND_STRICT may be given to clarify what to do in some corner cases. It is recommended to read CPU binding for details.
On the memory binding side, things are more complicated. First, there are multiple API for binding existing memory buffers, allocating new ones, etc. Second, multiple policies exist (first-touch, bind, interleave, etc.) but some of them are not implemented by all operating systems. Third, some of these policies have slightly different meanings. For instance, hwloc's bind (HWLOC_MEMBIND_BIND) uses Linux' MPOL_PREFERRED_MANY (or MPOL_PREFERRED) by default, but it switches to MPOL_BIND when strict binding is requested (HWLOC_MEMBIND_STRICT). Reading Memory binding is strongly recommended.
Joint CPU and Memory Binding (or not)
Some operating systems do not systematically provide separate functions for CPU and memory binding. This means that CPU binding functions may have have effects on the memory binding policy. Likewise, changing the memory binding policy may change the CPU binding of the current thread. This is often not a problem for applications, so by default hwloc will make use of these functions when they provide better binding support.
If the application does not want the CPU binding to change when changing the memory policy, it needs to use the HWLOC_MEMBIND_NOCPUBIND flag to prevent hwloc from using OS functions which would change the CPU binding. Additionally, HWLOC_CPUBIND_NOMEMBIND can be passed to CPU binding function to prevent hwloc from using OS functions would change the memory binding policy. Of course, using these flags will reduce hwloc's overall support for binding, so their use is discouraged.
One can avoid using these flags but still closely control both memory and CPU binding by allocating memory, touching each page in the allocated memory, and then changing the CPU binding. The already-really-allocated memory will then be "locked" to physical memory and will not be migrated. Thus, even if the memory binding policy gets changed by the CPU binding order, the already-allocated memory will not change with it. When binding and allocating further memory, the CPU binding should be performed again in case the memory binding altered the previously-selected CPU binding.
Current Memory Binding Policy
Not all operating systems support the notion of a "current" memory binding policy for the current process, but such operating systems often still provide a way to allocate data on a given node set. Conversely, some operating systems support the notion of a "current" memory binding policy and do not permit allocating data on a specific node set without changing the current policy and allocate the data. To provide the most powerful coverage of these facilities, hwloc provides:
An application can thus use the two first sets of functions if it wants to manage separately the global process binding policy and directed allocation, or use the third set of functions if it does not care about the process memory binding policy. Again, reading Memory binding is strongly recommended.
