Threadprocs – executables sharing one address space (0-copy pointers)
This project launches multiple independent programs into a single shared virtual address space, while still behaving like separate processes (independent binaries, globals, and lifetimes). When threadprocs share their address space, pointers are valid across them with no code changes for well-behaved Linux binaries. Unlike threads, each threadproc is a standalone and semi-isolated process. Unlike dlopen-based plugin systems, threadprocs run traditional executables with a `main()` function. Unlike POSIX processes, pointers remain valid across threadprocs because they share the same address space. This means that idiomatic pointer-based data structures like `std::string` or `std::unordered_map` can be passed between threadprocs and accessed directly (with the usual data race considerations). This accomplishes a programming model somewhere between pthreads and multi-process shared memory IPC. The implementation relies on directing ASLR and virtual address layout at load time and implementing a user-space analogue of `exec()`, as well as careful manipulation of threadproc file descriptors, signals, etc. It is implemented entirely in unprivileged user space code: < https://github.com/jer-irl/threadprocs/blob/main/docs/02-imp... >. There is a simple demo demonstrating “cross-threadproc” memory dereferencing at < https://github.com/jer-irl/threadprocs/tree/main?tab=readme-... >, including a high-level diagram. This is relevant to systems of multiple processes with shared memory (often ring buffers or flat tables). These designs often require serialization or copying, and tend away from idiomatic C++ or Rust data structures. Pointer-based data structures cannot be passed directly. There are significant limitations and edge cases, and it’s not clear this is a practical model, but the project explores a way to relax traditional process memory boundaries while still structuring a system as independently launched components.
- Cloud Native
- Code Generation
- Integrations
✨ AI Summary
Threadprocs allows multiple independent executables to run within a single shared virtual address space, enabling direct pointer access across them while maintaining process-like isolation. This approach offers a middle ground between threads and traditional multi-process shared memory.
Best For
Systems programmers exploring novel inter-process communication methods, Developers working with large, interconnected data structures across components, Researchers investigating memory isolation and sharing techniques
Why It Matters
Enables direct, zero-copy sharing of pointer-based data structures between independently launched executables by allowing them to share a single address space.
Key Features
- Launches independent programs into a single shared virtual address space.
- Maintains separate process-like behavior (binaries, globals, lifetimes).
- Enables 0-copy pointer sharing across 'threadprocs' on Linux.
- Supports direct access to idiomatic C++/Rust data structures (e.g., std::string).
Use Cases
- A game developer could use threadprocs to load different game modules (e.g., AI, physics, rendering) as independent executables that share the same memory space. This would allow for direct, zero-copy passing of complex data structures like scene graphs or character state between modules, simplifying inter-module communication compared to traditional IPC or serialization.
- A systems programmer building a high-performance data processing pipeline might employ threadprocs to run distinct processing stages (e.g., ingestion, transformation, analysis) as separate binaries. This architecture would enable direct manipulation of shared data buffers, such as large arrays or complex object graphs, without the overhead of copying or serialization, potentially improving throughput.
- A researcher developing a complex simulation environment could leverage threadprocs to manage different simulation components (e.g., agent behavior, environmental interactions, data logging) as independently compiled programs. This would allow for seamless sharing of simulation state and direct access to shared data structures, facilitating rapid prototyping and iteration on simulation logic.