Common Profiler Library Documentation

HyperLogLog (HLL)

A probabilistic data structure that estimates the cardinality (number of unique elements) in large datasets with minimal memory overhead.

Key Features:

• Memory Efficient: Uses logarithmic space relative to dataset size

• Configurable Precision: Supports 4-16 precision bits

• Fast Updates: Constant-time element insertion

• Merge Support: Combine multiple HLL structures for distributed counting

Use Cases:

• Approximate working set size estimation

• Unique cache line counting

• Memory footprint analysis with low overhead

Files:

• include/hll.h

• src/hll.c

MurmurHash3

A fast, non-cryptographic hash function optimized for hash table lookups and distributed hashing.

Key Features:

• High-quality hash distribution

• Platform-independent output

• Optimized for x86/x64 architectures

Files:

• include/MurmurHash3.h

• src/MurmurHash3.c

Dependencies:

• Used internally by HyperLogLog for element hashing

Working Set Tree Search (ws_tsearch)

An exact working set tracking data structure based on balanced binary trees. This structure maintains precise counts of unique memory accesses and provides statistics for memory footprint analysis.

Key Features:

• Exact Counting: Tracks every unique access with perfect accuracy

• Windowed Statistics: Reset and track working sets over time windows

• Efficient Lookup: O(log n) insertion and search using GNU tsearch

Tracked Metrics:

• Total number of references

• Distinct unique addresses

• Single-access addresses (temporal locality indicator)

Files:

• include/ws_tsearch.h

• src/ws_tsearch.c

Memory Trace (Protobuf)

A Google Protocol Buffers-based serialization format for memory access traces. Provides compact binary encoding with structured schema for trace analysis.

Key Features:

• Compact Format: Binary protobuf serialization reduces file size

• Structured Schema: Well-defined message format for interoperability

• Optional Dependency: Gracefully degrades if protobuf is unavailable

• Fast I/O: Efficient buffered writes

Trace Fields:

• timestamp_us – Microsecond-resolution timestamp

• thread_id – Thread identifier

• address – Memory address accessed

• is_write – Read/write operation type

• cache_hit – Hit/miss status (if available)

Files:

• include/memory_trace.h

• src/memory_trace.cpp

• proto/memory_trace.proto

Environment Capture

A standalone library for capturing system and runtime metadata including hostname, OS information, architecture, and environment variables.

Key Features:

• Static Capture: Captures environment at initialization time

• Minimal Dependencies: Uses only standard C library

• Python Wrapper: Integration with BaseMetadata.py for unified metadata collection

• Comprehensive Coverage: Captures all environment variables and system info

Captured Metadata:

• Hostname

• Operating system name and version

• System architecture (x86_64, ARM, etc.)

• Current working directory

• All environment variables (PATH, HOME, CUDA_HOME, etc.)

Files:

• include/environment_capture.h

• src/environment_capture.c

• Python wrapper: tools/environment_capture.py

CMake Integration

Add the common library as a subdirectory in your CMakeLists.txt:

# Add common library
add_subdirectory(../common common)

# Link against the library
target_link_libraries(your_profiler profiler_common)

Header Inclusion

Include the necessary headers in your C/C++ source:

#include "hll.h"                 // HyperLogLog cardinality estimation
#include "MurmurHash3.h"         // Fast hashing
#include "ws_tsearch.h"          // Exact working set tracking
#include "memory_trace.h"        // Protobuf trace output (optional)
#include "environment_capture.h" // System metadata capture

Example: Working Set Tracking

#include "ws_tsearch.h"
#include "hll.h"

int main() {
    // Exact working set tracking
    ws_ctx_t* ws = ws_create();
    ws_record(ws, 0x400000);  // Record cache line access
    ws_record(ws, 0x400040);

    ws_stats_t stats;
    ws_get_stats(ws, &stats);
    printf("Distinct accesses: %lu\n", stats.distinct);
    ws_destroy(ws);

    // Approximate working set tracking with HLL
    HLL hll;
    hll_init(&hll, 8);  // 8 precision bits

    uintptr_t addr = 0x400000;
    hll_add(&hll, &addr, sizeof(addr));

    double estimated_unique = hll_count(&hll);
    printf("Estimated unique lines: %.0f\n", estimated_unique);

    hll_destroy(&hll);
    return 0;
}

Example: Memory Trace Output

#include "memory_trace.h"

int main() {
    // Create trace writer
    memory_trace_writer_t* writer = memory_trace_create_writer();
    if (!writer) {
        fprintf(stderr, "Protobuf not available\n");
        return 1;
    }

    // Record memory events
    memory_trace_add_event(writer,
                           1000000,      // timestamp (us)
                           123,          // thread_id
                           0x400000,     // address
                           MEM_READ,     // read operation
                           CACHE_HIT);   // cache hit

    // Write to file
    memory_trace_write_to_file(writer, "trace.pb");
    memory_trace_destroy_writer(writer);

    return 0;
}

Example: Environment Metadata Capture

#include "environment_capture.h"

int main() {
    system_environment_t* env = environment_capture_create();

    printf("Hostname: %s\n", env->hostname);
    printf("OS: %s\n", env->os_name);
    printf("Architecture: %s\n", env->arch);
    printf("Working Directory: %s\n", env->cwd);

    // Get specific environment variable
    const char* cuda_home = environment_capture_get_var(env, "CUDA_HOME");
    if (cuda_home) {
        printf("CUDA_HOME: %s\n", cuda_home);
    }

    environment_capture_destroy(env);
    return 0;
}

Core Dependencies

• Standard C Library (required)

• GNU C Library for tsearch/tfind/twalk/tdestroy (required on Linux)

Optional Dependencies

• Google Protocol Buffers (libprotobuf-dev, protobuf-compiler)

  Required for memory trace functionality. If not available, the library will compile without protobuf support.

Installing Protocol Buffers

Ubuntu/Debian:

sudo apt-get install libprotobuf-dev protobuf-compiler

CentOS/RHEL:

sudo yum install protobuf-devel protobuf-compiler

macOS:

brew install protobuf