diff options
| author | Paul Buetow <paul@buetow.org> | 2025-07-04 15:35:16 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-07-04 15:35:16 +0300 |
| commit | d37f32deb6cd6a575cc169adf1a1c1fba44e53d9 (patch) | |
| tree | aaf5f6abc90066892a6a23cb619969ddd4ef5574 /CLAUDE.md | |
| parent | 1249f9ec51b1355ca17f73244dcbe0acc5556516 (diff) | |
feat: add Profile-Guided Optimization (PGO) support
- Add comprehensive PGO module in internal/tools/pgo/
- Integrate PGO into dtail-tools command with full CLI support
- Add Makefile targets for PGO workflow:
- make pgo: Full PGO workflow
- make pgo-quick: Quick PGO with smaller datasets
- make pgo-generate: Generate profiles only
- make build-pgo: Build with existing profiles
- make install-pgo: Install optimized binaries
- Add convenience functions to data generator for PGO
- Document PGO workflow in CLAUDE.md
Performance improvements observed:
- DCat: 3.8-7.0% additional improvement over turbo mode
- DGrep: Up to 19% improvement for low hit rates
- DMap: Variable impact, up to 64% for min_max on large files
Benchmarks show total performance gains (pre-turbo → turbo+PGO):
- DCat: 14-21x faster
- DGrep: 9-15x faster
- DMap: 9-29% faster
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Diffstat (limited to 'CLAUDE.md')
| -rw-r--r-- | CLAUDE.md | 76 |
1 files changed, 76 insertions, 0 deletions
@@ -28,6 +28,12 @@ DTAIL_USE_ACL=yes make build # Enable proprietary features DTAIL_USE_PROPRIETARY=yes make build + +# Build PGO-optimized binaries (requires existing profiles) +make build-pgo + +# Generate PGO profiles and build optimized binaries +make pgo ``` ## Testing & Development @@ -68,6 +74,43 @@ make benchmark-baseline make benchmark-compare BASELINE=benchmarks/baselines/baseline_TIMESTAMP.txt ``` +## Profile-Guided Optimization (PGO) + +```bash +# Full PGO workflow: generate profiles and build optimized binaries +make pgo + +# Quick PGO with smaller datasets (faster) +make pgo-quick + +# PGO for specific commands only +make pgo-commands COMMANDS='dcat dgrep' + +# Generate PGO profiles only (without building) +make pgo-generate + +# Build PGO-optimized binaries using existing profiles +make build-pgo + +# Install PGO-optimized binaries to system +make install-pgo + +# Clean PGO artifacts +make pgo-clean + +# Show PGO help +make pgo-help +``` + +### PGO Notes + +- PGO provides additional performance improvements on top of turbo mode +- Typical improvements: 5-10% for DCat, up to 19% for DGrep with low hit rates +- Profiles are saved in `pgo-profiles/` directory +- Optimized binaries are built in `pgo-build/` directory +- Use `make build-pgo` to rebuild optimized binaries without regenerating profiles +- PGO profiles are workload-specific; consider custom profiles for your use case + ## Profiling ```bash @@ -159,6 +202,39 @@ make benchmark-mapreduce make benchmark-ssh ``` +## Profile-Guided Optimization (PGO) + +```bash +# Run PGO for all commands +make pgo + +# Quick PGO with smaller datasets +make pgo-quick + +# PGO for specific commands +make pgo-commands COMMANDS='dcat dgrep' + +# Clean PGO artifacts +make pgo-clean + +# Show PGO help +make pgo-help + +# Direct usage with dtail-tools +dtail-tools pgo # Optimize all commands +dtail-tools pgo dcat dgrep # Optimize specific commands +dtail-tools pgo -v -iterations 5 # Verbose with 5 iterations + +# After PGO, optimized binaries are in pgo-build/ +``` + +### PGO Notes + +- PGO uses profile data from real workloads to optimize binary performance +- The process involves: building baseline → generating profiles → building with PGO +- Typical improvements range from 5-20% depending on the workload +- Optimized binaries are placed in the `pgo-build/` directory + ## Architecture & Code Organization ### Binary Entry Points |
