Valkey Benchmark Wrapper

Description

This wrapper facilitates the automated execution of the Valkey benchmark. Valkey is an open-source, in-memory data store forked from Redis, providing high-performance key-value storage with sub-millisecond latency. This benchmark measures throughput (requests per second) across various operations to evaluate server performance.

The wrapper provides:

• Automated Valkey service management and benchmark execution.
• Customizable connection counts, request volumes, and thread configurations.
• Multi-iteration testing with best-result selection.
• Result collection, processing, and verification.
• CSV and JSON output formats.
• System configuration metadata capture.
• Integration with test_tools framework.
• Optional Performance Co-Pilot (PCP) integration.

Command-Line Options

Valkey Options:
  --connections <value>: Number of parallel connections to use during the benchmark.
      Defaults to 50 connections.
  --threads <value>: Comma-separated list of thread counts to test with (e.g., "2,4,8").
      If not specified, automatically generates intervals from 1 to number of CPUs
      (up to 8 intervals). Each thread count is tested separately.
  --total_requests <value>: Total number of requests to execute during the benchmark.
      Defaults to 400,000 requests.

General test_tools options:
  --home_parent <value>: Parent home directory. If not set, defaults to current working directory.
  --host_config <value>: Host configuration name, defaults to current hostname.
  --iterations <value>: Number of times to run the test, defaults to 1.
  --run_user: User that is actually running the test on the test system. Defaults to current user.
  --sys_type: Type of system working with (aws, azure, hostname). Defaults to hostname.
  --sysname: Name of the system running, used in determining config files. Defaults to hostname.
  --tuned_setting: Used in naming the results directory. For RHEL, defaults to current active tuned profile.
      For non-RHEL systems, defaults to 'none'.
  --use_pcp: Enable Performance Co-Pilot monitoring during test execution.
  --tools_git <value>: Git repo to retrieve the required tools from.
      Default: https://github.com/redhat-performance/test_tools-wrappers
  --usage: Display this usage message.

What the Script Does

The valkey.sh script performs the following workflow:

1. Environment Setup:

   • Clones the test_tools-wrappers repository if not present (default: ~/test_tools).
   • Sources error codes and general setup utilities.
   • Gathers system hardware information.

2. Package Installation:

   • Installs required dependencies via package_tool based on valkey.json configuration.
   • Dependencies vary by OS variant (RHEL, Ubuntu, SLES, Amazon Linux).

3. Thread Configuration:

   • If --threads not specified, automatically determines thread counts:
     • For systems with < 8 CPUs: tests with intervals up to CPU count
     • For systems with ≥ 8 CPUs: tests with up to 8 intervals
   • Generates evenly distributed thread counts for comprehensive testing.

4. Test Execution:

   • For each iteration (--iterations count):
     • For each thread count:
       • Starts Valkey service via systemctl.
       • Waits for service to become active.
       • Runs valkey-benchmark with specified parameters:
         • Host: 127.0.0.1
         • Port: 6379
         • Requests: --total_requests (default 400,000)
         • Connections: --connections (default 50)
         • Threads: current thread count being tested
         • Output: CSV format
       • Captures start and end timestamps.
       • Stops Valkey service after test completion.
       • Optionally records PCP performance data during execution.

5. Data Collection:

   • Captures raw output from each iteration/thread combination in CSV files.
   • Extracts performance metrics for each test type:
     • PING_INLINE: Simple PING command performance
     • PING_MBULK: PING command with bulk protocol
     • SET: Write performance for SET operations
     • GET: Read performance for GET operations
     • INCR: Increment operation performance
     • LPUSH: List push operation performance
     • RPUSH: Right-side list push performance
     • LPOP: List pop operation performance
     • RPOP: Right-side list pop performance
     • SADD: Set add operation performance
     • HSET: Hash set operation performance
     • SPOP: Set pop operation performance
     • ZADD: Sorted set add performance
     • ZPOPMIN: Sorted set pop minimum performance
     • LPUSH (variadic): Variadic list push performance
     • LRANGE_100: Range query with 100 elements
     • LRANGE_300: Range query with 300 elements
     • LRANGE_500: Range query with 500 elements
     • LRANGE_600: Range query with 600 elements
     • MSET: Multiple SET operations performance

6. Result Processing:

   • For each test type, selects the highest RPS (requests per second) across all thread counts.
   • Generates CSV files with configuration and performance data.
   • Creates JSON output for verification.
   • Validates results against Pydantic schema (results_schema.py).

7. Result Aggregation (Multi-iteration runs):

   • When --iterations > 1:
     • Collects results from all iterations.
     • For each test type, computes average RPS:
       • If ≥ 5 iterations: excludes highest and lowest values, averages the rest.
       • If < 5 iterations: averages all values.
     • Generates combined report in valkey_results.csv.

8. Verification:

   • Validates results against Pydantic schema ensuring:
     • All test names match expected values.
     • All performance values are positive, finite numbers.
     • Timestamps are valid datetime objects.
   • Uses csv_to_json and verify_results from test_tools.

9. Output:

   • Creates timestamped results directory in ${HOME}/export_results/valkey_<YYYY.MM.DD-HH.MM.SS>
   • Saves all raw output files, processed CSV/JSON, and system metadata.
   • Optionally saves PCP performance data.
   • Archives results to configured storage location.

Dependencies

Location of underlying workload: Valkey (open-source Redis fork)

General packages required: Valkey server and valkey-benchmark tool (specific packages defined in valkey.json)

To run:

git clone https://github.com/redhat-performance/valkey-wrapper
cd valkey-wrapper/valkey
./valkey.sh

The script will automatically detect your CPU configuration and run benchmarks across multiple thread counts.

The Valkey Benchmark

Valkey is an open-source, in-memory data store that provides high-performance key-value storage. The benchmark tests various data structure operations to measure throughput and performance characteristics.

Operations Tested

1. PING_INLINE: Simple PING command using inline protocol.

2. PING_MBULK: PING command using bulk/binary protocol.

3. SET: Write performance for string SET operations.

4. GET: Read performance for string GET operations.

5. INCR: Atomic increment operation performance.

6. LPUSH: List data structure - push to left side.

7. RPUSH: List data structure - push to right side.

8. LPOP: List data structure - pop from left side.

9. RPOP: List data structure - pop from right side.

10. SADD: Set data structure - add member.

11. HSET: Hash data structure - set field.

12. SPOP: Set data structure - pop random member.

13. ZADD: Sorted set - add member with score.

14. ZPOPMIN: Sorted set - pop member with minimum score.

15. LPUSH (variadic): List push with multiple values.

16. LRANGE_100: List range query retrieving 100 elements.

17. LRANGE_300: List range query retrieving 300 elements.

18. LRANGE_500: List range query retrieving 500 elements.

19. LRANGE_600: List range query retrieving 600 elements.

20. MSET: Multiple SET operations in single command.

Performance Metrics

Each operation reports:

1. RPS (Requests Per Second): Throughput measurement showing how many operations can be completed per second.

   • Higher values indicate better performance.
   • Measured separately for each thread count configuration.

2. Best Thread Performance: The script selects the highest RPS achieved across all tested thread counts for each operation.

3. Timestamps: Start and end time for each test run, enabling duration calculation and temporal analysis.

Output Files

The results directory contains:

• valkey_iter_*.csv: CSV files with performance metrics from each iteration
• valkey_verify.json: JSON file with validated results data
• valkey_iter_*_*_threads.csv: Raw benchmark output for each iteration/thread combination
• valkey_results.csv: Aggregated results across all iterations (when --iterations > 1)
• test_results_report: Detailed test execution report
• meta_data*.yml: System metadata (CPU info, memory, kernel version)
• PCP data (if --use_pcp option used): Performance Co-Pilot monitoring data

Examples

Basic run with defaults

./valkey.sh

This runs with:

• 400,000 total requests
• 50 concurrent connections
• Auto-detected thread counts (1 to number of CPUs, up to 8 intervals)
• 1 iteration
• Best result selected for each operation across all thread counts

Run with specific thread counts

./valkey.sh --threads 2,4,8,16

Tests with exactly 2, 4, 8, and 16 threads, selecting best performance for each operation.

Run multiple iterations for consistency

./valkey.sh --iterations 5

Runs the complete test suite 5 times and averages the results (excluding highest and lowest values).

Run with higher request volume

./valkey.sh --total_requests 1000000

Executes 1 million requests for more stable long-running performance measurements.

Run with more connections

./valkey.sh --connections 100

Tests with 100 parallel connections instead of the default 50.

Run with PCP monitoring

./valkey.sh --use_pcp

Collects Performance Co-Pilot data during the run for detailed performance analysis.

Combination example

./valkey.sh --iterations 3 --threads 4,8,16 --total_requests 500000 --connections 75 --use_pcp

Runs 3 iterations with specific thread counts, 500k requests, 75 connections, and PCP monitoring.

How Thread Selection Works

Automatic Mode (Default)

When --threads is not specified:

1. Detects CPU count using nproc.
2. Determines interval count:
   • Systems with < 8 CPUs: interval = CPU count
   • Systems with ≥ 8 CPUs: interval = 8
3. Generates evenly distributed thread counts from 1 to CPU count.
4. Tests each thread count separately.
5. Selects best RPS for each operation across all thread counts.

Manual Mode

When --threads is specified:

• Tests only the specified thread counts (comma-separated list).
• Useful for targeted testing or comparing specific configurations.
• Still selects best result across specified thread counts.

How Result Selection Works

For each operation:

1. The operation is tested with each thread count.
2. Each thread count produces an RPS value.
3. The script selects the highest RPS across all thread counts.
4. This best-case performance is recorded in the final results.

This approach ensures:

• Optimal thread count for each operation is automatically found.
• Results represent peak achievable performance.
• Different operations may achieve best performance at different thread counts.

How Result Averaging Works

When running multiple iterations (--iterations > 1):

1. Each iteration produces a complete set of results for all operations.
2. Raw results are saved in separate valkey_iter_*.csv files.
3. The wrapper collects RPS values from each iteration.
4. Final results calculation:
   • If iterations ≥ 5: Excludes highest and lowest values, averages the remaining samples.
   • If iterations < 5: Averages all values.
5. Combined results saved in valkey_results.csv.

This approach:

• Reduces impact of transient system effects.
• Provides more reliable performance measurements.
• For ≥5 iterations, removes outliers automatically.
• Helps identify result variance across runs.

Return Codes

The script uses standardized error codes from test_tools error_codes:

• 0 (E_SUCCESS): Success
• 101: Git clone failure
• E_GENERAL: General execution errors (service failures, benchmark failures, validation failures)
• E_USAGE: Invalid usage/arguments

Exit codes indicate specific failure points for automated testing workflows.

Notes

Supported Platforms

• Linux: x86_64 and aarch64 architectures
• OS Support: RHEL, Ubuntu, SLES, Amazon Linux
• Tested Distributions: RHEL 8/9/10, Ubuntu 20.04+, SLES 15+, Amazon Linux 2/2023

Performance Considerations

• Valkey is memory-intensive; ensure sufficient RAM is available.
• Different operations have different performance characteristics:
  • Fastest: Simple operations (PING, SET, GET, INCR)
  • Moderate: List and Set operations (LPUSH, SADD, etc.)
  • Slower: Range queries (LRANGE_*) due to data retrieval overhead
• Thread scaling varies by operation:
  • Some operations scale linearly with threads.
  • Others may plateau or show diminishing returns beyond certain thread counts.
• Run multiple iterations (--iterations 3 or higher) for reliable results.
• Ensure system is idle during testing for best consistency.

Thread Count Tuning

The wrapper's default thread count generation is optimized for:

• Modern multi-core processors (8+ cores)
• Finding optimal thread count per operation automatically
• Comprehensive coverage without excessive test time

For different requirements:

• Targeted testing: Use --threads with specific values
• Quick validation: Use --threads 1,$(nproc) to test single-thread and maximum
• Detailed scaling analysis: Use --threads with many intermediate values

Connection Count Considerations

The default 50 connections provides a balance between:

• Realistic concurrent load simulation
• Avoiding connection overhead domination
• Stable, reproducible results

Adjust based on your use case:

• Higher connections (100-200): Simulate high-concurrency scenarios
• Lower connections (10-25): Focus on per-connection efficiency
• Match your production workload characteristics when possible

Performance Tips

• Run multiple test iterations (--iterations 3+) to verify consistency
• Ensure system is idle (no other workloads) for best results
• Disable CPU frequency scaling (use performance governor) for reproducible results
• Consider the active tuned profile on RHEL systems
• For production benchmarking, allow system to warm up with a test run first
• PCP monitoring (--use_pcp) adds minimal overhead but provides detailed metrics
• Ensure Valkey service is properly configured before running benchmarks

Troubleshooting

• If Valkey service fails to start, check systemctl status and logs
• If benchmark fails, verify valkey-benchmark tool is installed and in PATH
• If results seem inconsistent, run more iterations and check system load during testing
• Use --use_pcp to collect detailed performance counters for analysis
• Check valkey_iter_* files for detailed error messages
• Verify sufficient memory is available (Valkey is in-memory)
• For connection errors, check Valkey configuration (maxclients setting)

Comparison with Redis

• Valkey: Open-source fork of Redis with community-driven development
• Redis: Original project, now with different licensing
• Compatibility: Valkey maintains protocol compatibility with Redis
• Performance: Generally equivalent for most operations
• Use Valkey for open-source deployments without licensing concerns

References

• Valkey Official Site: https://valkey.io/
• Valkey GitHub: https://github.com/valkey-io/valkey
• Redis Benchmark Documentation: https://redis.io/docs/management/optimization/benchmarks/
• test_tools Framework: https://github.com/redhat-performance/test_tools-wrappers