Skip to content

bug: missing benchmarks in gym list / gym search output #1899

Description

@marta-sd

Describe the bug

Benchmarks are discovered based on a config naming convention, that not all of them follow:

config_paths = glob("**/config.yaml", root_dir=BENCHMARKS_DIR, recursive=True)

Additionally the current code (with fixed globbing) won't work for finance_sec_search because it references undefined ${tavily_api_key} - resolution raises InterpolationKeyError and breaks the code. As of now only the user that knows about the benchmark and the key it requires can list it.

We need to unify the config naming (see #1317) and/or make the config search more robust. Additionally we need to make all benchmarks visible without the need for the user to configure fields they require.

Steps/Code to reproduce bug

$ gym search "tau2"
No benchmarks match 'tau2'.

Expected behavior

All benchmarks defined in BENCHMARKS_DIR appear in gym list benchmarks output and are searchable through gym search

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions