Describe the bug
Benchmarks are discovered based on a config naming convention, that not all of them follow:
|
config_paths = glob("**/config.yaml", root_dir=BENCHMARKS_DIR, recursive=True) |
Additionally the current code (with fixed globbing) won't work for finance_sec_search because it references undefined ${tavily_api_key} - resolution raises InterpolationKeyError and breaks the code. As of now only the user that knows about the benchmark and the key it requires can list it.
We need to unify the config naming (see #1317) and/or make the config search more robust. Additionally we need to make all benchmarks visible without the need for the user to configure fields they require.
Steps/Code to reproduce bug
$ gym search "tau2"
No benchmarks match 'tau2'.
Expected behavior
All benchmarks defined in BENCHMARKS_DIR appear in gym list benchmarks output and are searchable through gym search
Describe the bug
Benchmarks are discovered based on a config naming convention, that not all of them follow:
Gym/nemo_gym/cli/eval.py
Line 115 in 59295cd
Additionally the current code (with fixed globbing) won't work for
finance_sec_searchbecause it references undefined${tavily_api_key}- resolution raisesInterpolationKeyErrorand breaks the code. As of now only the user that knows about the benchmark and the key it requires can list it.We need to unify the config naming (see #1317) and/or make the config search more robust. Additionally we need to make all benchmarks visible without the need for the user to configure fields they require.
Steps/Code to reproduce bug
Expected behavior
All benchmarks defined in
BENCHMARKS_DIRappear ingym list benchmarksoutput and are searchable throughgym search