-
Notifications
You must be signed in to change notification settings - Fork 53
Support one-file open_dataset #1479
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rajeeja
wants to merge
17
commits into
main
Choose a base branch
from
rajeeja/onefile_open_dataset
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+127
−18
Open
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
5605c18
Support one-file open_dataset
rajeeja bfb6e17
Update test/core/test_api.py
rajeeja 7edf0e1
Refine one-file dataset loading
rajeeja 2dd95de
Address review comments
rajeeja 93c0676
Merge branch 'main' into rajeeja/onefile_open_dataset
erogluorhan f3af73c
Merge branch 'main' into rajeeja/onefile_open_dataset
rajeeja a4eb83c
Merge branch 'main' into rajeeja/onefile_open_dataset
rajeeja 8a0d66e
Merge branch 'main' into rajeeja/onefile_open_dataset
rajeeja dfaf6f1
Merge branch 'main' into rajeeja/onefile_open_dataset
rajeeja fdfe369
Merge branch 'main' into rajeeja/onefile_open_dataset
rajeeja 7e0683f
Merge remote-tracking branch 'origin/main' into rajeeja/onefile_open_…
rajeeja bc94b31
Merge branch 'main' into rajeeja/onefile_open_dataset
rajeeja 59f6dd7
improve error message when can't parse grid
Sevans711 f263fae
Clarify one-file open_dataset docstring and directory error message
rajeeja 01bccc0
Document that partial or non-Earth grids can be opened
rajeeja 4a8aea5
Merge branch 'main' into rajeeja/onefile_open_dataset
erogluorhan 05fba3f
Reword single-argument open_dataset docstring per review
rajeeja File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -353,7 +353,7 @@ def list_grid_names( | |
|
|
||
| def open_dataset( | ||
| grid_filename_or_obj: str | os.PathLike[Any] | dict | Dataset, | ||
| filename_or_obj: str | os.PathLike[Any], | ||
| filename_or_obj: str | os.PathLike[Any] | Dataset | None = None, | ||
| chunks=None, | ||
| chunk_grid: bool = True, | ||
| use_dual: bool | None = False, | ||
|
|
@@ -364,14 +364,22 @@ def open_dataset( | |
|
|
||
| Parameters | ||
| ---------- | ||
| grid_filename_or_obj : str | os.PathLike[Any] | dict | xr.dataset | ||
| Strings and Path objects are interpreted as a path to a grid file. Xarray Datasets assume that | ||
| each member variable is in the UGRID conventions and will be used to create a ``ux.Grid``. Similarly, a dictionary | ||
| containing UGRID variables can be used to create a ``ux.Grid`` | ||
| filename_or_obj : str | os.PathLike[Any] | ||
| grid_filename_or_obj : str | os.PathLike[Any] | dict | xr.Dataset | ||
| Grid information for the ``UxDataset``. Strings and Path objects are interpreted as a path to a grid | ||
| file. Xarray Datasets assume that each member variable is in the UGRID conventions and will be used to | ||
| create a ``ux.Grid``. Similarly, a dictionary containing UGRID variables can be used to create a | ||
| ``ux.Grid``. A path to a directory containing grid files (e.g. a FESOM2 ASCII grid) is also accepted, | ||
| but only when ``filename_or_obj`` is provided; directory input is not supported in the single-argument | ||
| form below. | ||
| filename_or_obj : str | os.PathLike[Any] | xr.Dataset, optional | ||
| String or Path object as a path to a netCDF file or an OpenDAP URL that | ||
| stores the actual data set. It is the same ``filename_or_obj`` in | ||
| ``xarray.open_dataset``. | ||
| stores the actual data set, or an already-open ``xarray.Dataset``. It | ||
|
Comment on lines
+367
to
+376
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would like to clarify the docs a bit:
The third point I am saying because these changes enable you to do open_dataset(grid_file) for any grid file, regardless of whether it is "combined" with data or not. |
||
| is the same ``filename_or_obj`` in ``xarray.open_dataset``. If omitted, | ||
| ``grid_filename_or_obj`` is also used as the data source, allowing a | ||
| combined grid-and-data file or ``xarray.Dataset`` to be opened with a | ||
| single argument. In this single-argument form the file is used to build | ||
| the grid and also treated as data; all variables it contains, including | ||
| grid variables, will be included as data. | ||
| chunks : int, dict, 'auto' or None, default: None | ||
| If provided, used to load the grid into dask arrays. | ||
|
|
||
|
|
@@ -400,29 +408,76 @@ def open_dataset( | |
| uxds : uxarray.UxDataset | ||
| Dataset with linked `uxgrid` property of type `Grid`. | ||
|
|
||
| Notes | ||
| ----- | ||
| The grid need not be a complete or fully UGRID-compliant mesh, nor represent | ||
| a spherical/Earth-science domain. A partial grid can be opened; operations | ||
| requiring information absent from the file will raise when used, but | ||
| unrelated functionality remains available. | ||
|
|
||
| Examples | ||
| -------- | ||
| Open a dataset with a grid file and data file | ||
|
|
||
| >>> import uxarray as ux | ||
| >>> ux_ds = ux.open_dataset("grid_file.nc", "data_file.nc") | ||
|
|
||
| Open a dataset stored in a single combined grid-and-data file | ||
|
|
||
| >>> ux_ds = ux.open_dataset("combined_file.nc") | ||
| """ | ||
| import xarray as xr | ||
|
|
||
| if grid_kwargs is None: | ||
| grid_kwargs = {} | ||
|
|
||
| # Construct a Grid, validate parameters, and correct chunks | ||
| uxgrid, corrected_chunks = _get_grid( | ||
| grid_filename_or_obj, chunks, chunk_grid, use_dual, grid_kwargs, **kwargs | ||
| ) | ||
| if filename_or_obj is None: | ||
| if isinstance(grid_filename_or_obj, (str, os.PathLike)): | ||
|
rajeeja marked this conversation as resolved.
|
||
| if os.path.isdir(grid_filename_or_obj): | ||
| raise ValueError( | ||
| "ux.open_dataset() with a single directory argument is not supported. " | ||
| "Supply a path to a grid file instead. Directory-based grids (e.g. a " | ||
| "FESOM2 ASCII grid) are only recognized when a separate data file is " | ||
| "also provided, i.e. ux.open_dataset(grid_directory, data_file)." | ||
| ) | ||
|
|
||
| # Load the data as a Xarray Dataset | ||
| ds = _open_dataset_with_fallback(filename_or_obj, chunks=corrected_chunks, **kwargs) | ||
| ds = _open_dataset_with_fallback( | ||
|
rajeeja marked this conversation as resolved.
|
||
| grid_filename_or_obj, | ||
| chunks=match_chunks_to_ugrid(grid_filename_or_obj, chunks), | ||
| **kwargs, | ||
| ) | ||
| elif isinstance(grid_filename_or_obj, xr.Dataset): | ||
| ds = grid_filename_or_obj | ||
| else: | ||
|
rajeeja marked this conversation as resolved.
|
||
| raise ValueError( | ||
| "If filename_or_obj is omitted, grid_filename_or_obj must be a file path or xarray.Dataset." | ||
| ) | ||
|
rajeeja marked this conversation as resolved.
|
||
|
|
||
| uxgrid, _ = _get_grid(ds, chunks, chunk_grid, use_dual, grid_kwargs, **kwargs) | ||
| filename_or_obj = grid_filename_or_obj | ||
| else: | ||
|
rajeeja marked this conversation as resolved.
|
||
| # Construct a Grid, validate parameters, and correct chunks | ||
| uxgrid, corrected_chunks = _get_grid( | ||
| grid_filename_or_obj, chunks, chunk_grid, use_dual, grid_kwargs, **kwargs | ||
| ) | ||
|
|
||
| # Load the data as a Xarray Dataset | ||
| if isinstance(filename_or_obj, xr.Dataset): | ||
| ds = filename_or_obj | ||
| else: | ||
| ds = _open_dataset_with_fallback( | ||
| filename_or_obj, chunks=corrected_chunks, **kwargs | ||
| ) | ||
|
|
||
| # Map original dimensions to the UGRID conventions | ||
| ds = _map_dims_to_ugrid(ds, uxgrid._source_dims_dict, uxgrid) | ||
|
|
||
| # Create a UXarray Dataset by linking the Xarray Dataset with a UXarray Grid | ||
| return UxDataset(ds, uxgrid=uxgrid, source_datasets=str(filename_or_obj)) | ||
| source_datasets = ( | ||
| None if isinstance(filename_or_obj, xr.Dataset) else str(filename_or_obj) | ||
| ) | ||
|
|
||
| return UxDataset(ds, uxgrid=uxgrid, source_datasets=source_datasets) | ||
|
|
||
|
|
||
| def open_mfdataset( | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.