additional recv record hardening

[alek-p]

Motivation and Context

This extends the recv checking done in #18623

Description

This patch refactors and enhances how we do recv record validation. I've made sure to keep all existing check and added a few new ones.

How Has This Been Tested?

zfs-tests

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[Copilot]

Pull request overview

This PR refactors receive-side replay record validation in dmu_recv.c, centralizing checks into per-record validators and adding additional hardening (notably overflow checks) before payload reads and before applying records to the DMU.

Changes:

• Introduces shared helper routines for validating DRR_* record fields and enforcing pool/on-wire limits consistently.
• Moves/expands validation so malformed records are rejected earlier (before reading payloads and before applying changes).
• Adds overflow checks in write batching and raw maxblkid range handling to prevent unsafe arithmetic.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[behlendorf]

Yeah, the additional validation is welcome.

Thinking about this more broadly, it would be nice if we could structure things such that zstream could leverage these same checks without duplicating code. zstream is already linked against libzpool.so so if the new recv_check_* functions weren't declared static and added to sys/dmu_recv.h you should be able to just call them.

Better yet, if these new checks returned a descriptive error message in addition to EINVAL that would be quite helpful. I see there's already an errors nvlist plumbed through to zfs_ioc_recv_impl() we could easily leverage to report the details back to user space. So this might not be that big of a lift actually.

That said, I'm fine with making the hardening changes in this stand alone PR and then either investigating the above improvements in a follow up PR, or as additional commits in this PR.