remove lock for spa_update_config

[tiehexue]

Motivation and Context

This is another PR which try to solve #18648 , which is very clear, that spa_config_update holds a namespace lock, and calls txg_sync_wait. If txg_sync_wait blocks or hangs, the spa is in deadlock, no return, no other commands are available.

Description

spa_config_update has two-pass, all in a namespace lock, with two txg_wait_sync. The approach of this PR is bluntly remove namespace lock for spa_config_update. Inside it, when update vdev or cachefile, related lock are acquired. So the process relates to spa_config_update are retained.

However, when doing so, a lot of regression happens in CI, so my previous PR is dropped, and I make this PR from master branch again. Now most of OSs in CI pass all tests.

This PR must be carefully reviewed. spa_config_update is changed without namespace lock, and added two more calling site. In the other side, we are sure, no deadlock in spa update and txg wait, and this PR does not touch data read/write path, only the spa config update.

How Has This Been Tested?

Tested in local ubuntu26, freebsd15, and CI in my personal zfs repo.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
  spa_config_update now can not be called inside a namespace lock which is a big change.
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[kylemanna]

I think I'm encountering this upon upgrading from 2.3.7 -> 2.4.2 (haven't test 2.3.8 or 2.4.3 yet) on several solo pools (4x different pools on single HDDs, each with NVMe mirror of special vdevs). these same pools have lived lives on Arch Linux server with SAS (SAS2008) connectivity, and now on NixOS in a smaller SATA (ASM1166) NAS. Both Arch Linux (when zfs 2.4.x was release) and NixOS (25.11 -> 26.11) tripped when upstream released the 2.4.x series.

Downgrading to 2.3.x resolve the issue for months (original Arch Linux + SAS HW) and weeks (current NixOS + SATA HW).

On the most recent attempt on the NixOS zfs 2.4.2 upgrade, the array would hang approximately every other day for the week I endured before downgrading.

In each of these cases the disks (this happened on 3 of the 4 pools in this config) had no logged errors and read fine with dd if=/dev/sda bs=1M iflag=direct of=/dev/null largely dismissing the HW. The NVMe drives for the special vdevs were shared across all pools and rootfs and didn't miss a beat either.

Some notes from a hang:

pool02 would hang whenever ZFS touched it: zpool import -N pool02, zfs mount pool02/vol7, and later zpool online -e all got stuck. The kernel symptom was repeated blocked txg_sync warnings. It seems userspace was waiting behind a transaction group sync that could not complete:

txg_sync
  zio_wait
  dsl_pool_sync
  spa_sync
  txg_sync_thread

Later stack dumps showed the same root issue:

txg_sync:
  zio_wait
  dsl_pool_sync
  spa_sync

zpool online -e:
  txg_wait_synced
  spa_vdev_state_exit
  zfs_ioc_vdev_set_state

spa_async:
  txg_wait_synced
  spa_config_update

I can follow-up with more details as needed, but do not want to derail this PR.

Would love to help get this fixed and can patch my NixOS system and give it a patch shot after the initial review from devs who understand this better.

[tiehexue]

I think I'm encountering this upon upgrading from 2.3.7 -> 2.4.2 (haven't test 2.3.8 or 2.4.3 yet) on several solo pools (4x different pools on single HDDs, each with NVMe mirror of special vdevs). these same pools have lived lives on Arch Linux server with SAS (SAS2008) connectivity, and now on NixOS in a smaller SATA (ASM1166) NAS. Both Arch Linux (when zfs 2.4.x was release) and NixOS (25.11 -> 26.11) tripped when upstream released the 2.4.x series.

Downgrading to 2.3.x resolve the issue for months (original Arch Linux + SAS HW) and weeks (current NixOS + SATA HW).

On the most recent attempt on the NixOS zfs 2.4.2 upgrade, the array would hang approximately every other day for the week I endured before downgrading.

In each of these cases the disks (this happened on 3 of the 4 pools in this config) had no logged errors and read fine with dd if=/dev/sda bs=1M iflag=direct of=/dev/null largely dismissing the HW. The NVMe drives for the special vdevs were shared across all pools and rootfs and didn't miss a beat either.

Some notes from a hang:

pool02 would hang whenever ZFS touched it: zpool import -N pool02, zfs mount pool02/vol7, and later zpool online -e all got stuck. The kernel symptom was repeated blocked txg_sync warnings. It seems userspace was waiting behind a transaction group sync that could not complete:

txg_sync
  zio_wait
  dsl_pool_sync
  spa_sync
  txg_sync_thread

Later stack dumps showed the same root issue:

txg_sync:
  zio_wait
  dsl_pool_sync
  spa_sync

zpool online -e:
  txg_wait_synced
  spa_vdev_state_exit
  zfs_ioc_vdev_set_state

spa_async:
  txg_wait_synced
  spa_config_update

I can follow-up with more details as needed, but do not want to derail this PR.

Would love to help get this fixed and can patch my NixOS system and give it a patch shot after the initial review from devs who understand this better.

Looks same problem.