The Call Home reverse listener backlog is hardcoded:
src/session_p.h: #define NC_REVERSE_QUEUE 5
src/session_server.c: listen(sock, NC_REVERSE_QUEUE)
In deployments with many devices performing SSH Call Home, a value
of 5 causes the kernel listen queue to overflow and silently drop
incoming SYNs. Tuning net.core.somaxconn does not help, because
listen(2) caps the effective backlog at the application-supplied
value.
We are requesting an API to configure this value at runtime
(or, at minimum, a build-time CMake option), so applications can
size the backlog appropriately for their deployment without
patching the library.
Version / branch
libnetconf2 [v4.2.14 / devel @ ]
libyang [version]
OS / distro [e.g. Ubuntu 24.04 in a Kubernetes pod]
Reproduction
- N NETCONF devices configured for SSH Call Home to a single server.
- Server's accept loop is busy (typical: schema retrieval, config
push, or any other per-session work taking > a few hundred ms).
- After 5 unaccepted connections queue up, additional SYNs are
silently dropped by the kernel. Devices retry, fail in bursts.
Observed on the server node:
ss -lnt 'sport = :4334'
State Recv-Q Send-Q Local Address:Port
LISTEN 5 5 0.0.0.0:4334 <- queue at cap
nstat -az | grep -i listen
TcpExtListenOverflows <incrementing>
TcpExtListenDrops <incrementing>
In the device-side pcap, SYNs to the server get no SYN-ACK
response during these windows.
After locally patching NC_REVERSE_QUEUE from 5 to 256 and
rebuilding, the drops stop and all devices complete Call Home.
Proposed API
A new public setter, defaulting to current behavior:
/**
* @brief Set the listen() backlog used for Call Home reverse listeners.
* Affects subsequently created listeners only.
* @param[in] backlog Backlog value (1..INT_MAX). 0 = library default (5).
* @return 0 on success, -1 on invalid argument.
*/
int nc_server_ch_set_listen_backlog(int backlog);
(Mirror on the client side if the client also exposes the
reverse listener: nc_client_ch_set_listen_backlog().)
Internally, replace the hardcoded use with the stored value:
listen(sock, server_opts.ch_listen_backlog ? server_opts.ch_listen_backlog : NC_REVERSE_QUEUE);
Default remains 5 — fully backward compatible.
Alternatives considered
- CMake option (e.g. -DNC_REVERSE_QUEUE=256). Easy, but still
requires a custom build per deployment.
- Reading from an environment variable. Works but is non-idiomatic
for libnetconf2.
- Exposing it via the ietf-netconf-server YANG model under the
call-home container (e.g. an "accept-queue-length" leaf, perhaps
a CESNET augmentation). Cleanest for users who already configure
via YANG; more work to implement.
We'd be happy with any of (1)–(3); a setter API is what we'd
prefer because it requires no rebuild and no YANG changes for
existing users.
The Call Home reverse listener backlog is hardcoded:
src/session_p.h: #define NC_REVERSE_QUEUE 5
src/session_server.c: listen(sock, NC_REVERSE_QUEUE)
In deployments with many devices performing SSH Call Home, a value
of 5 causes the kernel listen queue to overflow and silently drop
incoming SYNs. Tuning net.core.somaxconn does not help, because
listen(2) caps the effective backlog at the application-supplied
value.
We are requesting an API to configure this value at runtime
(or, at minimum, a build-time CMake option), so applications can
size the backlog appropriately for their deployment without
patching the library.
Version / branch
libnetconf2 [v4.2.14 / devel @ ]
libyang [version]
OS / distro [e.g. Ubuntu 24.04 in a Kubernetes pod]
Reproduction
push, or any other per-session work taking > a few hundred ms).
silently dropped by the kernel. Devices retry, fail in bursts.
Observed on the server node:
ss -lnt 'sport = :4334'
State Recv-Q Send-Q Local Address:Port
LISTEN 5 5 0.0.0.0:4334 <- queue at cap
In the device-side pcap, SYNs to the server get no SYN-ACK
response during these windows.
After locally patching NC_REVERSE_QUEUE from 5 to 256 and
rebuilding, the drops stop and all devices complete Call Home.
Proposed API
A new public setter, defaulting to current behavior:
(Mirror on the client side if the client also exposes the
reverse listener: nc_client_ch_set_listen_backlog().)
Internally, replace the hardcoded use with the stored value:
listen(sock, server_opts.ch_listen_backlog ? server_opts.ch_listen_backlog : NC_REVERSE_QUEUE);
Default remains 5 — fully backward compatible.
Alternatives considered
requires a custom build per deployment.
for libnetconf2.
call-home container (e.g. an "accept-queue-length" leaf, perhaps
a CESNET augmentation). Cleanest for users who already configure
via YANG; more work to implement.
We'd be happy with any of (1)–(3); a setter API is what we'd
prefer because it requires no rebuild and no YANG changes for
existing users.