File delivery range support

[pzl]

What is the problem this PR solves?

Enables clients (e.g. endpoint) to gracefully resume downloads of (large) files via the HTTP Range header.

This allows for completing large file transfers under unstable or periodic network conditions, as opposed to starting the file download over.

N.B.:

• This only supports single ranges (e.g. bytes=100-400) and not multipart range requests (e.g. bytes=0-500, 1024-2048, -600).
• Also HTTP Spec allows for graceful degradation here. For unclear range headers (or for any reason if the server chooses not to fulfill a particular range), either a 416 Range Unsatisfiable error OR HTTP 200 with full file contents are both acceptable responses. This implementation mostly generates 416s for clients to handle malformed byte strings, but in narrow cases where intent is unclear (e.g. the header is present without a value: Range: ) then we do fall back to HTTP 200 with file contents, as spec defines.

How does this PR solve the problem?

Only fetch chunks that fall inside the requested range

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

[pzl]

@elastic/elastic-agent-control-plane sorry for the short turnaround, could this be looked over before FF?

It is additional behavior only: when serving files through the file delivery route, if a Range header is present, delivers the subset of bytes instead of the whole file (follows HTTP spec). Oherwise all behavior is the same

[cmacknz]

You don't set X-File-SHA2 on this path, should you?

[pzl]

ambiguous when sending byte ranges (would it apply to the range or still the hash of the whole file?) so it wasn't used for range requests. It could be, it would just have to be an explicit call if it were needed