fix(api): bound Celery worker concurrency to a configurable default

[b-abderrahmane]

Context

Fix #10968.

Without an explicit worker_concurrency setting, Celery falls back to os.cpu_count() of the host. On Kubernetes nodes with many CPUs (16+) the worker container spawns one prefork child per CPU, each loading the full SDK working set, and routinely OOMKills under typical 4 GiB pod limits even when idle. The existing --max-tasks-per-child 1 recycles each child after one task — it bounds per-task leaks but not the number of concurrent slots.

Description

Adds CELERY_WORKER_CONCURRENCY = env.int("DJANGO_CELERY_WORKER_CONCURRENCY", default=4) in config/settings/celery.py. The Celery app already loads Django settings via config_from_object("django.conf:settings", namespace="CELERY"), so the setting flows through to app.conf.worker_concurrency and the prefork pool size with no entrypoint changes.

A default of 4 matches the per-process SDK memory footprint observed in typical deployments; operators size up via DJANGO_CELERY_WORKER_CONCURRENCY (also added to api/.env.example next to the sibling DJANGO_CELERY_DEADLOCK_ATTEMPTS).

Changes:

• api/src/backend/config/settings/celery.py — one Django setting, matching the existing env.int("DJANGO_*", default=...) convention used by DJANGO_CELERY_DEADLOCK_ATTEMPTS.
• api/.env.example — document the new env var alongside its siblings.
• api/CHANGELOG.md — entry under [1.29.0] (Prowler UNRELEASED) → 🚀 Added.

Steps to review

1. Read the 1-line settings change and the changelog/env-example additions — three files, +7 lines.
2. Confirm the setting name/namespace is consistent with existing siblings:
   • CELERY_DEADLOCK_ATTEMPTS = env.int("DJANGO_CELERY_DEADLOCK_ATTEMPTS", default=5) (existing)
   • CELERY_WORKER_CONCURRENCY = env.int("DJANGO_CELERY_WORKER_CONCURRENCY", default=4) (this PR)
3. Optionally reproduce the validation below in any K8s deployment.

Evidence

Validated end-to-end on a 20-core AKS node with the worker image rebuilt from this branch (prowler-api, dockerized from api/). Worker Deployment patched to use the test image; same pod spec (requests.memory=1Gi, limits.memory=4Gi).

Before vs after

Metric	Before (master, no setting)	After (this PR, default 4)
Django settings.CELERY_WORKER_CONCURRENCY	(unset → falls through)	4
app.conf.worker_concurrency	os.cpu_count() = 20	4
Prefork children per pod	20	4
Idle resident memory	~3.4 GiB	~880 MiB
Restart count (over 26h soak)	4 and 8 (OOMKilled)	0

Process tree (after) — 1 parent + 4 ForkPool children:

$ kubectl exec prowler-worker-7c56d9dd6d-4lcdt -- ls /proc/[0-9]*/comm | grep -E celery
pid=10  python -m celery -A config.celery worker -n c5...
pid=38  python -m celery -A config.celery worker -n c5...   # ForkPoolWorker-1
pid=39  python -m celery -A config.celery worker -n c5...   # ForkPoolWorker-2
pid=40  python -m celery -A config.celery worker -n c5...   # ForkPoolWorker-3
pid=41  python -m celery -A config.celery worker -n c5...   # ForkPoolWorker-4

Wiring proof — Django setting → Celery config:

$ kubectl exec prowler-worker-... -- manage.py shell -c "
    from django.conf import settings; from config.celery import celery_app
    print(settings.CELERY_WORKER_CONCURRENCY)
    print(celery_app.conf.worker_concurrency)
  "
4
4

Override demonstration — DJANGO_CELERY_WORKER_CONCURRENCY=2 propagates through:

$ kubectl set env deploy/prowler-worker DJANGO_CELERY_WORKER_CONCURRENCY=2
$ kubectl exec <new-pod> -- printenv DJANGO_CELERY_WORKER_CONCURRENCY
2
$ kubectl exec <new-pod> -- manage.py shell -c "..."
settings.CELERY_WORKER_CONCURRENCY=2
celery_app.conf.worker_concurrency=2
$ kubectl top pod <new-pod>
NAME                              CPU(cores)   MEMORY(bytes)
prowler-worker-5cb6bb6bc4-2b9q7   521m         564Mi          # was 3.4 GiB

Process count goes to 1 parent + 2 children (with brief overlaps from --max-tasks-per-child 1 recycling, expected).

Checklist

• Review if the code is being covered by tests.
  • The change is a single Django settings line wrapping env.int() (the same pattern already exercised for DJANGO_CELERY_DEADLOCK_ATTEMPTS with no dedicated test in api/src/backend/api/tests/test_celery_settings.py); validated in a real cluster as documented above.
• Review if code is being documented following pyguide §3.8.
• Review if backport is needed.
• Review if is needed to change the Readme.md — N/A.
• Ensure new entries are added to CHANGELOG.md — added under [1.29.0] → 🚀 Added.

API

• All issue/task requirements work as expected on the API — see Evidence above.
• Endpoint response output (if applicable) — N/A (worker-side change, no endpoints touched).
• EXPLAIN ANALYZE output for new/modified queries or indexes (if applicable) — N/A.
• Performance test results (if applicable) — included above (memory + process-count).
• Any other relevant evidence of the implementation (if applicable) — included above.
• Verify if API specs need to be regenerated — N/A.
• Check if version updates are required — N/A (no dependency or spec changes).
• Ensure new entries are added to CHANGELOG.md.

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

✅ Conflict Markers Resolved

All conflict markers have been successfully resolved in this pull request.

[cesararroba]

Thanks for the contribution!

[jfagoagas]

Please:

• Move this to the next release as API 1.27.0 was released as part of Prowler v5.26.0.
• Include it in ### 🚀 Added

[b-abderrahmane]

@jfagoagas It should be good now.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.97%. Comparing base (7d03bc5) to head (8a78b42).
⚠️ Report is 21 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #11075   +/-   ##
=======================================
  Coverage   93.97%   93.97%           
=======================================
  Files         237      237           
  Lines       34829    34873   +44     
=======================================
+ Hits        32729    32772   +43     
- Misses       2100     2101    +1     

Flag	Coverage Δ
api	93.97% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
prowler	∅ <ø> (∅)
api	93.97% <100.00%> (+<0.01%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[AdriiiPRodri]

Thanks for the PR. The configurability is useful, but this currently changes default runtime behavior. Please address the inline comments below before merge.

[AdriiiPRodri]

This introduces a behavior change by forcing concurrency to 4 when the env var is not set.
Could we keep the current default behavior (Celery auto-detected concurrency) and only set CELERY_WORKER_CONCURRENCY when DJANGO_CELERY_WORKER_CONCURRENCY is explicitly provided?

[AdriiiPRodri]

Suggestion: make this variable clearly optional (e.g., commented out with an explanation).
Keeping it active in .env.example makes 4 look like the default expected runtime behavior, which changes current behavior.

[AdriiiPRodri]

The changelog text says "default 4", but that implies a global behavior change.
Could we reword this to: optional setting that bounds concurrency when explicitly configured?

[AdriiiPRodri]

Could we add tests for the new setting behavior?

1. env var missing/empty -> concurrency is not forced
2. valid numeric value -> concurrency is applied
3. invalid value -> explicit error path

Dismissing duplicate review entry to keep a single clean review state.

[AdriiiPRodri]

Please address the inline comments before merge.

[b-abderrahmane]

Please address the inline comments before merge.

Good catch @AdriiiPRodri
Restored default behavior and added tests.