feat(docker): dockerise dev workflow (test, lint, format, ICAT, app) by joshdimanteto · Pull Request #607 · ral-facilities/datagateway-api

joshdimanteto · 2026-07-14T09:44:06Z

Description

Fix count on search-api tests due to improper clean up when creating a new investigation
This pr dockerise the development process for datagateway-api. This has been done to be similar to https://github.com/ral-facilities/icat-doi-minter-api/blob/main/docker-compose.yaml .

Testing Instructions

Add a set up instructions describing how the reviewer should test the code

Review code
Check GitHub Actions build
If icatdb Generator Script Consistency Test CI job fails, is this because of a deliberate change made to the script to change generated data (which isn't actually a problem) or is here an underlying issue with the changes made?
Review changes to test coverage
Does this change mean a new patch, minor or major version should be made? If so, does one of the commit messages feature fix:, feat: or BREAKING CHANGE: so a release is automatically made via GitHub Actions upon merge?

Agile Board Tracking

Closes #612

joshdimanteto · 2026-07-20T12:31:22Z

This (CI / icatdb Generator Script Consistency Test (pull_request)) is expected to fail because i am now using docker for the icat server

patrick-austin

Changes look OK to me, but I had a couple of lines where I'm not sure whether we need it as is or it's an artifact from the previous version that could/should be changed?

patrick-austin · 2026-07-20T13:31:33Z

+      - ./icat/post-boot-commands.asadmin:/config/post-boot-commands.asadmin
+    environment:
+      - POSTBOOT_COMMANDS=/config/post-boot-commands.asadmin
+      - ROOT_USERS="simple/root anon/anon"


Lines 31/32 imply that adding anon/anon to the root users was done at build time, then that image tagged with dg-api. Is this environment variable still used?

No, this needs to be removed. I didn't notice this. Thanks

patrick-austin · 2026-07-20T13:46:36Z

+  pip-install-test:
+    container_name: dg_api_pip-install-test
+    build:
+      context: .
+      target: test
+
+    command:
+      - sh
+      - -c
+      - |
+        timeout 10 uv run python -m datagateway_api.main || code=$?
+        if [ "${code:-0}" -ne 124 ] && [ "${code:-0}" -ne 0 ]; then
+          exit "${code:-1}"
+        fi
+
+    # volumes:
+    ## use for local development
+    # - ./datagateway_api:/datagateway-api-run/datagateway_api
+    depends_on:
+      icat_payara:
+        condition: service_healthy
+    networks:
+      - dg_network
+    profiles: [tests]


Despite the name, this doesn't seem to be calling pip anywhere? So I'm thinking:

Do we still want a test for this? (probably?)

If so, is it more appropriate to use the test target or the prod target? (my feeling is the latter, since I can't think it needs anything dev-y for this and the closer it is to prod the better the test is?)

Should the title for this be changed to not reference pip? (probably?)

I might revert this as this test was need due to this issue which was reported here #467 and a specific test for for production docker container.

patrick-austin · 2026-07-20T13:47:29Z

  pip-install-testing:
    runs-on: ubuntu-24.04
-    strategy:
-      fail-fast: false
    name: Pip Install Test
+
    steps:
-      - name: Add apt repo
-        run: sudo add-apt-repository universe
-      # Checkout DataGateway API and setup Python
-      - name: Check out repo
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v3.5.3
-      - name: Add apt repo
-        run: sudo add-apt-repository universe
-      # Setup Python
-      - name: Setup Python
-        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
-        with:
-          python-version: "3.11"
-          architecture: x64
-        # ICAT Ansible clone and install dependencies
-      - name: Checkout icat-ansible
+      - name: Checkout DataGateway API
        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
-        with:
-          repository: icatproject-contrib/icat-ansible
-          ref: payara7
-          path: icat-ansible
-      - name: Install Ansible
-        run: pip install -r icat-ansible/requirements.txt
-      # Prep for running the playbook
-      - name: Create hosts file
-        run: echo -e "[icatdb_minimal_hosts]\nlocalhost ansible_connection=local" > icat-ansible/hosts
-      - name: Prepare vault pass
-        run: echo -e "icattravispw" > icat-ansible/vault_pass.txt
-      - name: Move vault to directory it'll get detected by Ansible
-        run: mv icat-ansible/vault.yml icat-ansible/group_vars/all
-      - name: Replace default payara user with Actions user
-        run: |
-          sed -i -e "s/^payara_user: \"glassfish\"/payara_user: \"runner\"/" icat-ansible/group_vars/all/vars.yml

-      # Force hostname to localhost - bug fix for previous ICAT Ansible issues on Actions
-      - name: Change hostname to localhost
-        run: sudo hostname -b localhost
+      - name: Start test dependencies
+        run: docker compose --profile dependencies up -d

-      # Remove existing MySQL installation so it doesn't interfere with GitHub Actions
-      - name: Remove existing mysql
-        run: |
-          sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
-          sudo apt-get remove --purge "mysql*"
-          sudo rm -rf /var/lib/mysql* /etc/mysql
-
-      # Create local instance of ICAT
-      - name: Run ICAT Ansible Playbook
-        run: |
-          ansible-playbook icat-ansible/icatdb_minimal_hosts.yml -i icat-ansible/hosts --vault-password-file icat-ansible/vault_pass.txt -vv
-
-      # rootUserNames needs editing as anon/anon is used in search API and required to pass endpoint tests
-      - name: Add anon user to rootUserNames
-        run: |
-          awk -F" =" '/rootUserNames/{$2="= simple/root anon/anon";print;next}1' /home/runner/install/icat.server/run.properties > /home/runner/install/icat.server/run.properties.tmp
-      - name: Apply rootUserNames change
-        run: |
-          mv -f /home/runner/install/icat.server/run.properties.tmp /home/runner/install/icat.server/run.properties
-      - name: Reinstall ICAT Server
-        run: |
-          cd /home/runner/install/icat.server/ && ./setup -vv install
-
-        # Create virtual environment and install DataGateway API
-      - name: Create and activate virtual environment
-        run: |
-          python3 -m venv dg-api-venv
-          source dg-api-venv/bin/activate
-      - name: Install DataGateway API via pip
-        run: pip install .
-
-      # Prep for using the API for tests
-      - name: Configure datagateway extension
-        run: echo "`yq \
-          '.datagateway_api.extension="/datagateway-api"' datagateway_api/config.yaml.example`" > datagateway_api/config.yaml.example
-      - name: Create config.yaml
-        run: cp datagateway_api/config.yaml.example datagateway_api/config.yaml
-      - name: Create search_api_mapping.json
-        run: cp datagateway_api/search_api_mapping.json.example datagateway_api/search_api_mapping.json
-      - name: Create logging.ini
-        run: cp datagateway_api/logging.example.ini datagateway_api/logging.ini
-
-      # Launch API to see if it starts correctly or has a startup issue
-      # Code logic used from https://stackoverflow.com/a/63643845
-      - name: Start API
-        run: timeout 10 python -m datagateway_api.main || code=$?; if [[ $code -ne 124 && $code -ne 0 ]]; then exit $code; fi
+      - name: Run pip install test
+        run: docker compose run pip-install-test


See other comment about the name pip-install-testing and apparent lack of pip?

joshdimanteto force-pushed the dockerise-development branch 7 times, most recently from ec630e6 to 0e1b2ae Compare July 20, 2026 11:24

feat(docker): dockerise dev workflow (test, lint, format, ICAT, app)

1ae4160

joshdimanteto force-pushed the dockerise-development branch from 0e1b2ae to 1ae4160 Compare July 20, 2026 11:28

base the "test" image with "dev" image

0da2e31

joshdimanteto requested a review from patrick-austin July 20, 2026 12:19

joshdimanteto marked this pull request as ready for review July 20, 2026 12:19

update the readme #612

97183a0

add comment to icat_payara image #612

6caed82

patrick-austin reviewed Jul 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(docker): dockerise dev workflow (test, lint, format, ICAT, app)#607

feat(docker): dockerise dev workflow (test, lint, format, ICAT, app)#607
joshdimanteto wants to merge 4 commits into
developfrom
dockerise-development

joshdimanteto commented Jul 14, 2026 •

edited

Loading

Uh oh!

joshdimanteto commented Jul 20, 2026

Uh oh!

patrick-austin left a comment

Uh oh!

patrick-austin Jul 20, 2026

Uh oh!

joshdimanteto Jul 20, 2026

Uh oh!

patrick-austin Jul 20, 2026

Uh oh!

joshdimanteto Jul 20, 2026

Uh oh!

patrick-austin Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joshdimanteto commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing Instructions

Agile Board Tracking

Uh oh!

joshdimanteto commented Jul 20, 2026

Uh oh!

patrick-austin left a comment

Choose a reason for hiding this comment

Uh oh!

patrick-austin Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

joshdimanteto Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

patrick-austin Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

joshdimanteto Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

patrick-austin Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshdimanteto commented Jul 14, 2026 •

edited

Loading