[Closed][BE] Cache & Optimize Woodpecker CI #3450

This is a branch I have been working on to optimise performance of the woodpecker CI using a drone cache plugin & some changes to the clippy step. This branch in its current state will not pass, as there requires some ancillary services.

What this PR does is two things:

• Adds in a cache restore and publish step, using a local minio instance to store/retrieve the cache. Currently the following folders are cached & restored each run:

  • target/
  • .cargo/
  • api_tests/node_modules

• Adjusts the cargo clippy step to run once only with all lints present. To acheive this, in test modules the unwrap_used and indexing_slicing are turned off (hence the many changes).

Testing on my own woodpecker instance, with a warmed cache, the CI run takes <8mins:

For the cache there are two methods available for storing/retrieving the build cache between runs:

• Use an S3 compatible object store, i.e, minio or actually amazon S3. I've been using local minio instance to test & have include a docker-compose.yml. If this is the path that people want to take, then I would suggest having it fairly close, if not the same host as the woodpecker ci.

• Use a volume mount cache. This unfortunately requires that the project settings needs to be set to trusted, which I'm not sure the ramifications of it are.

Here's the docker-compose.yml I've been using:

# docker-compose.yml
version: '3'

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:latest
    ports:
      - 8000:8000
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker/
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=${WOODPECKER_HOST}
      - WOODPECKER_GITHUB=true
      - WOODPECKER_GITHUB_CLIENT=${WOODPECKER_GITHUB_CLIENT}
      - WOODPECKER_GITHUB_SECRET=${WOODPECKER_GITHUB_SECRET}
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
      - WOODPECKER_ADMIN=cetra3

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:latest
    command: agent
    restart: always
    depends_on:
      - woodpecker-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - agent-cache:/cache/
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}

  minio:
    image: minio/minio
    ports:
      - 9000:9000
      - 9090:9090
    command:
      - minio
      - server
      - --console-address
      - ":9090"
    environment:
      - MINIO_ROOT_USER=testing123
      - MINIO_ROOT_PASSWORD=testing123
      - MINIO_VOLUMES=/mnt/data
    volumes:
      - minio-cache:/mnt/data
     
volumes:
  woodpecker-server-data:
  agent-cache:
  minio-cache: