Перейти к содержанию

CI/CD

Для упрощения и автоматизации процессов разработки настроили CI/CD.

GitHub Actions pre-commit

Автоматическая проверка кода используя уже настроенный .pre-commit-config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
name: pre-commit hooks check

on:
  pull_request:
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: "pre-commit"
  cancel-in-progress: false

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    permissions:
      contents: write

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
          python-version: '3.13'

    - name: Run pre-commit on codebase
      uses: pre-commit/action@v3.0.1
      continue-on-error: true

    - name: Auto commit action
      uses: stefanzweifel/git-auto-commit-action@v5
      with:
          commit_message: Apply pre-commit hooks auto-fix
Если были обнаружены ошибки, то скрип их исправит и сделает коммит в ветку с изменениями.

GitHub Pages

Автоматический деплой сайта на прямо в github pages после коммита в ветку main:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
name: Deploy documentation with GitHub Pages

on:
  push:
    branches: ["main"]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Pages
        uses: actions/configure-pages@v5

      - uses: actions/setup-python@v5
        with:
          python-version-file: "pyproject.toml"

      - name: Install Poetry
        uses: snok/install-poetry@v1
        with:
            virtualenvs-create: true
            virtualenvs-in-project: true
            virtualenvs-path: .venv
            installer-parallel: true

      - name: Load cached venv
        id: cached-poetry-dependencies
        uses: actions/cache@v4
        with:
            path: .venv
            key: venv-mkdocs-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}

      - name: Install dependencies
        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
        run: poetry install --without client,linters --no-interaction --no-root

      - name: Build documentation
        run: poetry run mkdocs build --clean --verbose

      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: site

      - name: Deploy to GitHub Pages
        uses: actions/deploy-pages@v4
Статичный сайт доступен по адресу https://AI-YP-24-6.github.io/img_classifier/.

Docker

Для упрощения проверки докер файлов используется hadolint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
name: docker linter

env:
    GLOBAL_FAILED: false

on:
  pull_request:
      paths:
          - "**.Dockerfile"  # Trigger only on changes to files (commits).
          - "compose.yaml"
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: "docker-linter"
  cancel-in-progress: true

jobs:
  lint:
    runs-on: ubuntu-latest
    permissions:
        pull-requests: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Run hadolint
        run: |
           cat *.Dockerfile | docker run --rm -i hadolint/hadolint > lint-report.txt
        # cat compose.yaml | docker run --rm -i hadolint/hadolint >> lint-report.txt
        # doesn't work: https://github.com/hadolint/hadolint/issues/1091
        continue-on-error: true

      - name: Limit report to 100 lines
        if: ${{ github.event_name == 'pull_request'}}
        run: |
            if [[ $(wc -l < lint-report.txt) != 0 ]]; then
                tail -n 100 lint-report.txt > lint-limited-report.txt;
                echo true >> $GLOBAL_FAILED;
            else
                echo "No docker📦 errors found 🎉✨" > lint-limited-report.txt;
                echo false >> $GLOBAL_FAILED;
            fi


      - name: Upload lint report
        if: ${{ github.event_name == 'pull_request' }}
        uses: actions/upload-artifact@v4
        with:
          name: lint-report
          path: lint-limited-report.txt

      - name: Post lint results as PR comment
        if: ${{ github.event_name == 'pull_request' }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          header: Dockerfile errors
          path: lint-limited-report.txt
          hide: true
          skip_unchanged: true

      - name: Fail build
        if: ${{ env.GLOBAL_FAILED == true }}
        run: exit 1
Если ошибок нет, то скрип напишет сообщение “No docker📦 errors found 🎉✨” в комментарии к pull request.

Pycodestyle & Pylint

Автоматическая проверка кода на соответствие стандартам pycodestyle и pylint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
name: Pylint and pycodestyle

on:
  pull_request:
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: "linters"
  cancel-in-progress: true

jobs:
  lint:
    runs-on: ubuntu-latest
    permissions:
        pull-requests: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version-file: "pyproject.toml"

      - name: Install Poetry
        uses: snok/install-poetry@v1.4
        with:
            virtualenvs-create: true
            virtualenvs-in-project: true
            virtualenvs-path: .venv
            installer-parallel: true

      - name: Load cached venv
        id: cached-poetry-dependencies
        uses: actions/cache@v4
        with:
            path: .venv
            key: venv-linters-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}

      - name: Install dependencies
        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
        run: poetry install --without docs --no-interaction --no-root

      - name: Run pycodestyle
        run: |
          poetry run pycodestyle . --exclude=.venv --statistics --max-line-length=120 > lint-report.txt || true
        continue-on-error: true

      - name: Run Pylint
        run: |
          poetry run pylint **/*.py --ignore='.venv' --recursive=y --max-line-length=120 "--disable=C0103,C0114,C0115" >> lint-report.txt || true
        continue-on-error: true

      - name: Limit report to 100 lines
        run: |
          tail -n 100 lint-report.txt > lint-limited-report.txt

      - name: Upload lint report
        if: ${{ github.event_name == 'pull_request' }}
        uses: actions/upload-artifact@v4
        with:
          name: lint-report
          path: lint-limited-report.txt

      - name: Post lint results as PR comment
        if: ${{ github.event_name == 'pull_request' }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          header: Lint Results (First 100 Lines)
          path: lint-limited-report.txt
          hide: true
          skip_unchanged: true
Если были обнаружены ошибки, то скрип их исправит и сделает комментарий в pull request.

Notebook-formater

У ноутбуков из Google colab есть раздел metadata, которые не может обработать nbconvert. Jq скрипт очищает файл от metadata.
Так же скрипт делает корректные execution_count ячеек

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
name: Run notebook.sh

on:
    pull_request:
        paths:
            - "**.ipynb"  # Trigger only on changes to files (commits).

jobs:
  run-notebook:
    runs-on: ubuntu-latest
    permissions:
      contents: write

    steps:
      # Checkout the repository
      - name: Checkout code
        uses: actions/checkout@v4

      # Run the notebook script
      - name: Run Tools/notebook.sh
        run: |
          chmod +x Tools/notebook.sh
          ./Tools/notebook.sh

      # Commit the changes
      - name: Commit changes
        uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "Apply changes from Tools/notebook.sh"
          file_pattern: '*.ipynb'

Если были обнаружены ошибки, то скрип их исправит и сделает коммит в pull request.