Automating Development Testing with k3s and GitHub Actions

In today’s fast-paced development environment, automated testing is crucial for maintaining code quality and ensuring rapid deployment. This guide will walk you through setting up an automated testing pipeline using k3s (a lightweight Kubernetes distribution) and GitHub Actions.

Prerequisites

Before we begin, make sure you have:

• A GitHub account and repository
• Basic knowledge of Docker and Kubernetes concepts
• A server or VM for hosting k3s
• SSH access to your server

Setting Up k3s

First, let’s install k3s on your server. k3s is perfect for CI/CD environments due to its lightweight nature and easy setup.

# Install k3s
curl -sfL https://get.k3s.io | sh -

# Verify installation
sudo kubectl get nodes

After installation, copy the k3s kubeconfig file:

sudo cat /etc/rancher/k3s/k3s.yaml

Store this configuration as a GitHub secret named KUBE_CONFIG in your repository.

Creating the Testing Environment

Let’s create a basic testing setup using a custom Docker image:

FROM python:3.9-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .
CMD ["pytest"]

Create a Kubernetes deployment manifest (test-deployment.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-runner
  template:
    metadata:
      labels:
        app: test-runner
    spec:
      containers:
      - name: test-runner
        image: ${DOCKER_IMAGE}
        imagePullPolicy: Always

Setting Up GitHub Actions

Create .github/workflows/test.yml in your repository:

name: Automated Testing Pipeline

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1

    - name: Login to GitHub Container Registry
      uses: docker/login-action@v1
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Build and push Docker image
      uses: docker/build-push-action@v2
      with:
        push: true
        tags: ghcr.io/${{ github.repository }}/test-runner:${{ github.sha }}

    - name: Install kubectl
      uses: azure/setup-kubectl@v1

    - name: Set up kubeconfig
      run: |
        mkdir ~/.kube
        echo "${{ secrets.KUBE_CONFIG }}" > ~/.kube/config        

    - name: Deploy to k3s and run tests
      env:
        DOCKER_IMAGE: ghcr.io/${{ github.repository }}/test-runner:${{ github.sha }}
      run: |
        envsubst < test-deployment.yaml | kubectl apply -f -
        kubectl rollout status deployment/test-runner
        kubectl logs -l app=test-runner -f        

How It Works

1. Trigger: The workflow triggers on pushes to main and pull requests.

2. Build Phase:

   • Checks out the code
   • Sets up Docker Buildx
   • Logs in to GitHub Container Registry
   • Builds and pushes the test image

3. Test Phase:

   • Sets up kubectl
   • Configures kubeconfig from GitHub secrets
   • Deploys the test runner to k3s
   • Streams logs from the test pod

Advanced Configuration

Adding Test Reports

Modify your GitHub Actions workflow to collect test reports:

    - name: Collect test reports
      if: always()
      run: |
        kubectl cp $(kubectl get pod -l app=test-runner -o jsonpath="{.items[0].metadata.name}"):/app/test-reports ./test-reports        

    - name: Upload test results
      if: always()
      uses: actions/upload-artifact@v2
      with:
        name: test-reports
        path: test-reports

Resource Limits

Add resource limits to your test deployment:

spec:
  template:
    spec:
      containers:
      - name: test-runner
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
          requests:
            memory: "256Mi"
            cpu: "250m"

Best Practices

1. Clean Up Resources: Add cleanup steps to your workflow:

   - name: Cleanup
     if: always()
     run: kubectl delete deployment test-runner
   

2. Cache Dependencies: Use GitHub Actions cache to speed up builds:

   - uses: actions/cache@v2
     with:
       path: ~/.cache/pip
       key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
   

3. Security:

   • Use minimal base images
   • Implement resource quotas
   • Regularly update dependencies
   • Use secret management for sensitive data

Troubleshooting

Common issues and solutions:

1. Connection Issues:

   • Verify k3s is running: systemctl status k3s
   • Check network policies: kubectl get networkpolicies

2. Pod Failures:

   kubectl describe pod -l app=test-runner
   kubectl logs -l app=test-runner --previous
   

3. Resource Constraints:

   • Monitor resource usage: kubectl top pods
   • Adjust limits if needed

Conclusion

Automating your testing pipeline with k3s and GitHub Actions provides a robust, scalable solution for continuous integration. This setup allows you to:

• Run tests automatically on code changes
• Scale testing resources as needed
• Maintain consistent test environments
• Track test results and failures
• Integrate with your existing GitHub workflow

Remember to monitor your k3s cluster regularly and update your dependencies to maintain a secure and efficient testing environment.