The codebeaver.yml configuration file is the central point for defining and managing your CodeBeaver test suite configuration. This file specifies how CodeBeaver executes tests across your codebase, including test environments, frameworks, and execution parameters.

It is broadly divided into test types. A simple example is the following:

unit:
  from: pytest
e2e:
  check-if-homepage-has-signup: # name of the E2E
    url: https://en.wikipedia.org/wiki/Julius_Caesar
    steps:
      - check if the page mentions the word Julius
      - check if the page mentions the word Caesar

If you don’t need Unit Tests or End 2 End tests, you can simply not define them in the codebeaver.yml and they will not run.

​
Unit Tests

The Unit tests section defines how CodeBeaver will run and iterate on your Unit tests.

​
Workspace

A Codebeaver.yml file is divided in workspaces. A workspace represent a separate test environment.

​
Single Workspace

For repositories requiring one test environment, you can define the workspace at the root level like so:

unit:
  from: pytest
  environment:
    - DJANGO_DEBUG=True

​
Multiple Workspaces

For monorepos or projects requiring multiple test environments, you can define separate workspaces like so:

unit:
  workspaces:
    - name: django # Name of the workspace. Required.
      path: django-service # Path to the root of the workspace inside your repository. Required.
      from: pytest # In this case, we are using a template to configure the workspace.
    - name: react # This is the name of the second workspace.
      path: react-service
      from: jest

​
Workspace reference

A workspace can have the following properties:

services: # The services that you need to run your tests. These work exactly like docker-compose services. (Required)
  python:
    image: python:3.11
    environment: # You can define environment variables for your service. If you do, this environment variable will only be available to this service.
      - DJANGO_DEBUG=True
    depends_on:
      - redis
  redis:
    image: redis:latest
  postgres:
    build: # like docker-compose, you can define a build context and a Dockerfile to build your service.
      context: .
      dockerfile: Dockerfile
main_service: python # Primary service for test execution. It has to be one of the services defined in the services section. (Required)
environment: # Workspace-level environment variables. These will be available to all services in this workspace.
  - ENV=test

test_commands: # Test execution commands (Required)
  - coverage run -m pytest --show-capture=no --json-report
  - coverage combine
  - coverage json -i || true

single_file_test_commands: # Single file test execution (Required).
  - coverage run -m pytest --json-report "$TEST_FILE" # You can use the $TEST_FILE and the $FILE_TO_COVER environment variables
  - coverage json

setup_commands: # What you need to set up your test environment. These are executed only once per workspace
  - python -m ensurepip || true

linter_commands: # Optional. Run after the code for a particular file has been generated. Useful to keep linting in check
  - npx run eslint --fix $TEST_FILE # You can use the $TEST_FILE and the $FILE_TO_COVER environment variables

ignore: # A list of paths that CodeBeaver will ignore. Supports wildcards.
  - "**/node_modules"
  - "**/dist"

Services work just like docker-compose services. You can define ports, depends_on, and so on.

If not using a template, you must define the following properties:

  • services
  • main_service
  • test_commands
  • single_file_test_commands

​
Templates

CodeBeaver provides pre-configured templates to simplify setup. Templates can be referenced using the from directive. For example, to use the pytest template without any customization:

unit:
  from: pytest

You can see a list of the available templates below.

​
Template Customization

Templates serve as a foundation that you can extend. For example, to add environment variables to a template:

unit:
  from: pytest
  environment:
    - DJANGO_DEBUG=True

The @merge directive is a powerful feature that allows you to combine any root-level configuration from a template with your custom configuration. It can be used in two ways:

  1. As a list marker (@merge):
    • Place it as the first item in a list to combine with the template’s list
  2. As a dictionary directive ({"@merge": value}):
    • Use it to merge nested dictionaries or lists

The @merge directive can be used with any root-level configuration, including:

  • services
  • environment
  • setup_commands
  • test_commands
  • single_file_test_commands

​
List Merging Example

unit:
  from: pytest
  setup_commands:
    - "@merge" # This will include all setup_commands from the pytest template
    - pip install -q selenium
    - pip install -q playwright

​
Dictionary Merging Example

unit:
  from: pytest
  services:
    python:
      environment:
        - "@merge": # This will merge with the template's python service environment
        - CUSTOM_VAR=true
        - ANOTHER_VAR=123

Without the @merge directive, your configuration would completely override the template’s configuration. Including @merge ensures you’re extending rather than replacing the template’s settings:

# With @merge - Extends template
unit:
  from: pytest
  environment:
    - "@merge"
    - CUSTOM_VAR=true # Adds to template's environment variables

# Without @merge - Replaces template
unit:
  from: pytest
  environment:
    - CUSTOM_VAR=true # Replaces ALL template's environment variables

​
Templates list

CodeBeaver provides official templates for common testing frameworks:

  • pytest
  • pytest-django
  • unittest
  • jest
  • nyc
  • vitest
  • bun
  • ruby
  • go

You can all official templates in our template repository.

​
End-to-End (E2E) Testing

End-to-End (E2E) testing in CodeBeaver simulates real user interactions within a browser environment, validating the complete functionality of your application from a user’s perspective. CodeBeaver currently supports browser-based E2E testing using natural language test descriptions.

​
Configuring E2E Tests

E2E tests are defined in the codebeaver.yml file using a human-readable format. Each test consists of a unique identifier, a starting URL, and a sequence of test steps. Here’s an example:

e2e:
  article-navigation:
    url: https://en.wikipedia.org/wiki/Julius_Caesar
    steps:
      - Navigate to the `Gallic Wars` link
      - Click on the `Julius Caesar` link
      - Check if the page contains an image of Caesar

​
Test Structure

Each E2E test requires the following components:

  • Test Identifier: A unique name that describes the test scenario
  • URL: The starting point for the test execution
  • Steps: A sequence of actions and validations written in natural language

​
Validation Requirements

Each test must include at least one validation step that begins with the word “Check”. This validation step is crucial as it’s used by CodeBeaver’s LLM model to determine test success or failure.

​
Best Practices

To ensure maintainable and reliable E2E tests:

  1. Keep Tests Focused

    • Write concise tests that validate specific user flows
    • Avoid combining multiple unrelated scenarios in a single test

  2. Write Resilient Tests

    • Use high-level, semantic descriptions rather than implementation details
    • Focus on user interactions and outcomes rather than specific UI elements
    • Write steps that can withstand minor UI changes

  3. Use Clear Descriptions

    • Write steps that clearly communicate the intended action
    • Include explicit validation steps to verify expected outcomes

​
Integration Tests

Coming soon.

​
Support and Feedback

CodeBeaver is currently in open beta. We welcome your feedback and suggestions: