judgevet.testing.conformance

Status: draft.

judgevet.testing.conformance

Reusable pytest conformance tests for a SystemOnePort provider.

A provider package subclasses BaseProviderConformance in its own test suite and overrides fixtures that supply its implementation. Each rule is one test method, so a failure names the rule it breaks:

  • test_port_shape: system_one is callable with the port's positional and keyword arguments.
  • test_typed_answers_pass_policy: the kit questions receive a SystemOneResponse whose answers pass judgevet.policy.evaluate_policy, which also requires the answer type of each question kind.
  • test_failure_raises_provider_error: the failing port raises a ProviderError subclass.
  • test_scope_entry_yields_usable_port, test_scope_exit_runs_once and test_scope_propagates_body_exception: the factory keeps the provider_scope lifecycle rules.
  • test_media_refused_without_media_support: a port without every MediaSystemOnePort method callable exposes no partial media surface, and judge_with_images refuses it with ProviderCapabilityError.
  • test_media_port_refuses_undeclared_media: judge_with_images refuses media outside the declared capabilities before any media dispatch.
  • test_media_port_judges_declared_media: judge_with_images sends one kit-owned image of a declared type to the media port and receives a SystemOneResponse whose answers pass evaluate_policy. Any exception fails the rule.

BaseAsyncProviderConformance applies the shape, typed-answer, failure and scope rules to an AsyncSystemOnePort provider. Its scope rules use judgevet.providers.async_provider_scope. Source: https://github.com/Alberto-Codes/judgevet/issues/251. Its three media rules call judgevet.media.async_judge_with_images. Source: https://github.com/Alberto-Codes/judgevet/issues/252. Its provider_factory fixture returns a judgevet.providers.AsyncProviderFactory. The synchronous fixture returns a judgevet.providers.ProviderFactory.

The kit needs pytest, which the conformance extra installs. Importing this module without pytest raises ImportError. The kit needs no credentials and no inference dependency. Source: https://github.com/Alberto-Codes/judgevet/issues/241#issuecomment-5902293909.

Examples:

from contextlib import nullcontext

import pytest

from judgevet.providers import ProviderTransportError
from judgevet.testing import FakeSystemOnePort
from judgevet.testing.conformance import VALID_ANSWERS, BaseProviderConformance


class TestFakeProvider(BaseProviderConformance):
    @pytest.fixture
    def provider_factory(self):
        return lambda: nullcontext(FakeSystemOnePort(answers=VALID_ANSWERS))

    @pytest.fixture
    def failing_port(self):
        return FakeSystemOnePort(error=ProviderTransportError("offline"))
See Also

CONFORMANCE_MODEL = 'conformance-model' module-attribute

The default model label the kit passes to a provider.

CONFORMANCE_POLICY = validate_policy(Policy((NoulRule('conformance_noul', minimum=0.0), ChoiceRule('conformance_choice', 'billing'), ScoreRule('conformance_score', minimum=0.0))), CONFORMANCE_QUESTIONS) module-attribute

A policy that accepts any well-typed answer to the kit questions.

CONFORMANCE_QUESTIONS = MappingProxyType({'conformance_noul': Noul(instructions='Does the state report a duplicate charge?'), 'conformance_choice': Choice(criteria={'billing': 'A payment problem', 'technical': 'A product fault'}, instructions='Which team should handle the state?'), 'conformance_score': Score(criteria=['calm', 'concerned', 'angry'], instructions='How upset is the customer?')}) module-attribute

One question of each kind, keyed by question name.

CONFORMANCE_STATE = 'A customer reports that one invoice was charged twice.' module-attribute

The synthetic state every kit judgment receives.

INVALID_ANSWERS = MappingProxyType({'noul_for_choice': MappingProxyType({**VALID_ANSWERS, 'conformance_choice': NoulAnswer(noul=0.9)}), 'choice_off_list': MappingProxyType({**VALID_ANSWERS, 'conformance_choice': ChoiceAnswer(choice='refund', confidence=1.0, probabilities={'refund': 1.0})}), 'score_off_scale': MappingProxyType({**VALID_ANSWERS, 'conformance_score': ScoreAnswer(score=4.0, confidence=0.5, legend={0: 'a', 1: 'b', 2: 'c', 3: 'd', 4: 'e'}, probabilities={0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 1.0})})}) module-attribute

Named answer sets that the public policy evaluator rejects.

VALID_ANSWERS = MappingProxyType({'conformance_noul': NoulAnswer(noul=0.8), 'conformance_choice': ChoiceAnswer(choice='billing', confidence=0.7, probabilities={'billing': 0.85, 'technical': 0.15}), 'conformance_score': ScoreAnswer(score=1.2, confidence=0.4, legend={0: 'calm', 1: 'concerned', 2: 'angry'}, probabilities={0: 0.2, 1: 0.4, 2: 0.4})}) module-attribute

Typed answers that pass CONFORMANCE_POLICY, keyed by question name.

AsyncProviderFactory

Bases: Protocol

Construct an async context that owns an asynchronous provider.

This Protocol mirrors ProviderFactory for an AsyncSystemOnePort. The application owns rollback if entry fails before returning a port. Source: https://github.com/Alberto-Codes/judgevet/issues/251.

Attributes:

Name Type Description
__call__ method

Construct the provider's owning async context.

Examples:

from contextlib import nullcontext

import anyio

from judgevet import NoulAnswer
from judgevet.providers import AsyncProviderFactory
from judgevet.testing import AsyncFakeSystemOnePort

factory: AsyncProviderFactory = lambda: nullcontext(
    AsyncFakeSystemOnePort(answers={"q": NoulAnswer(0.9)})
)


async def main():
    async with factory() as port:
        return await port.system_one(
            "synthetic", {"q": {"type": "noul"}}, "fixture"
        )


assert anyio.run(main).answers["q"] == NoulAnswer(0.9)

__call__()

Return an async context responsible for acquisition and cleanup.

Returns:

Type Description
AbstractAsyncContextManager[AsyncSystemOnePort]

An async context yielding the selected asynchronous judgment port.

Raises:

Type Description
ProviderUnavailableError

Application support is unavailable.

BaseAsyncProviderConformance

Conformance rules an AsyncSystemOnePort provider inherits in its tests.

Name the subclass Test... so pytest collects it. Override the fixtures below as pytest fixtures on the subclass. Each rule enters one async with provider_factory() scope inside its own anyio.run call. The scope rules enter it through async_provider_scope and record each context the factory constructs.

Attributes:

Name Type Description
provider_factory fixture

Required. A zero-argument callable that returns an async context manager yielding a working port.
failing_port fixture

Required. A port configured so that every awaited system_one call fails.
provider_model fixture

The model label the kit passes. The default is CONFORMANCE_MODEL.
media_port fixture

Optional. An AsyncMediaSystemOnePort that both media-port rules check. The default is None, and then a media-capable port from provider_factory is used.

Examples:

class TestMyAsyncProvider(BaseAsyncProviderConformance):
    pass

failing_port()

Fail until the provider supplies a port configured to fail.

Raises:

Type Description
Exception

Always.

media_port()

Supply no dedicated media port, so the kit checks the factory's port.

provider_factory()

Fail until the provider supplies its factory.

Raises:

Type Description
Exception

Always.

provider_model()

Supply the kit's default model label.

Returns:

Type Description
str

The model label.

test_failure_raises_provider_error(failing_port, provider_model)

Rule 3: an awaited failed judgment raises a ProviderError subclass.

Parameters:

Name Type Description Default
failing_port AsyncSystemOnePort

The port configured to fail.

 required
provider_model str

The selected model.

 required

test_media_port_judges_declared_media(provider_factory, media_port, provider_model)

Rule 6: declared media reaches the port and returns typed answers.

A system_one_media that returns a value that is not awaitable fails this rule. The rule skips when the declaration admits no kit image.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory, used without a media port.

 required
media_port AsyncMediaSystemOnePort | None

The dedicated media port, or None.

 required
provider_model str

The selected model.

 required

test_media_port_refuses_undeclared_media(provider_factory, media_port, provider_model)

Rule 5: media outside the declared capabilities is refused first.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory, used without a media port.

 required
media_port AsyncMediaSystemOnePort | None

The dedicated media port, or None.

 required
provider_model str

The selected model.

 required

test_media_refused_without_media_support(provider_factory, provider_model)

Rule 5: a text-only port exposes no partial media surface.

A media-capable port from the factory skips this rule, and test_media_port_refuses_undeclared_media checks it instead.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory.

 required
provider_model str

The selected model.

 required

test_port_shape(provider_factory, provider_model)

Rule 1: system_one is callable as the port declares it.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory.

 required
provider_model str

The selected model.

 required

test_scope_entry_yields_usable_port(provider_factory, provider_model)

Rule 4: scope entry validates the port and yields one that answers.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory.

 required
provider_model str

The selected model.

 required

test_scope_exit_runs_once(provider_factory)

Rule 4: each scope gets its own context, and each exits exactly once.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory.

 required

test_scope_propagates_body_exception(provider_factory)

Rule 4: a body exception reaches the caller and exits the scope once.

The kit also enters one context directly, as AsyncProviderFactory callers may, and requires that its exit does not suppress the error.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory.

 required

test_typed_answers_pass_policy(provider_factory, provider_model)

Rule 2: awaited typed answers per question kind pass the evaluator.

A system_one that returns a value that is not awaitable fails this rule. A failure message also names every answer whose type does not match its question kind.

Parameters:

Name Type Description Default
provider_factory AsyncProviderFactory

The provider's factory.

 required
provider_model str

The selected model.

 required

BaseProviderConformance

Conformance rules a SystemOnePort provider inherits in its test suite.

Name the subclass Test... so pytest collects it. Override the fixtures below as pytest fixtures on the subclass.

Attributes:

Name Type Description
provider_factory fixture

Required. A ProviderFactory whose context yields a working port. The kit calls it once per scope.
provider_port fixture

A port that answers the kit questions. The default enters provider_scope(factory=provider_factory).
failing_port fixture

Required. A port configured so that every system_one call fails, such as a client of an unreachable server.
media_port fixture

Optional. A MediaSystemOnePort that both media port rules check. It must answer the kit questions with answers the kit policy accepts. The default is None, and then a media-capable provider_port is used.
provider_model fixture

The model label the kit passes. The default is CONFORMANCE_MODEL.

Examples:

class TestMyProvider(BaseProviderConformance):
    pass

failing_port()

Fail until the provider supplies a port configured to fail.

Raises:

Type Description
Exception

Always.

media_port()

Supply no dedicated media port, so the kit checks provider_port.

provider_factory()

Fail until the provider supplies its factory.

Raises:

Type Description
Exception

Always.

provider_model()

Supply the kit's default model label.

Returns:

Type Description
str

The model label.

provider_port(provider_factory)

Enter one scope of the provider's factory.

Parameters:

Name Type Description Default
provider_factory ProviderFactory

The provider's factory.

 required

Yields:

Type Description
SystemOnePort

The port the factory's context yields.

test_failure_raises_provider_error(failing_port, provider_model)

Rule 3: a failed judgment raises a ProviderError subclass.

provider_error_problem holds the message, which the asynchronous kit shares.

Parameters:

Name Type Description Default
failing_port SystemOnePort

The port configured to fail.

 required
provider_model str

The selected model.

 required

test_media_port_judges_declared_media(provider_port, media_port, provider_model)

Rule 6: declared media reaches the port and returns typed answers.

The kit sends one kit-owned image whose MIME type and size the port declares for the model. The rule skips when the declaration admits no kit image.

Parameters:

Name Type Description Default
provider_port SystemOnePort

The provider port, used when it supports media.

 required
media_port MediaSystemOnePort | None

The dedicated media port, or None.

 required
provider_model str

The selected model.

 required

Raises:

Type Description
Exception

If judge_with_images raises. A message for an exception outside ProviderError names that contract.

test_media_port_refuses_undeclared_media(provider_port, media_port, provider_model)

Rule 5: media outside the declared capabilities is refused first.

The rule selects the media port and checks its declaration as the declared-media rule does.

Parameters:

Name Type Description Default
provider_port SystemOnePort

The provider port, used when it supports media.

 required
media_port MediaSystemOnePort | None

The dedicated media port, or None.

 required
provider_model str

The selected model.

 required

test_media_refused_without_media_support(provider_port, provider_model)

Rule 5: a text-only port exposes no partial media surface.

judge_with_images treats a port as media-capable only when every MediaSystemOnePort method is callable. A port that exposes some of them, or a non-callable one, fails this rule. The library would refuse the media support that such a port appears to offer. A media-capable provider_port skips this rule, and test_media_port_refuses_undeclared_media checks it instead. The asynchronous kit shares the media member check.

Parameters:

Name Type Description Default
provider_port SystemOnePort

The provider port.

 required
provider_model str

The selected model.

 required

test_port_shape(provider_port, provider_model)

Rule 1: system_one is callable as the port declares it.

SystemOnePort is not a runtime-checkable protocol, so the kit checks the method and its signature directly. signature_problem holds the check, which the asynchronous kit shares.

Parameters:

Name Type Description Default
provider_port SystemOnePort

The provider port.

 required
provider_model str

The selected model.

 required

test_scope_entry_yields_usable_port(provider_factory, provider_model)

Rule 4: scope entry validates the port and yields one that answers.

Parameters:

Name Type Description Default
provider_factory ProviderFactory

The provider's factory.

 required
provider_model str

The selected model.

 required

test_scope_exit_runs_once(provider_factory)

Rule 4: each scope gets its own context, and each exits exactly once.

Parameters:

Name Type Description Default
provider_factory ProviderFactory

The provider's factory.

 required

test_scope_propagates_body_exception(provider_factory)

Rule 4: a body exception reaches the caller and exits the scope once.

The kit also enters one context directly, as ProviderFactory callers may, and requires that its exit does not suppress the error.

Parameters:

Name Type Description Default
provider_factory ProviderFactory

The provider's factory.

 required

test_typed_answers_pass_policy(provider_port, provider_model)

Rule 2: typed answers per question kind pass the public evaluator.

evaluate_policy is the enforcing check, including the answer type each question kind requires. A failure message also names every answer whose type does not match its question kind. The declared-media rule applies the same check to a media response.

Parameters:

Name Type Description Default
provider_port SystemOnePort

The provider port.

 required
provider_model str

The selected model.

 required