CDPO

Multi-LLM Direct Preference Optimization (DPO) class. This implementation:

Uses multiple LLMs to judge preferences between responses
Implements the DPO algorithm to train a policy model
Features proper typing and logging with loguru

Key components include:

ModelConfig: Pydantic model for language model configuration
DPOConfig: Configuration for the DPO training process
MultiLLMDPO: Main class implementing the DPO algorithm with multi-LLM support

collect_preferences(): Collects preference judgments from various LLMs
train(): Trains the policy model using the DPO algorithm
evaluate(): Evaluates the trained model against a reference model