Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) and related alignment methods are techniques used to shape how an AI model behaves, especially in terms of safety, helpfulness, and adherence to human preferences. It involves collecting human feedback on model outputs, ranking or scoring responses, and fine-tuning the model to prefer better behaviour.

Compliance

RLHF offers support in complying with the NIST AI Risk Management Framework, which calls for trustworthy, safe, and human-aligned AI outcomes, and with the EU AI Act, which requires risk mitigation and safe system behaviour.  

In Practice

In practice, RLHF looks like aligning language models through feedback, as part of training, rule-based alignment, supervised fine-tuning with curated examples, or hybrid models that combine human and automated feedback. Currently, there is a wide range of practices that experiment with open RLHF pipelines, community fine-tuning datasets, and alternative alignment methods. However, there is no established approach that underpins all of this, resulting in a wide range of practices and restricted replicability.  

In RLHF pipelines, human annotators rank model outputs, train reward models, and optimize the base model against that reward. Rules-based alignment involves embedding explicit principles into training and guiding model behaviour through constraints. Iterative alignment involves repeated cycles of deployment, feedback collection, and refinement.

Alignment is a continuous practice, and not a one-and-done approach. It is typically coupled with product goals, calling for ongoing and regular engagement. Alignment fundamentally aims to shape helpfulness, harmlessness, honesty, tone, and behaviour.

Embedding Responsibility and Ethical Practices

Raw AI models do not behave in ways that are safe or socially acceptable. RLHF addresses this concern by prioritizing behaviour control at the model level, scalable safety mechanisms, and a way to encode human preferences into systems. It offers a way for those who will most be affected by the technology to have a say in its outputs. RLHF signals a commitment to embed governance within the model rather than outside of it, which makes a powerful impact by shaping model behaviour during training, and creates embedded governance systems. RLHF is scalable and works in real time.