Aligning AI to values

What models should be trained toward, who supplies that target, and how values are elicited from real people.

Curated by
TBD
Relevant for
10 grid cellsAgent-agent vows and 9 others
Write to
values@agi-institutions.orgAn operator routes it to the right experts.

Selected papers

3

Must-reads for understanding the field and why it matters for AGI institutions.

  1. Co‑Aligning AI and Institutions with Thick Models of Value
    Joe Edelman, Tan Zhi-Xuan, Ryan Lowe et al.2025 · Preprint

    Argues that aligning powerful AI requires institutions able to represent and revise thick, context-sensitive human values, not only individual preference aggregation.

  2. Beyond Preferences in AI Alignment
    Tan Zhi-Xuan, Micah Carroll, Matija Franklin et al.2024 · Philosophical Studies · Peer-reviewed

    Shows why preference satisfaction is too thin a target for alignment and develops a broader account of what AI systems should respond to.

  3. What are human values, and how do we align AI to them?
    Oliver Klingefjord, Ryan Lowe, Joe Edelman2024 · Preprint

    Elicits the considerations behind people's choices rather than ratings, and reconciles them across a population into a moral graph that can serve as an alignment target.

Work in the field

13

Recent work in the field worth knowing.

  1. Claude's Constitution
    Anthropic2026 · Report
  2. Model Integrity and Character
    Oliver Klingefjord2026 · Essay
  3. C3AI: Crafting and Evaluating Constitutions for Constitutional AI
    Yara Kyrychenko, Ke Zhou, Edyta Bogucka et al.2025 · WWW · Peer-reviewed
  4. Cultivating Pluralism in Algorithmic Monoculture: The Community Alignment Dataset
    Lily Hong Zhang, Smitha Milli, Karen Jusko et al.2025 · ICLR · Peer-reviewed
  5. Resource Rational Contractualism Should Guide AI Alignment
    Sydney Levine, Matija Franklin, Tan Zhi-Xuan et al.2025 · IASEAI · Peer-reviewed
  6. The Lock-in Hypothesis: Stagnation by Algorithm
    Tianyi Alex Qiu, Zhonghao He, Tejasveer Chugh et al.2025 · ICML · Peer-reviewed
  7. A Roadmap to Pluralistic Alignment
    Taylor Sorensen, Jared Moore, Jillian Fisher et al.2024 · ICML · Peer-reviewed
  8. Collective Constitutional AI: Aligning a Language Model with Public Input
    Saffron Huang, Divya Siddarth, Liane Lovitt et al.2024 · FAccT · Peer-reviewed
  9. Representative Social Choice: From Learning Theory to AI Alignment
    Tianyi Qiu2024 · Journal of Artificial Intelligence Research · Peer-reviewed
  10. Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
    Vincent Conitzer, Rachel Freedman, Jobst Heitzig et al.2024 · ICML · Peer-reviewed
  11. Machine Love
    Joel Lehman2023 · Preprint
  12. Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties
    Taylor Sorensen, Liwei Jiang, Jena Hwang et al.2023 · AAAI · Peer-reviewed
  13. What are you optimizing for? Aligning Recommender Systems with Human Values
    Jonathan Stray, Ivan Vendrov, Jeremy Nixon et al.2021 · Preprint

Foundations

3

Older work that gives background on the field.

  1. Constitutional AI: Harmlessness from AI Feedback
    Yuntao Bai et al.2022 · Preprint
  2. Training Language Models to Follow Instructions with Human Feedback
    Long Ouyang et al.2022 · NeurIPS
  3. Artificial Intelligence, Values, and Alignment
    Iason Gabriel2020 · Minds and Machines

This list isn't exhaustive. About the lists.