JaleesBench: Are AI Assistants Good Spiritual Company?

A benchmark for whether an AI agent is a righteous companion, judged by the residue an exchange leaves on the user

Abstract

Large language models are already advisors to millions of people of faith who bring them real decisions. The pressing question for a person of faith is not what a model knows or professes but what its counsel does to the person who receives it.

We introduce JaleesBench, which measures whether an AI agent is a righteous companion, judged by the residue an exchange leaves on the user, in the manner of the perfume-seller and the blacksmith.

It comprises 140 two-turn scenarios drawn from a classical compilation organized by virtue (Riyāḍ al-Ṣāliḥīn), under six adversarial pressures and three framings, scored by two frontier judges against each scenario’s own supporting texts.

Across eight systems, we find:

Generic frontier models are only middling companions out of the box but a one-page guide makes them genuinely good ones, on par with the domain-tuned assistant: the frontier APIs climb from +0.28/+0.23 to a Guided +0.84–0.87, so most of the expert’s edge is companionship instruction that fits in a prompt.
Every system caves under relational pressure, insistence and personal appeal.
The domain-tuned assistant’s advantage is overwhelmingly its retrieval-and-prompting layer, not its base model (+0.74 over the identical underlying model).
It can be used to improve existing systems: guided by its diagnosis, a single steadfastness instruction lifts a deployed Islamic assistant from +0.48 to +0.84 (Faith unstated, after pressure), matching the best guided frontier systems while preserving first-response quality.

The construct is faith-general; we instantiate it for Islam as the first of a planned cross-tradition family. Code, scenario bank, and rubric are open source (github.com/iaser-ai/jaleesbench), with an interactive results browser at s.iaser.ai/jb.

Read the full paper (PDF)