PCoKG: Personality-aware Commonsense Reasoning with Debate

AI summary

Overview

Research area: Commonsense reasoning and knowledge graph construction in natural language processing, with a focus on personality-aware (MBTI-conditioned) inference and personalised dialogue generation.

Technical level: Intermediate. The paper assumes familiarity with knowledge graph triplets (ATOMIC/COMET), large language model prompting, role-play and multi-agent debate, and standard generation metrics (BLEU, ROUGE), but it explains its pipeline step by step.

Scope: The paper introduces PCoKG, a personality-aware commonsense knowledge graph of 521,316 quadruples built with LLM role-play and a debate mechanism, and validates it through dataset analysis, fine-tuning and ablation experiments, and a persona-based dialogue generation application.

What This Paper Is About

Existing commonsense reasoning resources such as ATOMIC and COMET treat the relation between an event and its causes or effects as universal: a given event produces one plausible inference regardless of who is doing the reasoning. The paper argues this ignores individual cognitive differences, and specifically personality traits, which shape how people interpret the same event. The goal is to build a large-scale knowledge graph that pairs each event and reasoning dimension with a specific personality type, and to show that this personality-aware knowledge improves both reasoning models and downstream persona-based dialogue generation.

Key Contributions

1. A personality-aware knowledge representation. The paper introduces the Personality-aware Commonsense Knowledge Graph (PCoKG), which extends the conventional (e, r, t) triplet into a quadruple that adds a personality variable p based on the Myers-Briggs Type Indicator (MBTI). The abstract writes the quadruple as (e, p, r, t); Section 3 writes it as (e, r, p, t).

2. A scalable LLM-driven construction pipeline. Rather than crowd-sourcing annotations from people with specific personality traits — which the paper describes as difficult and costly at scale — the pipeline simulates MBTI types with LLMs, using three evaluators to filter event–dimension pairs and sampling personality types according to their global population distribution.

3. A multi-agent debate mechanism for quality control. Generated inferences are refined by a debate among three roles — a Proponent, an Opponent, and a Judge — across multiple rounds, with the Judge providing feedback that drives iterative regeneration when an output is judged unacceptable.

4. Extensive validation plus a downstream application. The dataset is analysed through readability, mutual information, and human evaluation; fine-tuned and ablated across multiple LLM backbones; and applied to persona-based dialogue generation on the SPC dataset.

Main Findings

• Dataset scale and composition: PCoKG contains 521,316 quadruples covering 9 inference dimensions and all 16 MBTI types, with 15,077 events, an average event length of 4.79, and an average reasoning outcome length of 8.75.

• Filtering reduced the source data substantially: From ATOMIC20 events, language_tool_python filtering produced 19,184 well-formed events, and the three-evaluator screening yielded 95,783 (e, r) pairs involving 15,227 events. (The final dataset statistics report 15,077 events, a slightly different count from the 15,227 events in the filtered pair set.)

• Personality-linked readability differences: On the Flesch Reading Ease score (0–100, higher meaning simpler), ESFP scored 77.7 and ESTP 74.0 at the high end, while INTJ scored 37.0 and INTP 39.6 at the low end, which the authors read as consistent with MBTI theory about Sensing/Feeling versus Intuitive/Thinking types.

• Reasoning clusters align with MBTI labels: Adjusted Mutual Information between MBTI types and K-means clusters of reasoning embeddings was highest for self-perception dimensions such as xAttr (0.511176) and xIntent (0.240094), and lowest for oReact (0.114518) and oWant (0.121604). Shuffling MBTI labels 100 times produced near-zero values, and Mann-Whitney U tests indicated all original AMI values were significantly higher (p < 0.01).

• Human evaluation was positive but personality consistency lagged: Three psychology graduate students rated 1,440 instances on a 3-point scale (0 = No, 1 = Somewhat, 2 = Yes). Reasoning coherence averaged 1.78, naturalness 1.71, and personality consistency 1.63, with a Fleiss' Kappa of 0.57.

• Fine-tuned models beat prompt-based LLMs and COMET: PCoKGM (MiniCPM4) reached BLEU-4 14.50, R-1 32.99, R-2 15.27, R-L 31.38, versus 5.38 / 15.34 / 2.09 / 14.28 for o4-mini and 12.58 / 30.51 / 12.77 / 28.91 for COMET with LLaMA3. Among the prompted LLMs, R1 (2.67 BLEU-4), 1.6-Thinking (3.39), and o4-mini (5.38) all fell well short of PCoKGM.

• Every pipeline component contributed: Removing MBTI caused the largest drop (LLaMA3 fell from 13.73 to 10.16 BLEU-4), followed by removing selection (11.25) and removing debate (12.09); removing both selection and debate gave 10.66. The same ordering held for Qwen3 and MiniCPM4.

• Larger models performed better: Across the Qwen3, LLaMA3, and MiniCPM4 families, performance improved with parameter scale on both BLEU-4 and ROUGE metrics, and the gap between model families narrowed at larger scales.

• Personality-aware knowledge improved dialogue generation: On the 967-sample SPC test set, every model improved when augmented: 1.6-Thinking rose from BLEU-4 16.51 to 18.28 with COMET and 18.92 with PCoKGM; R1 from 10.40 to 11.76 and 12.03; o4-mini from 6.05 to 6.65 and 6.85.

• Reported limitation: The work considers personality traits alone and does not account for factors such as gender or occupation.

Methodology in Plain English

The pipeline has two stages. First, the authors take events from the ATOMIC knowledge base and clean them with a grammar-checking tool (language_tool_python), keeping 19,184 well-formed events. They then ask three LLMs — Deepseek-R1, Qwen-Turbo, and Doubao-1.6-Seed — to score each event–dimension pair on a 10-point scale for whether it is likely to produce different reasoning across personality types. The paper states that only pairs scoring above 6 from all evaluators are kept, while Algorithm 1 in the paper uses a threshold of s ≥ 6. This produced 95,783 (e, r) pairs.

Second, they sample MBTI personality types according to their global population distribution (as published by 16personalities.com) and prompt an LLM to role-play each event–dimension pair as that personality type, producing a personalised inference. To improve reliability, each generation passes through a debate: a Proponent argues that the inference fits the target personality, an Opponent challenges it, and after multiple rounds a Judge rules on whether the output is acceptable. If not, the Judge's feedback is fed back into the model and the output is regenerated. The paper does not report the concrete values used for the maximum number of generations or the number of debate rounds.

For analysis, the authors check readability scores across MBTI types, cluster reasoning embeddings with K-means (with cluster counts chosen by Silhouette Score) and measure Adjusted Mutual Information against MBTI labels, and run a human evaluation. For modelling, they fine-tune Qwen3-0.6B, LLaMA3-1B, and MiniCPM4-0.5B on a 9 : 0.5 : 0.5 event-based split (13,576 training events / 468,479 quadruples; 750 validation events / 26,321 quadruples; 751 test events / 26,516 quadruples) using LLaMA-Factory on four 3090 GPUs, batch size 8 per GPU, gradient accumulation of 4, warmup ratio 0.1, a cosine learning rate scheduler, one epoch, validation every 300 steps, and early stopping after three non-improving evaluations. The setup section describes full-parameter fine-tuning, while the abstract and the Figure 4 caption describe LoRA-based fine-tuning. A COMET baseline was trained with reasoning dimensions and personality types as special tokens, and one-shot prompting was used for Deepseek-R1, Doubao-seed-1.6-thinking, and GPT-o4-mini. For the dialogue application, a BERT classifier trained on the Kaggle MBTI Personality Types 500 Dataset predicted the personality types of SPC participants from their character descriptions.

Why This Matters

Impact on research: The paper argues that personality is a missing variable in commonsense knowledge bases, and that the standard one-event-one-inference formulation is not enough to capture how diverse people reason. It offers a large-scale, automatically constructed alternative to the closest prior work (Yang et al. 2024), which the paper describes as manually built and therefore difficult to scale. It also positions the debate-based construction pipeline as reusable for other personality theories or additional role attributes beyond MBTI.

Real-world applications:

• Personalised dialogue systems and chatbots that adapt responses to a user's personality type rather than producing generic replies.
• Emotional support and empathetic dialogue agents, a setting the paper links to prior COMET-based applications.
• Character-driven content, such as game NPCs or virtual companions whose behaviour stays consistent with a defined personality profile.
• Persona-consistent response generation in interactive assistants, as demonstrated on the SPC dataset.

Industry relevance: Because the pipeline uses LLMs for both generation and quality control, it removes the cost and recruitment difficulty of collecting personality-labelled commonsense annotations from humans, making large-scale expansion practical for companies that need personalised reasoning data. The finding that performance rises with model scale also gives practitioners a concrete scaling lever, while the public code repository supports reuse.

Future Directions

• Extending the framework beyond MBTI to incorporate additional personal attributes such as gender and occupation, which the paper explicitly names as outside its current scope.
• Improving personality consistency, which received the lowest human evaluation score (1.63 of 2) and which the authors state still has room for improvement.
• Improving reasoning about other people's mental states: the oReact, oWant, and oEffect dimensions showed the lowest Adjusted Mutual Information with MBTI types.
• Testing whether the pipeline generalises to other personality theories or role attributes, as the authors claim it is designed to do but do not demonstrate in this paper.
• Clarifying and possibly reconciling the training setup, given that the setup section describes full-parameter fine-tuning while the abstract and Figure 4 caption describe LoRA-based fine-tuning.

Target Audience

Researchers and practitioners in natural language processing and knowledge graph construction who work on commonsense reasoning, personalisation, or dialogue generation. It is also useful for engineers building persona-aware conversational systems who need a structured, scalable data resource and a reusable LLM-driven annotation pipeline. Readers without prior exposure to ATOMIC-style triplets, MBTI, or multi-agent debate prompting will find the paper accessible but should expect the evaluation sections to assume familiarity with metrics like BLEU, ROUGE, and Adjusted Mutual Information.

Authors’ abstract

Most commonsense reasoning models overlook the influence of personality traits, limiting their effectiveness in personalized systems such as dialogue generation. To address this limitation, we introduce the Personality-aware Commonsense Knowledge Graph (PCoKG), a structured dataset comprising 521,316 quadruples. We begin by employing three evaluators to score and filter events from the ATOMIC dataset, selecting those that are likely to elicit diverse reasoning patterns across different personality types. For knowledge graph construction, we leverage the role-playing capabilities of large language models (LLMs) to perform reasoning tasks. To enhance the quality of the generated knowledge, we incorporate a debate mechanism consisting of a proponent, an opponent, and a judge, which iteratively refines the outputs through feedback loops. We evaluate the dataset from multiple perspectives and conduct fine-tuning and ablation experiments using multiple LLM backbones to assess PCoKG's robustness and the effectiveness of its construction pipeline. Our LoRA-based fine-tuning results indicate a positive correlation between model performance and the parameter scale of the base models. Finally, we apply PCoKG to persona-based dialogue generation, where it demonstrates improved consistency between generated responses and reference outputs. This work bridges the gap between commonsense reasoning and individual cognitive differences, enabling the development of more personalized and context-aware AI systems.