Visual Categorization Across Minds and Models: Cognitive Analysis of Human Labeling and Neuro-Symbolic Integration

AI summary

Overview

Research area: Computational cognitive science and neuro-symbolic AI — a comparative study of how humans and a deep neural network label ambiguous, low-resolution images.

Technical level: Intermediate. It assumes familiarity with basic deep learning (CNNs, softmax, Grad-CAM) alongside introductory cognitive-science concepts (bounded rationality, Marr's levels, PDP).

Scope (one sentence): The paper compares twelve human participants against a from-scratch ResNet-18 on ten 32×32 CIFAR-10 images, pairing verbal strategy reports and confidence ratings with Grad-CAM attention maps, and uses that comparison to argue for neuro-symbolic architectures.

What This Paper Is About

Deep neural networks classify images by learning statistical regularities in pixel data, while humans appear to rely on analogies, shape prototypes, and embodied context — yet the two are usually compared only on accuracy. This paper asks how human cognitive strategies for labeling ambiguous visual stimuli compare to the feature-based labeling processes of AI, and how those insights could inform cognitively aligned neuro-symbolic architectures. It answers by running the same ten ambiguous images through both a ResNet-18 baseline and human participants, then interpreting the divergences through Marr's tri-level hypothesis, Simon's bounded rationality, ACT-R/Soar-style architectures, PDP theory, and case-based reasoning.

Key Contributions

1. Operationalizing bounded rationality. Human confidence ratings and satisficing responses were collected and used as empirical evidence for heuristic, effort-economizing decision-making under uncertainty.
2. Mapping analogical reasoning against AI attention. Participant strategy reports (shape, texture, familiarity, guessing) were compared with the model's Grad-CAM heatmaps to contrast symbolic prototype inference with sub-symbolic feature activation.
3. Contrasting embodied and distributed cognition with connectionist activations. Contextual references in participant explanations (for example, background color and shadow cues) were set against AI's isolated pixel-level processing.
4. Proposing an integration pathway. The paper argues for neuro-symbolic architectures as a route toward AI that is both performant and cognitively grounded.

Main Findings

• Human accuracy was perfect on the reported aggregate. The paper states that all participants identified every image correctly, a 100 percent label accuracy across all 10 test images. Note that the sample response table in the paper shows individual entries with labels differing from the image file name (for example, an airplane image labeled "deer" and a bird image labeled "cat"), so the aggregate claim and the sample table do not fully agree as presented.
• Human confidence was uniformly high. Mean confidence ranged from 4.42 to 4.92 on a 1–5 scale across the ten images, with standard deviations of 0.53 or less; the mode was 5 for every image. The lowest mean, 4.42, was for bird_01.png; the highest, 4.92, was for airplane_00.png, automobile_01.png, and cat_00.png.
• AI accuracy was far below human level. The ResNet-18, trained from scratch for five epochs with Adam (learning rate 0.001) and cross-entropy loss, reached approximately 70.7 percent test accuracy (described elsewhere in the paper as approximately 70 percent).
• AI errors were concentrated on ambiguous images and were sometimes semantically distant. Examples include airplane_01.png predicted as deer at 0.486 confidence, bird_01.png predicted as cat at 0.531, and deer_00.png predicted as airplane at 0.369. High-confidence correct cases included airplane_00.png (airplane, 0.771), automobile_01.png (automobile, 0.957), and cat_00.png (cat, 0.887).
• Grad-CAM attention was localized and texture-driven. On airplane_00.png the heatmap concentrated on wing and body edges; on deer_00.png it highlighted diagonal shapes near the top that the model apparently read as airplane-like; on bird_00.png attention slid from head and beak toward body texture, producing a low-confidence correct classification.
• A mismatch between AI attention and human focus. For deer_00.png, Grad-CAM emphasized fur-like texture patches while participants reported focusing on the top of the image for "antler-like" shapes, which the paper attributes to CNNs prioritizing texture over shape.
• Confidence tracked feature clarity, not semantic meaning, in the AI. The paper reports that AI confidence correlated with how clear the local features were rather than with the object's semantic category.
• Adjusting effort tracked ambiguity in humans. Cognitive load ratings varied with image ambiguity (reported as values 2–4 in the response analysis, and as rising from a mean of 2 to 4 for clear versus ambiguous images), with confidence and cognitive load moving in opposite directions — framed as metacognitive monitoring.
• Trust and affect were high. Trust-in-AI ratings were reported in the 4–5 range, average trust for the deer_00.png discussion was 4.5, and positive emotion and enjoyment ratings accompanied high-confidence, schema-matching responses.
• Interpretation through Marr's levels. The paper concludes that humans and AI diverge at the algorithmic and implementation levels despite sharing computational aims, and Table I maps convolutional layers to cognitive functions (Conv1/2 to early visual cortex V1/V2, mid layers to perceptual schemata, fully connected layers to symbolic label decision, activation flow to PDP interactive activation).

Methodology in Plain English

Two systems were shown the same ten low-resolution (32×32) CIFAR-10 images chosen to span a range from unambiguous to ambiguous.

Human side: Participants were recruited through a Google Form distributed to classmates and colleagues — a convenience sample; all had completed the required CITI training. Twelve participants each supplied a class label from the ten CIFAR categories, a confidence rating on a 1–5 Likert scale, and a brief written explanation of their reasoning strategy, along with other cognitive baseline questions. Additional recorded dimensions per response included cognitive load, strategies used, focus area, emotion, trust in AI, enjoyment, prior knowledge, and an attention check.

AI side: A ResNet-18 was modified for the small CIFAR-10 input size by changing the first convolutional layer from the standard 7×7 stride-2 kernel to a 3×3 stride-1 kernel with no initial max pooling, so spatial information would not be lost. Data augmentation included random cropping, horizontal flips, and normalization to the CIFAR-10 mean and standard deviation. The model was trained from scratch for five epochs using the Adam optimizer (learning rate 0.001) and cross-entropy loss, reaching roughly 70 percent test accuracy. It produced softmax probability scores and Grad-CAM heatmaps for each of the ten images.

Comparison: For every stimulus the authors collected the AI's predicted label, its confidence score, and the Grad-CAM overlay, then set these against human labels, confidence ratings, and verbal explanations. The comparison is organized at three levels — computational (classification accuracy), algorithmic (heatmap versus verbal explanation), and implementation (connectionist model versus embodied cognitive processes).

Why This Matters

The work is a small but deliberately designed probe into why human and machine vision diverge under uncertainty, not just whether they do. Its contribution is methodological as much as empirical: it pairs a model's attention maps with participants' introspective strategy reports and confidence ratings, producing a comparison framework that other researchers could reuse.

Real-world applications:

• Medical imaging and radiology triage, where low-resolution or noisy scans must be classified and where calibrated confidence and human-interpretable reasoning are safety-critical.
• Autonomous driving and robotics, where systems must handle degraded sensor input and avoid confident errors caused by texture rather than shape.
• Assistive and accessibility technology, where recognizing objects from poor-quality imagery (for example, low-vision assistance) requires robustness to ambiguity.
• Human-AI collaborative decision tools, where trust calibration depends on users understanding when and why a model is uncertain.

Industry relevance: The findings speak directly to the design of interpretable AI. The paper's evidence that a ResNet-18 can be misled by texture while reporting moderate confidence, and that Grad-CAM attention often does not align with the cues humans use, argues against treating accuracy alone as sufficient for deployment. The proposed neuro-symbolic fallback logic — for example, validating the presence of wings before accepting an "airplane" label when neural confidence is low — is a concrete architectural suggestion relevant to any industry where explainability is required or regulated.

Future Directions

1. Building the hybrid system. The authors plan to design and evaluate a neuro-symbolic model that combines structured symbolic reasoning rules with neural attention, using Grad-CAM activations to trigger fallback logic checks when neural confidence is low. They cite Feature-CAM as prior evidence that such hybrids improve alignment with human attention and confidence.
2. Scaling and aligning representations. The paper flags known challenges in scaling neuro-symbolic architectures and maintaining unified representations, and proposes exploring graph-based embeddings to align Grad-CAM-derived feature spaces with symbolic prototypes, benchmarked on datasets such as CLEVR and GQA.
3. Trust, transparency, and collaboration studies. The next phase would integrate symbolic trace generation and confidence reporting into AI output, followed by controlled user studies on how such systems affect human trust, reliance, and collaborative task outcomes in ambiguous or low-fidelity conditions.
4. Neuromorphic and context-aware models. The paper suggests that systems incorporating multisensory representations and experiential priors — rather than static images alone — could reduce reliance on misleading textures and better approximate human multimodal cognition.

An open question the paper leaves implicit is how far the findings generalize beyond a single small architecture (ResNet-18) trained for five epochs on ten images with twelve participants; the paper reports no ablation, no alternative model comparison, and no formal limitations section.

Target Audience

This paper is best suited to students and researchers working at the intersection of cognitive science and machine learning — particularly those interested in neuro-symbolic AI, explainable AI, and human-AI comparison. It will also be useful to practitioners who need a concrete, readable example of how to combine Grad-CAM interpretability tools with human behavioral data. Readers looking for large-scale empirical benchmarking or state-of-the-art model performance will find the study small by design; its value lies in the interpretive framework and the design template it offers rather than in its scale.

Authors’ abstract

Understanding how humans and AI systems interpret ambiguous visual stimuli offers critical insight into the nature of perception, reasoning, and decision-making. This paper examines image labeling performance across human participants and deep neural networks, focusing on low-resolution, perceptually degraded stimuli. Drawing from computational cognitive science, cognitive architectures, and connectionist-symbolic hybrid models, we contrast human strategies such as analogical reasoning, shape-based recognition, and confidence modulation with AI's feature-based processing. Grounded in Marr's tri-level hypothesis, Simon's bounded rationality, and Thagard's frameworks of representation and emotion, we analyze participant responses in relation to Grad-CAM visualizations of model attention. Human behavior is further interpreted through cognitive principles modeled in ACT-R and Soar, revealing layered and heuristic decision strategies under uncertainty. Our findings highlight key parallels and divergences between biological and artificial systems in representation, inference, and confidence calibration. The analysis motivates future neuro-symbolic architectures that unify structured symbolic reasoning with connectionist representations. Such architectures, informed by principles of embodiment, explainability, and cognitive alignment, offer a path toward AI systems that are not only performant but also interpretable and cognitively grounded.