Encoding and Understanding Astrophysical Information in Large Language Model-Generated Summaries

AI summary

Overview

Research area: Natural Language Processing and model interpretability applied to astrophysics — specifically, whether LLM text embeddings encode physical quantities measured from telescope data, and which linguistic features carry that information.

Technical level: Intermediate. The paper is readable without deep astrophysics background, but assumes familiarity with embeddings, dimensionality reduction, autoencoders, and basic X-ray astronomy terminology.

Scope: The paper tests, on X-ray sources from the Chandra Source Catalog, whether LLM-generated summaries of scientific papers encode measurable physical properties, whether prompt wording changes that encoding, and which text features drive it.

What This Paper Is About

Astrophysical measurements — such as X-ray spectra — encode physical properties of a source, and humans describe those observations in language. This paper asks whether the embeddings of LLM-generated summaries of that language actually codify the underlying physics, and whether the way the summary is prompted changes how well they do so. Using sparse autoencoders, it also asks which specific concepts in the text are responsible for that encoding.

Key Contributions

1. A reproducible pipeline that pairs LLM-generated text summaries of Chandra X-ray sources with physical summary statistics from the Chandra Source Catalog, using NASA ADS papers as the text corpus and OpenAI's gpt-4o-mini to generate summaries, with embeddings produced by the ada-002 model.
2. A quantitative comparison of two prompts — an original baseline prompt and a longer, more structured prompt — measured by k-nearest neighbors purity analysis (k=10) over t-SNE projections of the embeddings.
3. An interpretability analysis using pre-trained sparse autoencoders (a 1-layer gelu-1l language model trained to reconstruct the activations of a one-layer transformer with a 512-neuron MLP layer, from Bricken et al. (2023)) to identify which monosemantic features correspond to clusters of summaries sharing similar physical properties.
4. Open-sourced code for the dataset construction, embedding generation, and feature extraction.

Main Findings

• Prompt structure strongly improves alignment. Going from the baseline prompt to the structured prompt, k-nearest neighbors clustering purity improved from 0.7998 to 0.8468 (5.9%) for hardness ratio, from 0.8185 to 0.9418 (15.1%) for power law gamma, and from 0.6346 to 0.9994 (57.5%) for variability index.
• Other physical parameters also improved. Black body temperature and variability probability showed clustering purity improvements from the baseline to the experimental prompt, though the specific values are not reported in the paper.
• The physics is inferred, not stated. In the general case the LLM does not have access to the numerical values of parameters such as the power law slope or hardness ratio, yet the t-SNE clustering indicates that information about these parameters is registered in the text. The authors suggest the pre-trained model may recognize that certain classes of astrophysical objects tend to display specific values, or that the descriptive language is sufficient to infer numerical quantities.
• Specific concepts drive specific clusters. Summaries clustering by power-law slope are also associated with particular concepts. One cluster's sources are largely associated with Eclipsing Binaries; the circled cluster was dominated by features relating to non-thermal sources, correlating with a lower power law gamma value.
• The model makes physical inferences. In the bottom cluster of Figure 2, sources had non-thermal characteristics; one source was described as a type-II QSO and referenced "energetic outbursts from the AGN." Although the power law gamma value and thermal/non-thermal status were never mentioned, the model still encoded a specific parameter value based on the object type and other characteristics.
• SAE features are semantically meaningful but cluster-dependent. Feature 411 was labeled "Galaxy Cluster Thermal Dynamics and Cooling Processes," and in the highlighted cluster of Figure 2 discussed non-thermal dynamics, while a similar feature in other clusters talked about thermal dynamics. Automated feature labeling used Claude Sonnet 4 and Google Gemini Pro 2.5, alongside handmade note taking.
• Feature labels from Table 2 include: "Scientific Inference and Possibility" (1212), "X-ray Binary Outburst Cycles" (839), "Galaxy Cluster Thermal Dynamics and Cooling Processes" (411), "X-ray Astrophysical Source Characteristics" (1262), "X-ray Source Variability and Emission Properties" (1181), and "Methodological Analysis of Variable Astrophysical Sources" (1160).

Methodology in Plain English

The researchers started with Chandra telescope observations and queried the Chandra Archives for scientific papers about each specific source. They paired these papers with the NASA Astrophysics Data System text corpus and with physical measurements from the Chandra Source Catalog.

They then fed the papers to gpt-4o-mini, asking it to summarize the physical characteristics of each source using a combination of source sky coordinates and source identifiers. Each generated summary was converted into a vector using the ada-002 embedding model. Alongside each embedding, they stored measurable physical quantities from the catalog, including hardness ratio (ACIS hard 2.0-7.0 keV minus soft 0.5-1.2 keV bands), the power-law photon index, black body temperature, a variability index ranging from 0 to 10 that combines the Gregory-Loredo variability probability with fractions of the multi-resolution light curve within 3-sigma and 5-sigma of the average count rate, and a variability probability.

To test whether prompting mattered, they ran the whole process twice — once with a baseline prompt asking for X-ray properties and hypotheses, and once with a structured prompt specifying formatting, handling of missing information, and explicit lists of the spectral and variability properties to look for. The structured prompt also tried to avoid encoding text that carries no physical content.

They projected the embeddings into two dimensions with t-SNE and measured clustering quality using k-nearest neighbors purity with k=10, which checks whether sources with similar physical properties land near each other. For the interpretability question, they ran the summaries through pre-trained sparse autoencoders, which force each neuron to represent a distinct, monosemantic feature. They then looked at which features fired in clusters that shared a physical property, restricting attention to the top five strongest and most unique features per cluster. The visualization tool outputs the top twenty words that activate each feature plus the snippet of the summary where the token appears.

Why This Matters

Impact on research: The work provides a way to check whether the language scientists use to describe observations actually preserves the measurable physics, and offers an interpretability method for auditing what scientific concepts a language model has learned. It also suggests a route to labeling physical properties of large source catalogs using text alone.

Real-world applications (as implied or supported by the paper):

• Automatically enriching astronomical catalogs by associating text-derived summaries with physical measurements.
• An expert-facing interface, currently being created, that gives astronomers a text summary alongside the accompanying spectra to validate possibly anomalous objects.
• Retrieval-augmented question-answering chatbots for research astronomy, a use case cited in the related work.
• Triaging the scientific literature by surfacing which described concepts correlate with which measured properties.

Industry relevance: The method — pairing a generative model's output with structured measurements, then using sparse autoencoders to explain which concepts drove the alignment — is not specific to astrophysics. The paper explicitly frames the pipeline as reproducible within other scientific fields, which makes it relevant to any domain where numerical measurements and free-text descriptions coexist.

Future Directions

• Expanding the dataset beyond the current Chandra source sample.
• Testing prompting with different models to find the optimum encoding, since results here are specific to gpt-4o-mini and ada-002.
• Completing and deploying the astronomer validation interface for anomalous objects, which pairs a text-based summary with the source spectra.
• Open questions the paper raises but does not resolve: whether the LLM is genuinely learning physics concepts or merely associating object classifications with typical parameter ranges, and how far prompting alone can push correlation before the approach plateaus.

Target Audience

Astronomers and astrophysics researchers interested in applying language models to archival data; NLP and interpretability researchers studying how physical or numerical information is represented in text embeddings; and machine learning practitioners in scientific domains who want a concrete template for pairing LLM-generated summaries with ground-truth measurements. Readers looking for a purely methodological NLP contribution may find the paper's dataset and interpretability framing more relevant than its modeling innovations.

Authors’ abstract

Large Language Models have demonstrated the ability to generalize well at many levels across domains, modalities, and even shown in-context learning capabilities. This enables research questions regarding how they can be used to encode physical information that is usually only available from scientific measurements, and loosely encoded in textual descriptions. Using astrophysics as a test bed, we investigate if LLM embeddings can codify physical summary statistics that are obtained from scientific measurements through two main questions: 1) Does prompting play a role on how those quantities are codified by the LLM? and 2) What aspects of language are most important in encoding the physics represented by the measurement? We investigate this using sparse autoencoders that extract interpretable features from the text.