Fine-tuning LLMs for Tourist Trajectory Prediction using Field Experiment Data

AI summary

Overview

Research area: Applied large language model fine-tuning for human mobility modeling and tourism behavior prediction, positioned at the intersection of AI for urban planning and next Point-of-Interest (POI) prediction.

Technical level: Intermediate. Readers should be comfortable with supervised fine-tuning, LoRA/QLoRA adapters, tokenizer vocabulary extension, and standard sequence-evaluation metrics (BLEU, n-gram overlap, Levenshtein distance).

Scope: The paper fine-tunes Llama-3.1-8B on 566 real tourist trajectories collected at Wakayama Castle Park, Japan, to predict the next POI from persona and environmental context, achieving 49.1% accuracy and reporting generalization to undersampled conditions such as rain.

What This Paper Is About

Regional tourist destinations face uneven visitor distribution, but planning mobility interventions (shuttles, e-bikes, walking support devices) requires predicting how tourists would actually change their behavior. Traditional Markov and deep learning models cannot handle the context-dependent nature of tourist decisions and require observing every scenario before they can predict it. The paper's goal is to show that fine-tuning a large language model on local trajectory data lets it combine pretrained commonsense about human behavior with destination-specific patterns, so it can predict next-POI choices even in scenarios it barely observed during training.

Key Contributions

1. A field-experiment dataset for trajectory prediction. The authors collected 566 tourist trajectories at Wakayama Castle Park in December 2023 during a Green Slow Mobility pilot, combining 87 GPS-tracked participants (1-second intervals) with 479 QR code stamp rally participants who scanned codes at 37 major attractions, and augmented coverage with 31 POIs from OpenStreetMap for a total of 68 POIs.

2. A text-based trajectory representation with vocabulary extension. Trajectories are serialized as structured text using angle-bracketed tags for persona, environment, time, action, area, category, and POI, and the Llama-3.1-8B tokenizer vocabulary is extended with special tokens for POI names, areas, categories, and structural tags, with embeddings initialized by averaging constituent subwords.

3. A QLoRA fine-tuning recipe for this task. The paper applies quantized low-rank adaptation with rank r = 32 and scaling alpha = 64 not only to attention projection layers (Query, Key, Value, Output) but also to the embedding (embed_tokens) and output (lm_head) layers, reporting that this inclusion was essential for learning representations of the newly added POI tokens.

4. Evidence that general multilingual pretraining beats language-specific pretraining for this task. Llama-3-Swallow-8B, a Japanese-specialized variant, reached 39.8% POI and 45.8% category accuracy under identical QLoRA settings, below Llama-3.1's 49.1% and 55.4%.

Main Findings

• Fine-tuned Llama-3.1-8B leads all baselines: 49.1% next-POI accuracy and 55.4% category accuracy on the 113-tourist test split.

• Statistical baselines perform poorly: first-order Markov reached 9.0%, a 2-state Hidden Markov Model reached 11.0%, and fifth-order Markov improved to 15.3% but suffered from data sparsity across 68 POIs.

• Fine-tuning matters more than model choice for GPT-4o: zero-shot GPT-4o achieved 18.7%, and fine-tuning via OpenAI's API raised this to 35.2%, still well below the fine-tuned Llama-3.1 result.

• A gap exists between category and POI prediction: the six-point gap between 55.4% category accuracy and 49.1% POI accuracy suggests general visitor intentions are predictable while specific choices among similar options retain inherent uncertainty.

• Sequence-level quality is reported: 31.2% 4-gram overlap, 25.8% BLEU, and 47.0% normalized Levenshtein distance, with generated trajectories averaging 7.2 POIs versus 7.7 in real data and anomalous outputs in only 5.4% of generations.

• Strong performance under undersampled rainy conditions: on 12 rainy test samples, the model maintained 41.7% accuracy versus 8.3% for Markov models, correctly shifting toward indoor predictions and away from garden visits.

• Temporal commonsense transfers: lunch-time predictions achieved 62.3% category accuracy for dining.

• The dataset's shape: the combined data averages 7.7 POI visits over 58 minutes, ranging from 2 to 40 POIs and 33 minutes to 5.6 hours.

Methodology in Plain English

The researchers treated trajectory prediction as a text generation problem. Each tourist is described by a persona (age, gender, group type) and an environment (weather, time), and each stop is written out as a line containing a timestamp, an action, an area, a category, and a POI name, wrapped in special tags. Because the model sees the whole sequence of prior stops in plain text, it does not need hand-engineered features describing weather or fatigue.

To collect the data, they combined two sources at Wakayama Castle Park: GPS traces from 87 participants, processed with stop detection (staying within 10 meters for over 60 seconds) and nearest-POI matching (within 25 meters), and QR code scan sequences from 479 stamp rally participants at 37 attractions, deduplicated and used directly. Demographics came from exit surveys, and the 18% of samples with missing attributes were inferred with GPT-4o from visit patterns and timestamps.

They then fine-tuned Llama-3.1-8B after adding all POI names, areas, categories, and structural tags to its vocabulary, initializing each new token's embedding by averaging the subword embeddings it was built from. Training used QLoRA with adapters on attention projections plus the embedding and output layers, the AdamW optimizer at a learning rate of 2e-5, batch size 8, and 10 epochs. The 566 trajectories were split 80% (453 tourists) for training and 20% (113 tourists) for testing with tourist-level separation to prevent leakage. Experiments ran on a server with two NVIDIA A6000 GPUs. At inference, the model is given the visit history up to the last area tag and asked to generate the following category and POI tags.

Why This Matters

Impact on research: The paper argues that fine-grained validation of LLM agents against real field-experiment behavior data has been limited, and that its results supply that evidence. It also offers a concrete data point against the assumption that language-specialized pretraining helps with physical movement prediction — the Japanese-specialized Llama-3-Swallow underperformed the general multilingual Llama-3.1 here.

Real-world applications:

• Mobility intervention planning: destination managers could test whether adding a shuttle service or e-bike availability to the context shifts predicted routes and dwell times before spending on infrastructure.
• Crowd redistribution at regional sites: the motivating problem is that popular landmarks are overcrowded while nearby culturally significant sites are underused, and the field site was chosen to connect an established southern castle area to newly developed northern facilities.
• Managing rare conditions: the rainy-day result (41.7% versus 8.3% for Markov models) addresses scenarios that are important for operations but too infrequent to learn from observation alone.
• Cross-destination adaptation: because context is expressed in natural language rather than fixed features, the same pipeline could be applied to other destinations with local fine-tuning.

Industry relevance: Tourism authorities and destination management organizations, mobility service operators, and urban planners facing investment decisions under uncertainty are the natural users. The 37 Green Slow Mobility pilot programs supported by Japan's Ministry of Land, Infrastructure, Transport and Tourism between 2019 and 2021 indicate the scale of public investment at stake.

Future Directions

1. Counterfactual generation and causal validation. The authors state explicitly that this work establishes predictive capability but does not validate causal claims, and propose generating counterfactuals by modifying input conditions (for example, adding shuttle service availability) and validating them against A/B test data or randomized field trials.

2. Handling entirely new POIs and major infrastructure changes. The model cannot predict behavior at POIs with no description and cannot reason about large infrastructure changes; the authors suggest architectural extensions such as retrieval-augmented generation or few-shot adaptation.

3. Reducing reliance on inferred attributes. Missing persona attributes (18%) were filled in with GPT-4o, which the authors note had minimal impact on manually verified subsets but remains a limitation.

4. Scaling from one destination to a validated behavior model. The paper frames the current model as a component for destination management tools, leaving open how predictive accuracy and generalization hold when the approach is deployed beyond Wakayama Castle Park.

Target Audience

Researchers and practitioners working on LLM agents, human mobility prediction, and next-POI modeling who want evidence from real field data rather than synthetic environments. It also suits urban planners and destination managers interested in how AI could support mobility investment decisions, and NLP or ML engineers looking for a concrete case study in extending a tokenizer vocabulary with domain-specific tokens and training QLoRA adapters on the embedding and output layers alongside attention projections. Readers primarily interested in causal inference should note the authors' own framing of this work as a predictive foundation rather than a causal analysis.

Authors’ abstract

Evaluating mobility interventions at tourist destinations requires predicting visitor behavior under varying conditions. Traditional methods struggle because tourist decisions depend heavily on context like weather and fatigue, yet models cannot generalize to unobserved scenarios. Large Language Models offer a solution by encoding commonsense knowledge about human behavior from pretraining, enabling reasoning about context-dependent decisions, while natural language representation flexibly integrates heterogeneous information. Fine-tuning on local trajectories adapts this general understanding to destination-specific patterns. We validate this approach using 566 trajectories from Wakayama Castle Park, Japan. Our fine-tuned Llama-3.1-8B achieves 49.1% next POI accuracy and maintains strong performance on undersampled scenarios like rainy days, demonstrating effective generalization. This establishes LLMs as high-fidelity behavior models for context-dependent tourist prediction, providing groundwork for counterfactual analysis of mobility interventions.