X1AOX1A/WorldModel-Webshop-Llama3.1-8B

WorldModel-Webshop-Llama3.1-8B: Implicit Text-based World Model

This model, developed by X1AOX1A, is an 8 billion parameter language model fine-tuned from the powerful Meta-Llama-3.1-8B architecture. It is specifically adapted for tasks within webshop environments, aiming to explore the concept of Large Language Models (LLMs) as implicit text-based world models.

Key Capabilities

• Webshop Interaction: Fine-tuned on the webshop_train_70790 dataset, this model is designed to understand and interact with web-based interfaces, particularly in e-commerce settings.
• Agentic Behavior: It serves as a foundation for research into how LLMs can unlock scalable agentic reinforcement learning by implicitly modeling the "world" described through text.
• Llama 3.1 Foundation: Benefits from the robust capabilities and extensive pre-training of the Meta-Llama-3.1-8B base model.

Training Details

The model was trained with a learning rate of 1e-05, a total batch size of 128 (achieved with 4 GPUs and 32 gradient accumulation steps), and for 5 epochs. The training utilized ADAMW_TORCH optimizer and a constant_with_warmup learning rate scheduler.

Research Context

This model is part of a broader research initiative titled "From Word to World: Can Large Language Models be Implicit Text-based World Models?" as detailed in the associated arXiv paper. It contributes to understanding how LLMs can build internal representations of environments from textual observations.