Emma02/LVM_ckpts

TEXT GENERATIONPricing:Input $0.4 / Cached $0.08 / Output $0.8Concurrent Unit Cost:1Model Size:7BQuant:FP8Context Size:4kPublished:Jun 13, 2024License:apache-2.0Architecture:Transformer0.0K Open Weights Featherless Exclusive Cold

Emma02's LVM (Large Vision Model) is a 7 billion parameter vision pretraining model that processes visual data by converting it into visual sentences and performing autoregressive next-token prediction. Based on OpenLLaMA and OpenMuse, it is designed for scalable learning in large vision models, as detailed in its CVPR 2024 paper. This model excels at sequential modeling of visual information and is compatible with both GPU and TPU environments. It features an improved tokenizer and a deeply filtered LAION dataset for enhanced performance.

Loading preview...

Overview

Emma02's LVM (Large Vision Model) is a 7 billion parameter vision pretraining model developed in collaboration with HuggingFace. It implements the approach described in the CVPR 2024 paper 'Sequential Modeling Enables Scalable Learning for Large Vision Models' (arXiv:2312.00785). LVM converts diverse visual data into "visual sentences" and then applies autoregressive next-token prediction, enabling scalable learning for large vision models.

Key Capabilities & Features

  • Vision Pretraining: Specializes in pretraining on various visual data types.
  • Sequential Modeling: Processes visual information sequentially, treating it as visual sentences.
  • Autoregressive Prediction: Utilizes next-token prediction for learning, similar to large language models.
  • Scalable Learning: Designed for efficient and scalable training of large vision models.
  • Hardware Compatibility: Supports both GPU and TPU for training and inference.
  • Improved Tokenizer: Features an enhanced tokenizer for better performance compared to earlier versions.
  • Refined Dataset: Trained on a deeply filtered version of the LAION dataset (1.2 billion images), ensuring higher quality data by removing low-quality, deduplicated, and known CSAM content.
  • Open-Source Foundation: Built upon the OpenLLaMA autoregressive model and the OpenMuse VQGAN for visual tokenization.

Use Cases

  • Research in Large Vision Models: Ideal for researchers exploring scalable learning and sequential modeling in computer vision.
  • Visual Data Understanding: Can be used as a foundational model for tasks requiring deep understanding of visual sequences.
  • Next-Token Prediction for Vision: Applicable in scenarios where predicting subsequent visual elements or patterns is crucial.

Key Differences from Original Paper Version

  • Model Size: The currently released version is 7B parameters, an increase from the previously mentioned 3B model.
  • Dataset Quality: The LAION dataset used for training has undergone deep filtering, reducing its size from 1.5B to 1.2B images while improving data quality.
  • Tokenizer Enhancements: The model incorporates an improved tokenizer for better overall performance.

Finetunes
1 models