|
|
--- |
|
|
license: other |
|
|
license_name: custom-apple-license |
|
|
license_link: https://github.com/apple/ml-tic-clip/blob/main/LICENSE |
|
|
tags: |
|
|
- vision |
|
|
- zero-shot-image-classification |
|
|
datasets: |
|
|
- apple/TiC-DataComp |
|
|
--- |
|
|
# Model Card for Model ID |
|
|
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
|
|
This repository contains TiC-CLIP models trained on TiC-DataComp-Yearly with data from 2014 to 2022 using our modified OpenCLIP code. |
|
|
For additional information refer to our [GitHub repo](https://github.com/apple/ml-tic-clip). |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
Keeping large foundation models up to date on latest data is inherently expensive. |
|
|
To avoid the prohibitive costs of constantly retraining, it is imperative to continually train these models. |
|
|
This problem is exacerbated by the lack of any large scale continual learning benchmarks or baselines. |
|
|
We introduce the first set of web-scale Time-Continual (TiC) benchmarks for training vision-language models: |
|
|
TiC-DataComp, TiC-YFCC, and TiC-Redcaps. TiC-DataComp, our largest dataset, |
|
|
contains over 12.7B timestamped image-text pairs spanning 9 years (2014-2022). |
|
|
We first use our benchmarks to curate various dynamic evaluations to measure temporal robustness of existing models. |
|
|
We show OpenAI's CLIP (trained on data up to 2020) loses ≈8% zero-shot accuracy on our curated retrieval task from 2021-2022 compared with more recently trained models in OpenCLIP repository. |
|
|
We then study how to efficiently train models on time-continuous data. |
|
|
We demonstrate that a simple rehearsal-based approach that continues training from the last checkpoint and replays old data reduces compute by 2.5× when compared to the standard practice of retraining from scratch. |
|
|
Code is available at [this https URL](https://github.com/apple/ml-tic-clip). |
|
|
|
|
|
|
|
|
|
|
|
- **Developed by:** Apple |
|
|
- **License:** See [LICENSE](https://github.com/apple/ml-tic-clip/blob/main/LICENSE) |
|
|
|
|
|
### Model Sources [optional] |
|
|
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
|
|
- **Repository:** [ml-tic-clip GitHub repo](https://github.com/apple/ml-tic-clip) |
|
|
- **Paper:** [TiC-CLIP: Continual Training of CLIP Models, Garg, S., Farajtabar, M., Pouransari, H., Vemulapalli, R., Mehta, S., Tuzel, O., Shankar, V. and Faghri, F., International Conference on Learning Representations (ICLR), 2024.](https://arxiv.org/abs/2310.16226) |
|
|
|
|
|
## Uses |
|
|
|
|
|
Researchers can use TiC-CLIP pretrained models for faster design of continual learning methods by start from a pretrained checkpoint and continually train on the next year or next month data. |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
The models are compatible with DataComp evaluation suite and our patched version of DataComp for evaluation on TiC-DataComp-Retrieval and TiC-DataCompNet. |
|
|
The models can also be used to resume a training or as initialization for new training using OpenCLIP code. |
|
|
Please follow instructions in our [GitHub repo](https://github.com/apple/ml-tic-clip) to create the evaluation sets or follow [DataComp](https://github.com/mlfoundations/datacomp) for the standard evaluations on 38 datasets. |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
### Training Procedure |
|
|
|
|
|
Please refer to Sections 2-3 of our [TiC-CLIP](https://github.com/apple/ml-tic-clip) paper. |
|
|
|
|
|
#### Preprocessing [optional] |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
|
|
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
|
|
#### Testing Data |
|
|
|
|
|
<!-- This should link to a Dataset Card if possible. --> |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
#### Metrics |
|
|
|
|
|
<!-- These are the evaluation metrics being used, ideally with a description of why. --> |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
### Results |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
#### Summary |
|
|
|
|
|
|
|
|
|
|
|
## Environmental Impact |
|
|
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
|
|
- **Hardware Type:** [More Information Needed] |
|
|
- **Hours used:** [More Information Needed] |
|
|
- **Carbon Emitted:** [More Information Needed] |
|
|
|
|
|
## Technical Specifications [optional] |
|
|
|
|
|
### Model Architecture and Objective |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
### Compute Infrastructure |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
#### Hardware |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
#### Software |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
## Citation [optional] |
|
|
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
|
|
**BibTeX:** |
|
|
|
|
|
[More Information Needed] |
|
|
|
