---
language:
- en
license: apache-2.0
tags:
- causal-lm
- point-in-time
- temporal-llm
- pretrained
---

# PIT-1B — Point-In-Time GPT (Pre-trained, 2016-12)

**Point-In-Time (PIT)** is a family of GPT-style language models trained on
chronologically-ordered monthly snapshots of [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb).
Each checkpoint captures the state of knowledge available up to a specific month,
making them suitable for temporal reasoning and point-in-time analysis tasks.

This is the **base** (pre-trained only) variant.

## Model details

| Property | Value |
|----------|-------|
| **Snapshot month** | 2016-12 |
| **Architecture** | Decoder-only Transformer (GPT) |
| **Layers** | 52 |
| **Hidden dim** | 1536 |
| **Attention heads** | 12 |
| **Vocab size** | 50304 |
| **Tokenizer** | GPT-2 BPE |
| **Position encoding** | RoPE |
| **Normalization** | RMSNorm on Q/K + pre-norm |
| **Activation** | Squared ReLU |
| **Weight tying** | Yes (input emb ↔ lm\_head) |

## Requirements

```bash
pip install transformers torch safetensors
```

## Quick start

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Diamegs/PIT-1B-201612"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,   # required for custom architecture
    torch_dtype=torch.bfloat16,
)
model = model.cuda()
model.eval()
```

## Text generation

```python
prompt = "In 2016, the global economy"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id,
)
n_prompt = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][n_prompt:], skip_special_tokens=True))
```

## Temporal reasoning example

Because this model was trained on data up to **2016-12**, it reflects the world
as it was known at that point. You can use this for point-in-time analysis:

```python
# What does the model "know" about events before its cutoff?
prompt = "The most important AI developments in early 2016 were"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    pad_token_id=tokenizer.eos_token_id,
)
n_prompt = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][n_prompt:], skip_special_tokens=True))
```

## Weights format

Weights are stored in [safetensors](https://huggingface.co/docs/safetensors) format
(`model.safetensors`) — memory-mapped, fast to load, and safe (no arbitrary code execution).

## Limitations

- Knowledge is limited to web text available up to **2016-12**.
- No RLHF or safety fine-tuning has been applied (base model).
- The model may reproduce biases present in FineWeb training data.
- Not suitable for safety-critical applications without further alignment.

## Citation

If you use these models in your research, please cite our paper:

```bibtex
@techreport{kelly2026pit,
  title       = {Scaling Point-in-Time Language Models},
  author      = {Kelly, Bryan T. and Malamud, Semyon and Schwab, Johannes and Xu, Teng Andrea},
  institution = {Swiss Finance Institute},
  type        = {Research Paper},
  number      = {26-37},
  year        = {2026},
  month       = apr,
  doi         = {10.2139/ssrn.6681860},
  url         = {https://ssrn.com/abstract=6681860}
}
```