Misc minor suggestions for the model card

README.md

@@ -12,10 +12,6 @@ language:
 - en
 ---
 
----
-
----
-
 
 # Model Card for Backpack-GPT2
 
@@ -27,7 +23,7 @@ See also [backpackmodels.science](backpackmodels.science).
 
 ![A depiction of the Backpack language modeling process, in which each word in the sequence is weighted and summed to predict each word in context.](http://backpackmodels.science/assets/backpack-process.gif)
 
-#  Table of Contents
+##  Table of Contents
 
 - [Model Card for Backpack-GPT2](#model-card-for--model_id-)
 - [Table of Contents](#table-of-contents)
@@ -50,9 +46,9 @@ See also [backpackmodels.science](backpackmodels.science).
 - [How to Get Started with the Model](#how-to-get-started-with-the-model)
 
 
-# Model Details
+## Model Details
 
-## Model Description
+### Model Description
 
 <!-- Provide a longer summary of what this model is/does. -->
 The Backpack-GPT2 is a [Backpack-based language model](https://arxiv.org/abs/2305.16765), an architecture intended to combine strong modeling performance with an interface for interpretability and control.
@@ -66,45 +62,64 @@ The Backpack-GPT2 is a [Backpack-based language model](https://arxiv.org/abs/230
     - [GitHub Repo](https://github.com/john-hewitt/backpacks-flash-attn)
     - [Associated Paper](https://huggingface.co/datasets/openwebtext)
 
-# Uses
+## Uses
 
 This model is intended for use in the study and development of increasingly interpretable methods in natural language processing.
 It is not directly fit for any production use.
 
 
-# Bias, Risks, and Limitations
+## Bias, Risks, and Limitations
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
 Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
 This model in particular is limited in its capabilities, and with a brand new architecture, less is known about its biases than, e.g., Transformer-based models.
 
+## How to Get Started with the Model
+
+```python
+import torch
+from transformers import AutoConfig, AutoModelForCausalLM
+
+model_id = "stanfordnlp/backpack-gpt2"
+config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
+torch_model = AutoModelForCausalLM.from_pretrained(model_id, config=config, trust_remote_code=True)
+torch_model.eval()
+
+input = torch.randint(0, 50264, (1, 512), dtype=torch.long)
+torch_out = torch_model(
+    input,
+    position_ids=None,
+)
+torch_out = torch.nn.functional.softmax(torch_out.logits, dim=-1)
+print(torch_out)
+```
 
-# Training Details
+## Training Details
 
-## Training Data
+### Training Data
 
 <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
 This model was trained on the [OpenWebText](https://huggingface.co/datasets/openwebtext) corpus.
 
 
-## Training Procedure
+### Training Procedure
 
 This model was trained for 100k gradient steps with a batch size of 512k tokens and a linearly decaying learning rate from 6e-4 to zero, with a linear warmup of 5k steps.
 
-# Environmental Impact
+### Environmental Impact
 
 - **Hardware Type:** 4 A100 GPUs (40G)
 - **Hours used:** Roughly 4 days.
 - **Cloud Provider:** Stanford compute.
 - **Compute Region:** Stanford energy grid.
 
-## Model Architecture and Objective
+### Model Architecture and Objective
 
 This model was trained to minimize the cross-entropy loss, and is a [Backpack language model](https://arxiv.org/pdf/2305.16765.pdf).
 
-## Compute Infrastructure
+### Compute Infrastructure
 
 This model was trained on a slurm cluster.
 
@@ -116,7 +131,7 @@ This model was trained on 4 A100s.
 
 This model was trained with [FlashAttention](https://github.com/HazyResearch/flash-attention) and [PyTorch](https://pytorch.org/)
 
-# Citation
+## Citation
 
 **BibTeX:**
 
@@ -132,42 +147,14 @@ This model was trained with [FlashAttention](https://github.com/HazyResearch/fla
 ```
 
 
-# Model Card Authors [optional]
+## Model Card Authors [optional]
 
 <!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->
 
 John Hewitt
 
-# Model Card Contact
+## Model Card Contact
 
 [email protected]
 
-# How to Get Started with the Model
-
-```
-import torch
-import transformers
-from transformers import AutoModelForCausalLM
-
-
-model_id = "stanfordnlp/backpack-gpt2"
-config = transformers.AutoConfig.from_pretrained(model_id, trust_remote_code=True)
-torch_model = AutoModelForCausalLM.from_pretrained(model_id, config=config, trust_remote_code=True)
-torch_model.eval()
-
-input = torch.randint(0, 50264, (1, 512), dtype=torch.long)
-torch_out = torch_model(
-    input,
-    position_ids=None,
-)
-torch_out = torch.nn.functional.softmax(torch_out.logits, dim=-1)
-print(torch_out)
-```
-
-
-<details>
-<summary> Click to expand </summary>
-
-More information needed
 
-</details>