Update README.md

README.md

@@ -1,61 +1,102 @@
----
-license: mit
-datasets:
-- carlosejimenez/wikitext__wikitext-2-raw-v1
-language:
-- en
-pipeline_tag: text-generation
-library_name: transformers
-tags:
-- torch
-- Tkinking
----
+license: mit  
+datasets:  
+- carlosejimenez/wikitext__wikitext-2-raw-v1  
+language:  
+- en  
+pipeline_tag: text-generation  
+library_name: transformers  
+tags:  
+- torch  
+- Tkinking  
 
 # Latent Recurrent Depth Language Model
 
-## Model Description
+## Overview
 
-This model is a Latent Recurrent Depth Language Model (LRD-LM), an experimental architecture designed for text generation.  It combines a "prelude" block for initial processing, a recurrent block with a latent state, and a "coda" block for final output.  The recurrent block allows for multiple iterations over the input sequence, potentially capturing deeper contextual information.
+The Latent Recurrent Depth Language Model (LRD-LM) is an experimental text-generation architecture designed to capture deeper contextual information through iterative, latent processing. Instead of generating verbose chain-of-thought sequences, LRD-LM refines its internal state over multiple recurrent iterations to improve text generation quality while keeping the parameter count modest.
 
+## Architecture
 
-## Intended Uses & Limitations
+The model is built around three key components:
 
-**Intended Uses:**
+- **Prelude Block:**  
+  This block handles the initial processing by embedding input tokens and applying self-attention with positional encodings.
+
+- **Recurrent Block:**  
+  A core, weight-shared block that iteratively refines a latent state. By repeatedly processing the prelude output along with its own evolving state, the model effectively “thinks” over the input without outputting intermediate tokens.
 
-* Text generation:  The primary purpose of this model is to generate text given a prompt.  It can potentially be fine-tuned for specific tasks like creative writing, code generation, or dialogue generation.
-* Research: This model serves as an exploration of novel architectures for language modeling, potentially leading to more effective methods for capturing long-range dependencies.
+- **Coda Block:**  
+  The final block decodes the refined latent state into output token probabilities.
+
+## Applications & Limitations
+
+**Intended Uses:**
+- **Text Generation:**  
+  Generate creative text, dialogue, code, or other natural language content.
+- **Research:**  
+  Serve as a testbed for exploring novel architectures and techniques in language modeling.
 
 **Limitations:**
+- **Data Constraints:**  
+  Trained on a small subset (first 1000 samples) of the Wikitext-2-raw-v1 dataset, which may limit its performance compared to models trained on larger corpora.
+- **Performance:**  
+  While it demonstrates the potential of latent recurrent depth, its overall performance is experimental and may not match state-of-the-art models.
+- **Computational Overhead:**  
+  The iterative processing introduces extra computation.
+- **Bias:**  
+  As with all language models, generated outputs may reflect biases present in the training data.
 
-* Data limitations: The model has been trained on a small subset of the Wikitext-2-raw dataset. Performance may be limited compared to models trained on larger, more diverse corpora.
-* Performance:  While the model demonstrates basic text generation capabilities, its overall performance is likely inferior to established state-of-the-art language models.  The provided training loop and hyperparameters are a starting point and may require significant adjustments for optimal results.
-* Computational cost: The iterative nature of the recurrent block can introduce computational overhead.
-* Bias: Like all language models, this model may exhibit biases present in its training data.
+## Training Details
 
+The model was fine-tuned on a subset of the Wikitext-2-raw-v1 dataset (first 1000 samples) using the AdamW optimizer and a cosine annealing learning rate scheduler. The training configuration and hyperparameters are provided in the accompanying code, and adjustments may be needed for improved performance.
 
-## Training Data
+## Usage
 
-The model was trained on a subset (first 1000 samples) of the Wikitext-2-raw-v1 dataset.  Further details regarding pre-processing and data cleaning can be found in the source code.  This limitation may reflect biases or inaccuracies in the generated output.
+The model can be used for text generation via its integrated `generate()` method, which allows you to control parameters such as the maximum sequence length, number of recurrent iterations, temperature, and top‑k filtering.
 
+### Example: Direct Inference
 
-## Evaluation Results
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-No formal evaluation metrics are provided at this time.  The model's performance is primarily demonstrated through qualitative assessment of generated samples during and after training.  Further evaluation using established metrics is recommended.
+# Load the model and tokenizer from the hub
+model = LatentRecurrentDepthModel.from_pretrained("codewithdark/latent-recurrent-depth-lm")
+tokenizer = AutoTokenizer.from_pretrained("codewithdark/latent-recurrent-depth-lm")
 
+prompt = "In the realm of language modeling"
+input_ids = tokenizer(prompt, return_tensors='pt').input_ids
 
-## Ethical Considerations
+# Generate logits using a specified number of recurrent iterations
+logits = model(input_ids, num_iterations=3)
 
-This model is provided for research and experimental purposes.  The user is responsible for ensuring ethical usage and mitigating potential risks associated with the generated output.
+# Sample from logits to produce generated text
+import torch
+probs = torch.softmax(logits[:, -1, :], dim=-1)
+next_token = torch.multinomial(probs, num_samples=1)
+generated_ids = torch.cat([input_ids, next_token], dim=1)
+generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
+print(generated_text)
+```
 
+### Alternative: Using the `generate()` Method
 
-## Model Usage Instructions
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
 
-The model can be used for text generation via the `generate()` method. The usage of this function is demonstrated in the example script.
+tokenizer = AutoTokenizer.from_pretrained("codewithdark/latent-recurrent-depth-lm")
+model = LatentRecurrentDepthModel.from_pretrained("codewithdark/latent-recurrent-depth-lm")
 
+prompt = "In the realm of language modeling"
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+generated_ids = model.generate(input_ids, max_length=50, num_iterations=3, temperature=0.8, top_k=50)
+generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
+print(generated_text)
+```
 
-## Training
+## Ethical Considerations
 
-The model has been trained using the AdamW optimizer and a cosine annealing learning rate scheduler for a number of epochs.  The training parameters can be configured in the provided script.
+This model is intended for research and experimental use. Users must ensure ethical application and carefully consider potential biases and misuse when deploying or further developing this technology.
 
+## License
 
-## Usage Example (Python)
+This project is licensed under the MIT License.