generated = original_model.predict(prompt, stop=["Observation:", "Human:"])
print(generated)
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
 'What's the difference between an iterator and an iterable?'
That’s not so impressive, is it? It didn’t follow the JSON format at all! Let’s try with the structured decoder.
JSONFormer LLM Wrapper#
Let’s try that again, now providing a the Action input’s JSON Schema to the model.
decoder_schema = {
    "title": "Decoding Schema",
    "type": "object",
    "properties": {
        "action": {"type": "string", "default": ask_star_coder.name},
        "action_input": {
            "type": "object",
            "properties": ask_star_coder.args,
        }
    }
} 
from langchain.experimental.llms import JsonFormer
json_former = JsonFormer(json_schema=decoder_schema, pipeline=hf_model)
results = json_former.predict(prompt, stop=["Observation:", "Human:"])
print(results)
{"action": "ask_star_coder", "action_input": {"query": "What's the difference between an iterator and an iter", "temperature": 0.0, "max_new_tokens": 50.0}}
Voila! Free of parsing errors.
previous
Huggingface TextGen Inference
next
Llama-cpp
 Contents
  
HuggingFace Baseline
JSONFormer LLM Wrapper
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on Jun 16, 2023.
rtdocs_stable/api.python.langchain.com/en/stable/modules/models/llms/integrations/jsonformer_experimental.html