<jupyter_start><jupyter_text>SQuAD 2.0 DatasetThe SQuAD (Stanford Question and Answering Dataset) is a hugely popular dataset containing question and answer pairs scraped from Wikipedia, covering topics ranging from Beyonce, to Physics. As one of the most comprehensive Q&A datasets available, it's only natural that we will be making use of it. So let's explore it.First, we'll need to download the data. There are two JSON files that we are interested in - train and dev, which we can downloaded from `http`. Here we will be storing the SQuAD data in the *../../data/squad* directory, so we must check if this already exists and if not create the directory.<jupyter_code>import os

squad_dir = '../../data/squad'

if not os.path.exists(squad_dir):
    os.mkdir(squad_dir)<jupyter_output><empty_output><jupyter_text>Now let's define our SQuAD URL and files.<jupyter_code>url = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/'
files = ['train-v2.0.json', 'dev-v2.0.json']<jupyter_output><empty_output><jupyter_text>And now we can download and write both files to our *../../squad* directory.<jupyter_code>import requests

for file in files:
    res = requests.get(url+file)
    # write to file in chunks
    with open(os.path.join(squad_dir, file), 'wb') as f:
        for chunk in res.iter_content(chunk_size=40):
            f.write(chunk)<jupyter_output><empty_output><jupyter_text>Now that we have both files stored locally, lets open up the training data and see what we have.<jupyter_code>import json

with open(os.path.join(squad_dir, 'train-v2.0.json'), 'rb') as f:
    squad = json.load(f)<jupyter_output><empty_output><jupyter_text>The JSON structure contains a top-level `'data'` key which contains a list of *groups*, where each group is a topic, such as *Beyonce*, *Chopin*, or *Matter*. We can take a look at the first and last groups respectively.<jupyter_code>squad['data'][0]
squad['data'][0]['paragraphs'][0]
squad['data'][-1]['paragraphs'][0]<jupyter_output><empty_output><jupyter_text>If we compare the first entry on *Beyonce* and the second on *Matter*, we can see that we sometimes return our answers in the `'answers'` key, and sometimes in the `'plausible_answers'` key. So when processing this data we will need to consider some additional logic to deal with this.Secondly, for all samples, we need to iterate through multiple levels. On the highest level we have *groups*, which is where our topics like *'Beyonce'* and *'Matter'* belong. At the next layer we have paragraphs, and in the next we have our question-answer pairs, this structure looks like this:![train-v2.0.json structure](../../assets/images/squad_train_json_structure.png)*(You can check out this structure by opening the **train-v2.0.json** in Jupyter)*We'll work through parsing this data into a cleaner format that we will be using in later notebooks. We need to create a format that consists of a list of dictionaries where each dictionary contains a single `question`, `answer`, and `context`.<jupyter_code># initialize list where we will place all of our data
new_squad = []

# we need to loop through groups -> paragraphs -> qa_pairs
for group in squad['data']:
    for paragraph in group['paragraphs']:
        # we pull out the context from here
        context = paragraph['context']
        for qa_pair in paragraph['qas']:
            # we pull out the question
            question = qa_pair['question']
            # now the logic to check if we have 'answers' or 'plausible_answers'
            if 'answers' in qa_pair.keys() and len(qa_pair['answers']) > 0:
                answer = qa_pair['answers'][0]['text']
            elif 'plausible_answers' in qa_pair.keys() and len(qa_pair['plausible_answers']) > 0:
                answer = qa_pair['plausible_answers'][0]['text']
            else:
                # this shouldn't happen, but just in case we just set answer = None
                answer = None
            # append dictionary sample to parsed squad
            new_squad.append({
                'question': question,
                'answer': answer,
                'context': context
            })
new_squad[:2], new_squad[-2:]<jupyter_output><empty_output><jupyter_text>We can save our parsed data to file as a JSON like so:<jupyter_code>with open(os.path.join(squad_dir, 'train.json'), 'w') as f:
    json.dump(new_squad, f)<jupyter_output><empty_output><jupyter_text>* * * Processing dev data<jupyter_code>with open(os.path.join (squad_dir, files[1]), 'rb') as f:
    squad = json.load(f)
squad['data'][0]['paragraphs'][0]['qas'][3]['answers'][0]['text']
# initialize list where we will place all of our data
new_squad = []

# we need to loop through groups -> paragraphs -> qa_pairs
for group in squad['data']:
    for paragraph in group['paragraphs']:
        # we pull out the context from here
        context = paragraph['context']
        for qa_pair in paragraph['qas']:
            # we pull out the question
            question = qa_pair['question']
            # now the logic to check if we have 'answers' or 'plausible_answers'
            if 'answers' in qa_pair.keys() and len(qa_pair['answers']) > 0:
                answer_list = qa_pair['answers']
            elif 'plausible_answers' in qa_pair.keys() and len(qa_pair['plausible_answers']) > 0:
                answer_list = qa_pair['plausible_answers']
            else:
                # this shouldn't happen, but just in case we just set answer = []
                answer_list = []
            # we want to pull our the 'text' of each answer in our list of answers
            answer_list = [item['text'] for item in answer_list]
            # we can remove duplicate answers by converting our list to a set, and then back to a list
            answer_list = list(set(answer_list))
            # we iterate through each unique answer in the answer_list
            for answer in answer_list:
                # append dictionary sample to parsed squad
                new_squad.append({
                    'question': question,
                    'answer': answer,
                    'context': context
                })
new_squad[10:15]
with open(os.path.join(squad_dir, 'dev.json'), 'w') as f:
    json.dump(new_squad, f)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup