<jupyter_start><jupyter_text>This notebook is developed using the `Python 3 (Data Science)` kernel on an `ml.t3.medium` instance. Downloading SQuAD-v2 from source<jupyter_code>import os
import sagemaker
sess = sagemaker.Session()
bucket = sess.default_bucket()
prefix = 'sagemaker-studio-book/chapter08'

local_prefix='buddhism'
os.makedirs(local_prefix, exist_ok=True)
!wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json -P {local_prefix}
import json
with open(f'{local_prefix}/train-v2.0.json') as f:
    squad_train = json.load(f)<jupyter_output><empty_output><jupyter_text>Fine tune with questions related to BuddhismExtract Buddhism from titles<jupyter_code>title_of_interest = [(i, j['title']) 
                     for i, j in enumerate(squad_train['data']) 
                     if j['title'] == 'Buddhism']
title_of_interest<jupyter_output><empty_output><jupyter_text>Buddhism is located at index 11. Take a look at the dictionary.<jupyter_code>squad_train['data'][title_of_interest[0][0]]['paragraphs'][0]<jupyter_output><empty_output><jupyter_text>Organize the `data.csv` Below is the requirement for the finetune dataset from the instruction page.>Input: A directory containing a 'data.csv' file.>- The first column of the 'data.csv' should have a question.>- The second column should have the corresponding context.>- The third column should have the integer character starting position for the answer in the context.>- The fourth column should have the integer character ending position for the answer in the context.The following nested for loop will go through each context, question and answer.<jupyter_code>rows = []
for paragraph in squad_train['data'][title_of_interest[0][0]]['paragraphs']:
    context = paragraph['context']
    for qas in paragraph['qas']:
        question = qas['question']
        for answer in qas['answers']:
            answer_text = answer['text']
            answer_start = answer['answer_start']
            answer_end = answer_start + len(answer_text) - 1
            rows.append([question, context, answer_start, answer_end])<jupyter_output><empty_output><jupyter_text>Saving the `rows` into a csv<jupyter_code>import csv

with open(f'{local_prefix}/data.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    writer.writerows(rows)
!head -n2 data.csv<jupyter_output><empty_output><jupyter_text>Uploading the `data.csv` to a S3 bucket<jupyter_code>sagemaker.s3.S3Uploader.upload(local_path=f'{local_prefix}/data.csv',
                               desired_s3_uri=f's3://{bucket}/{prefix}/{local_prefix}',
                               sagemaker_session=sess)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup