<jupyter_start><jupyter_text>Пример использования модели проекта opus-mt<jupyter_code># !pip install -q transformers[sentencepiece]
# !pip install -q datasets
# !pip install -q sacrebleu
# !pip uninstall -y tqdm
# !pip install -q tqdm
import pandas as pd
import numpy as np
import seaborn as sns 
import matplotlib.pyplot as plt
from datasets import Dataset,load_metric
from sklearn.model_selection import train_test_split
import torch
from tqdm.auto import tqdm
from transformers import DataCollatorForSeq2Seq, MarianConfig, MarianMTModel, MarianTokenizer,  AdamW
checkpoint = 'Helsinki-NLP/opus-mt-en-ru' # предобученная модель EN-RU<jupyter_output><empty_output><jupyter_text>Загрузка модели и токенизатора<jupyter_code>model = MarianMTModel.from_pretrained(checkpoint)
tokenizer = MarianTokenizer.from_pretrained(checkpoint)<jupyter_output><empty_output><jupyter_text>Использование полученной модели для перевода<jupyter_code>def translate(text='Hello, World!'):
    input_ids = tokenizer(text, return_tensors="pt").input_ids # Batch size 1
    output_ids = model.generate(input_ids)[0]
    output = tokenizer.decode(output_ids, skip_special_tokens=True)
    return output
translate('Security of online payments')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup