<jupyter_start><jupyter_text>Introduction to the VAPr package for the aggregation and analysis of genomic variant annotations  Author: <NAME>, <NAME>, <NAME>, <NAME> Contact Email: <EMAIL> Last Update: November 2017  BackgroundThe VAPr package retrieves variant information from [ANNOVAR](http://annovar.openbioinformatics.org/en/latest/) and [myvariant.info](myvariant.info) and consolidates it into a single local database for ease of use in investigating and filtering variant findings. The aggregated information is structured into lists of python dictionaries, with each variant being described by a multi-level dictionary. This approach flexibly accommodates the wide variety of information attributes available for different variants.  Further, this specific format permits its parsing to a MongoDb instance (dictionaries are the python representation of JSON objects), which enables the user to efficiently store, query and filter such data. Finally, the package also has the added functionality to create csv and vcf files from MongoDB. The built-in filters allow the user to rapidly query data that meets certain criteria as a list of documents, which can the be transformed into more widely accepted formats such as vcf and csv files. It should be noted that here, the main differential the package offers is the ability to write these files preserving all the annotation data. In the vcf files, for instance, outputs will have a 'Otherinfo' column where all the data coming from ANNOVAR and myvariant.info is condensed (while still preserving its structure). Having the data stored in a database offers a variety of benefits. In particular, it enables the user to set customized queries and rapidly iterate over a specific procedure and get maximum reproducibility. It also enables the storage of data coming from different sources, and its rapid access. **Notes on required software**the following libraries will be installed upon installing VAPr:- myvariant- pymongo- pyvcfOther libraries that are needed, but should natively e installed on most OS: - Pandas- NumpyFurther, a MongoDB database must be set up. Refer to the documentation page for more information. Similarly, ANNOVAR must be downloaded, alongside with its supporting databases (also listed on the documentation page).  Flowchart Diagram<jupyter_code>import os
from IPython.display import Image, display, HTML
Image(filename=os.path.dirname(os.path.realpath('__file__')) + '/simpler.jpg')<jupyter_output><empty_output><jupyter_text>Import libraries<jupyter_code>from VAPr import vapr_core
import os
IN_PATH = "/path/to/vcf"
OUT_PATH = "/path/to/out"
ANNOVAR_PATH = "/path/to/annovar"
MONGODB = 'VariantDatabase'
COLLECTION = 'CEU_trio_01012018'
annotator = vapr_core.VaprAnnotator(input_dir=IN_PATH,
                                    output_dir=OUT_PATH,
                                    mongo_db_name=MONGODB,
                                    mongo_collection_name=COLLECTION,
                                    build_ver='hg19',
                                    vcfs_gzipped=False,
                                    annovar_install_path=ANNOVAR_PATH)<jupyter_output><empty_output><jupyter_text>Downloading the Annovar Databases If you plan to use Annovar, please make sure to download the necessary Annovar databases. When Annovar is first installed, it does not install Annovar databases by default. The vapr_core has a method download_annovar_databases() that will download the necessary annovar databases.  Note: only run this command the first time that you use VAPr.<jupyter_code>annotator.download_annovar_databases()<jupyter_output><empty_output><jupyter_text>Run Annovar and MyVariant.info queries, upload variant annotations to MongoDBThe following command runs Annovar and processes the annotations, gets MyVariant.info annotations, merges the annotations by HGVS id into a JSON document, and uploads the documents to a MongoDB database. VAPr will also automatically include sample information from the VCF as well.<jupyter_code>dataset = annotator.annotate(num_processes=8)<jupyter_output><empty_output><jupyter_text>100%|██████████| 8/8 [00:17<00:00,  2.16s/it] Skip Annovar step, export only MyVariant data to MongoDBThis will export variant data from a vcf file to MongoDB, annotating variants solely with MyVariant.info. This method does not require annovar, and may result in some empty variant documents.<jupyter_code>dataset_light = annotator.annotate_lite(num_processes=8)<jupyter_output><empty_output><jupyter_text>Query Rare Deleterious VariantsYou will not have to instantiate a VaprDataset object if you still have your VaprAnnotator object. But for the purpose of demonstrating that the VaprDataset object will make the MongoDB connection to interact with your database, we will do so:<jupyter_code>dataset = vapr_core.VaprDataset(MONGODB, COLLECTION)
rare_deleterious_variants = dataset.get_rare_deleterious_variants()
rare_deleterious_variants[0]<jupyter_output><empty_output><jupyter_text>Notes on Variant Filtering & Output FilesHere we implement three different filters that allow for the retrieval of specific variants. The filters are implemented as MongoDB queries, and are designed to provide the user with a set of relevant variants. A template is provided for defining custom queries as well. The output of the queries is a list of dictionaries (JSON documents), where each dictionary contains a single variant document and its annotations. Further, the package allows the user to parse these variants into an annotated csv or vcf file. If needed, annotated, unfiltered vcf and csv files can also be created. They will have the same length (number of variants) as the original files, but will contain much more complete annotation data coming from myvariant.info and ANNOVAR databases.   Filter 1: Rare deleterious variants - criteria 1: 1000 Genomes (ALL) allele frequency (Annovar) < 0.05 or info not available - criteria 2: ESP6500 allele frequency (MyVariant.info - CADD) < 0.05 or info not available - criteria 3: cosmic70 (MyVariant.info) information is present - criteria 4: Func_knownGene (Annovar) is exonic, splicing, or both - criteria 5: ExonicFunc_knownGene (Annovar) is not "synonymous SNV"<jupyter_code># Apply filter.
rare_deleterious_variants = dataset.get_rare_deleterious_variants()<jupyter_output><empty_output><jupyter_text>Filter 2: Known disease variants- criteria: cosmic70 (MyVariant.info) information is present or ClinVar data is present and clinical significance is not Benign or Likely Benign<jupyter_code># Apply filter.
known_disease_variants = dataset.get_known_disease_variants()
len(known_disease_variants)<jupyter_output><empty_output><jupyter_text>Filter 3: Deleterious compound heterozygous variants- criteria 1: genotype_subclass_by_class (VAPr) is compound heterozygous- criteria 2: CADD phred score (MyVariant.info - CADD) > 10<jupyter_code># Apply filter
deleterious_compound_heterozygous = dataset.get_deleterious_compound_heterozygous_variants()<jupyter_output><empty_output><jupyter_text>Filter 4: De novo variants- criteria 1: Variant present in proband- criteria 2: Variant not present in either ancestor-1 or ancestor-2<jupyter_code># Apply filter.
denovo_variants = dataset.get_de_novo_variants(proband="NA12878",
                                               ancestor1="NA12891",
                                               ancestor2="NA12892")
denovo_variants[0]<jupyter_output><empty_output><jupyter_text>Create your own filter As long as you have a MongoDB instance running, filtering can be perfomed through pymongo as shown by the code below. If a list is intended to be created from it, simply add: `filter = list(filter)`If you'd like to customize your filters, a good idea would be to look at the available fields to be filtered. Looking at the myvariant.info [documentation](http://docs.myvariant.info/en/latest/doc/data.html), you can see what are all the fields avaialble and can be used for filtering.<jupyter_code>from pymongo import MongoClient

client = MongoClient()
db = getattr(client, MONGODB)
collection = getattr(db, COLLECTION)

filtered = collection.find({"$and": [
                                   {"$or": [{"func_knowngene": "exonic"},
                                            {"func_knowngene": "splicing"}]},
                                   {"cosmic70": {"$exists": True}},
                                   {"1000g2015aug_all": {"$lt": 0.05}}
                         ]})
filtered = list(filtered)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup