<jupyter_start><jupyter_text>Approach :1. Filter 'all_transcript_data.csv' having type as lnc_RNA. Save the RefSeq Gene IDs and RefSeq Transcript ID columns.2. Filter 'all_gene_data.csv' whose gene IDs lie in the 'RefSeq Gene ID' collected in point 1 above.3. Filter 'all_exon_data.csv' whose transcript IDs lie in the 'RefSeq Transcript ID' collected in point 1 above.<jupyter_code>import pandas as pd
df = pd.read_csv('/refseq_numeric/all_gene_data/all_transcript_data_corrected.csv', index_col=0)
total_exons = df['No. of Exons'].tolist()
sum(total_exons)
df.head(8)
df.drop('SNO', axis=1,inplace=True)
df.head(8)
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>Below code for getting all transcript data having type as lnc_RNA<jupyter_code>%%time
refseq_gene_ids = []
refseq_transcript_ids = []
lncrna_transcript_index_to_drop = []
for i in tqdm(range(len(df[df.columns[1]]))):
    if df[df.columns[6]][i] != 'lnc_RNA':
        lncrna_transcript_index_to_drop.append(i)
    elif df[df.columns[6]][i] == 'lnc_RNA':
        refseq_gene_ids.append(df[df.columns[1]][i])
        refseq_transcript_ids.append(df[df.columns[2]][i])
df.drop(lncrna_transcript_index_to_drop, axis=0,inplace=True)
s_no = []
for i in range(len(df[df.columns[1]])):
    s_no.append(i+1)
df['SNO'] = s_no
df = df[['SNO','Index', 'RefSeq Gene ID','RefSeq Transcript ID', 'Dbxref', 'Parent', 'gbkey', 'Type', 'Seq_id', 'Strand','Start', 'End','No. of Exons','Transcript Length']]     
df.reset_index(drop=True, inplace=True)
df.head(5)
len(df[df.columns[1]])
len(refseq_gene_ids)
df.to_csv('/refseq_numeric/all_type=lncRNA_data/all_lncrna_transcript_data_corrected.csv')<jupyter_output><empty_output><jupyter_text>gene data lncrna<jupyter_code>df = pd.read_csv('/dataset/refseq_numeric/all_gene_data/all_gene_data.csv', index_col=0)
df.drop('SNO', axis=1,inplace=True)
df.head(5)
# refseq_gene_ids[0:10]
%%time
s = set(refseq_gene_ids)
drop_values = []
for i in tqdm(range(len(df[df.columns[1]]))):
    if df[df.columns[1]][i] in s:
        continue
    else:
        drop_values.append(i)
df.drop(drop_values, axis=0,inplace=True)
s_no = []
for i in range(len(df[df.columns[1]])):
    s_no.append(i+1)

df['SNO'] = s_no
df.head(2)
df = df[['SNO','Index', 'RefSeq Gene ID', 'Dbxref','Name', 'gbkey', 'Biotype', 'Type', 'Seq_id', 'Strand','Start', 'End', 'No. of Transcripts']]     
df.reset_index(drop=True, inplace=True)
df.head(5)
len(df[df.columns[1]])
df.to_csv('/refseq_numeric/all_type=lncRNA_data/all_lncrna_gene_data.csv')<jupyter_output><empty_output><jupyter_text>Exon data lncrna below<jupyter_code>import pandas as pd
df = pd.read_csv('refseq_numeric/all_gene_data/all_exon_data.csv', index_col=0)
df.drop('SNO', axis=1,inplace=True)
df.head(5)<jupyter_output><empty_output><jupyter_text>Filtering exon data having refseq IDs collected in above cells<jupyter_code>%%time
s = set(refseq_transcript_ids)
drop_values = []
for i in tqdm(range(len(df[df.columns[1]]))):
    if df[df.columns[2]][i] in s:
        continue
    else:
        drop_values.append(i)
df.drop(drop_values, axis=0,inplace=True)
# len(df[df.columns[1]])
s_no = []
for i in range(len(df[df.columns[1]])):
    s_no.append(i+1)
df['SNO'] = s_no
df = df[['SNO','Index', 'RefSeq Gene ID','RefSeq Transcript ID','RefSeq Exon ID','Dbxref', 'Parent', 'gbkey', 'Type', 'Seq_id', 'Strand','Start', 'End','Exon Length']]     
df.reset_index(drop=True, inplace=True)
df.head(13)
df.to_csv('/all_type=lncRNA_data/all_lncrna_exon_data.csv')<jupyter_output><empty_output><jupyter_text>Once you load the csv files, do not forget to reset the index values ! [ df.reset_index(drop=True, inplace=True) ]. Default index values are not sequential (Since they got dropped)  Correcting Exon data. Removing all those refseq exon IDs which start with 'id-'<jupyter_code>import pandas as pd
df = pd.read_csv('all_type=lncRNA_data/all_lncrna_exon_data.csv', index_col=0)
df.drop('SNO', axis=1,inplace=True)
df.head(5)
len(df['Index'])
exon = 0
ID = 0
drop_index = []
O = 0
for i in tqdm(range(len(df['Index']))):
    if df['RefSeq Exon ID'][i][0:6] == 'exon-X' or df['RefSeq Exon ID'][i][0:6] == 'exon-N':
        exon = exon + 1
#     elif df['RefSeq Exon ID'][i][0:2] == 'id':
#         ID = ID + 1
#         drop_index.append(i)
    else:
        drop_index.append(i)
        O = O + 1
print(exon)
print(ID)
print(len(drop_index))
print(O)
df.drop(drop_index, axis=0,inplace=True)
s_no = []
for i in range(len(df[df.columns[1]])):
    s_no.append(i+1)

df['SNO'] = s_no
df = df[['SNO','Index', 'RefSeq Gene ID','RefSeq Transcript ID','RefSeq Exon ID','Dbxref', 'Parent', 'gbkey', 'Type', 'Seq_id', 'Strand','Start', 'End','Exon Length']]  

df.reset_index(drop=True, inplace=True)
df.head(5)
len(df[df.columns[1]])

df.to_csv('/all_type=lncRNA_data/all_lncrna_exon_data_corrected.csv')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup