<jupyter_start><jupyter_text>Defined functions to read the csv files and write the data into PostgreSQL<jupyter_code>def copy_from_stringio(conn, df, table,schema):

    # save dataframe to an in memory buffer
    buffer = StringIO()
    df.to_csv(buffer, index_label='id', header=False)
    buffer.seek(0)
    cursor = conn.cursor()
    cursor.execute("SET search_path = "+schema)
    try:
        cursor.copy_from(buffer, table, sep=",")
        conn.commit()
    except (Exception, psycopg2.DatabaseError) as error:
        
        print("Error: %s" % error)
        conn.rollback()
        cursor.close()
        return 1
    print("copy_from_stringio() done")
    cursor.close()

def read_files(path,name_col,header_rows,key):
    df_set =pd.read_csv(path,names=name_col,skiprows=header_rows,quotechar='"')
    df_set.replace(',',' ', regex=True, inplace=True)
    df_set.set_index(keys=key,inplace=True)
    return df_set<jupyter_output><empty_output><jupyter_text>Read data from input files and stores in stage schema tables<jupyter_code>copy_from_stringio(conn,read_files(Pollution_path,Pollution_col,1,'Exposure_ID'),'stg_pollution','stage_analysis')
copy_from_stringio(conn,read_files(population_path,Population_col,1,'Country_Name'),'stg_population','stage_analysis')<jupyter_output>copy_from_stringio() done
copy_from_stringio() done
StarcoderdataJupyterStructuredCleanDedup