<jupyter_start><jupyter_text>What are the most common programming languages used in Brazil?? A data analysis using Stackoverflow’s 2017 and 2018 Annual Developer Survey.   Introduction With the advent of new areas in Brazil such as Artificial Intelligence and Data Science, many programming languages that were poorly spoken and used are now experiencing a rise in popularity. You may have already heard **“Python is the programming language of the moment”** or **“Functional Paradigm is much better than Object Oriented”**. In this article, we are going to analyze real data to verify if these programming languages are really being used Brazil or if they are just rumors. For this, we are going to use data from Stackoverflow’s 2017 and 2018 Annual Developer Survey. Every year, Stack Overflow conducts a massive survey of people on the site, covering all sorts of information like programming languages, jobs, code style and various other information.  Every year, Stack Overflow conducts a massive survey of people on it’s website, covering all sorts of information like programming languages, jobs, code style and more. There were more than 150 questions as a part of the survey, including:**“What frameworks do you work with?”** **“Do you program as a hobby or contribute to open source projects?”** **“What IDE do you work with?”**<jupyter_code># to handle datasets
import numpy as np
import pandas as pd
from collections import Counter

# to visualise al the columns in the dataframe
pd.pandas.set_option('display.max_columns', None)

# to make plots
import matplotlib.pyplot as plt
import seaborn as sns

# to change plot style
sns.set(style="darkgrid")

# to ignore warnings
import warnings
warnings.filterwarnings("ignore")
# Reading Stackoverflow survey data
df_2017 = pd.read_csv('data/2017/survey_results_public.csv')
df_2018 = pd.read_csv('data/2018/survey_results_public.csv')
# Sample 2017 Data
df_2017.head()
# Sample 2018 Data
df_2018.head()<jupyter_output><empty_output><jupyter_text>1. What programming languages are most used to work and which programming languages are most required?<jupyter_code>def filter_country(df, column_filter, country, columns):
    ''' Filter a dataframe by country.,
        returns filtred dataframe.
    '''
    
    df_copy = df
    
    for column in columns: 
        df_copy = df_copy[df_copy[column_filter] == country].dropna(subset=[column])
        
    return df_copy

# Filter the dataframe by Brazil and selecting some columns.
brazil_2017 = filter_country(df_2017, 'Country', 'Brazil', ['HaveWorkedLanguage', 'WantWorkLanguage'])
brazil_2018 = filter_country(df_2018, 'Country', 'Brazil', ['LanguageWorkedWith', 'LanguageDesireNextYear'])
def split_column(df, column):
    ''' Split column by ;,
        returns a splited series.
    '''
    
    df_copy = df
    column_series = df_copy[column].apply(lambda x: x.split(';'))
    
    return column_series

# Splitting the Data Frame by column into a Series. 
worked_lang_2017 = split_column(brazil_2017, 'HaveWorkedLanguage')
wanted_lang_2017 = split_column(brazil_2017, 'WantWorkLanguage')

worked_lang_2018 = split_column(brazil_2018, 'LanguageWorkedWith')
wanted_lang_2018 = split_column(brazil_2018, 'LanguageDesireNextYear')
def disarray(array_list):
    ''' Flat a nested list,
        returns a flat list.
    '''
    
    objects = []
    
    for row in array_list:
        for obj in row:
            objects.append(obj.strip())
            
    return objects

# Flatting nested list objects.
list_worked_languages_2017 = disarray(worked_lang_2017)
list_wanted_languages_2017 = disarray(wanted_lang_2017)

list_worked_languages_2018 = disarray(worked_lang_2018)
list_wanted_languages_2018 = disarray(wanted_lang_2018)
def group_list(data_list, year):
    ''' Group by count to a list,
        returns a result dict.
    '''
    
    grouped_list = dict(Counter(data_list))
    grouped_dict = [{'Programming Language':key, 'Count': value, 'Year': year} for key, value in grouped_list.items()]
    
    return grouped_dict

# Groping a list and creating a dict.
dict_worked_languages_2017 = group_list(list_worked_languages_2017, '2017')
dict_wanted_languages_2017 = group_list(list_wanted_languages_2017, '2017')

dict_worked_languages_2018 = group_list(list_worked_languages_2018, '2018')
dict_wanted_languages_2018 = group_list(list_wanted_languages_2018, '2018')
def create_dataframe(data_dicts):
    ''' Create two dataframes and append them,
        returns a appended dataframe.
    '''
        
    df1 = pd.DataFrame(data_dicts[0])
    df2 = pd.DataFrame(data_dicts[1])

    df = df1.append(df2)
    
    return df

# Creating a Data Frame appending the data.
worked_languages = create_dataframe([dict_worked_languages_2017, dict_worked_languages_2018])
wanted_languages = create_dataframe([dict_wanted_languages_2017, dict_wanted_languages_2018])
def percentage(df, column):
    ''' Scale data,
        returns data scaled.
    '''
    
    df_copy = df
    series = []
    
    for val in df_copy[column].unique():
        series.append(df_copy[df_copy[column] == val]['Count'] / 
                      df_copy[df_copy[column] == val]['Count'].sum())
    
    joined_series = pd.Series()
    for i_series in series:
        joined_series = joined_series.append(i_series)
        
    return joined_series

# Calculating the percentage
worked_languages['Percentage'] = percentage(worked_languages, 'Year')
wanted_languages['Percentage'] = percentage(wanted_languages, 'Year')
# Get the top 10 languages
top_10_worked_lang = worked_languages.sort_values(by=['Percentage'], ascending=False).head(10)['Programming Language'].unique()
top_10_wanted_lang = wanted_languages.sort_values(by=['Percentage'], ascending=False).head(10)['Programming Language'].unique()

worked_lang_chart = worked_languages[worked_languages['Programming Language'].isin(top_10_worked_lang)]
wanted_lang_chart = wanted_languages[wanted_languages['Programming Language'].isin(top_10_worked_lang)]<jupyter_output><empty_output><jupyter_text>We can see that classic languages like: **Java Script**, **SQL**, **Java** are still in the top positions. Stack Overflow 2017 survey data and Stack Overflow 2018 survey data We can see that in 2018 two programming languages have risen such as **HTML** and **CSS**, but these programming languages are old and commonly used, this probably happened because in the StackOverflow Survey of 2017 it was probably not possible to select these two languages as option.<jupyter_code># Figure 1 — Percentage of use of programming languages among all the cited programming 
# languages by the respondents.

plt.figure(figsize=(14,8))
sns.barplot(x = 'Percentage', 
            y = 'Programming Language',
            hue = 'Year',
            data = worked_lang_chart.sort_values(by='Percentage', ascending=False))

plt.xlabel("Percentage", fontsize = 14)
plt.ylabel("Programming Languages", fontsize = 14)
plt.legend(fontsize='x-large', title_fontsize='40')
plt.title('Most common Programming Languages used in Brazil', size = 16)
plt.show()<jupyter_output><empty_output><jupyter_text>Part 2 — What are the most wanted programming languages in Brazil? We can see that **Python** have grown tremendously close to other languages. This is probably happening because its’s a programming language that is very versatile and has been used extensively in data related areas. Most of the programming languages that have appeared as programming languages most used at work, also appeared in the ranking of most wanted programming languages, this show us that are many people wanting to learn these languages.<jupyter_code># Figure 2 — Percentage of desire of programming languages among all the cited programming 
# languages by the respondents.

plt.figure(figsize=(14,8))
sns.barplot(x = 'Percentage', 
            y = 'Programming Language',
            hue = 'Year',
            data = wanted_lang_chart.sort_values(by='Percentage', ascending=False))

plt.title('Most wanted Programming Languages used in Brazil', size = 16)
plt.xlabel("Percentage", fontsize = 14)
plt.ylabel("Programming Languages", fontsize = 14)
plt.legend(fontsize='x-large', title_fontsize='40')
plt.show()<jupyter_output><empty_output><jupyter_text>Part 3 —How does programming languages used at work relates with programming languages people want to learn? By looking at the raw data, we can spot some patterns such as for people that use python at work, for example, python is also cited as language that those people wanted to learn in the next year. A natural question arise:**“The pattern observed for python holds for the other languages?”** To address this question, we built a heat map that indicate how the work programming languages ​​were related to the desired programming languages. The darker the position gets, the more related the programming language is.<jupyter_code># Grouping data to create a new dataframe

rows = []
for itens in list(zip(worked_lang_2018, wanted_lang_2018)):
    for i in itens[0]:
        rows.append({
            'Worked_PL': i, 
            'Wanted_PL': Counter(itens[0])
        })
        
programming_language_transition = pd.DataFrame(rows).groupby('Worked_PL').agg({'Wanted_PL': 'sum'}).reset_index()
# Creating a matrix

for row in programming_language_transition['Worked_PL']:
    programming_language_transition[row] = 0

for index, row in programming_language_transition.iterrows():
    total = sum([value for key, value in dict(row['Wanted_PL']).items()])
    for key, value in dict(row['Wanted_PL']).items():
         programming_language_transition[key].loc[index] = (value / total)
## Figure 3 — Percentage of relationship of work programming languages and desire programming 
## languages in 2017 and 2018.

matrix = programming_language_transition.drop('Wanted_PL', axis=1)\
                                        .set_index('Worked_PL')

plt.figure(figsize=(16, 10))
sns.heatmap(matrix, cmap="YlGnBu")
plt.title('Programming Languages Heatmap', size = 16)
plt.ylabel('')
plt.show()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup