<jupyter_start><jupyter_text>Profiling & Optimization in Python  About me             Optimization  3 Rules of optimization  Don't  💡Always have measurable performance goals before optimizing code.  Don't Yet 💡Developer time is expensive, sometimes the best solution is to upgrade hardware.  Profile before Optimizing 💡Profiling measures where code spends time.  Measuring time  Two modules- time    - time.clock() obsoleto en 3.3    - time.monotonic()    - time.perf_counter()    - time.process_time()    - time.time() - timeit - https://www.webucator.com/blog/2015/08/python-clocks-explained/  Examples perf_counter:<jupyter_code>def upto_for(n):
    total = 0
    for i in range(n):
        total += i
    return total

def upto_sum(n):
    return sum(range(n))
n = 1_000_000
from time import perf_counter
start = perf_counter()
upto_for(n)
duration = perf_counter() - start
print("upto_for", duration)
start = perf_counter()
upto_sum(n)
duration = perf_counter() - start
print("upto_sum", duration)<jupyter_output>upto_sum 0.0348250830000012<jupyter_text>Examples timeit:<jupyter_code>from timeit import timeit

items = {
    'a': 1,
    'b': 2,
}

default = -1

def use_catch(key):
    """Use try/catch to get a key with default"""
    try:
        return items[key]
    except KeyError:
        return default

def use_get(key):
    """Use dict.get to get a key with default"""
    return items.get(key, default)
# Key is in the dictionary
print('catch', timeit('use_catch("x")', 'from __main__ import use_catch', number=5000000))
print('get', timeit('use_get("x")', 'from __main__ import use_get', number=5000000))<jupyter_output>catch 0.7718442920000044
get 0.3403776250000021<jupyter_text>Magic function timeit:<jupyter_code>%timeit use_catch("a")
%timeit use_get("a")<jupyter_output>72 ns ± 0.201 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)<jupyter_text>CPU Profiling  Deterministic profiling *record every function call and return, as well as exceptions*  Statistical profiling*record where the program is at small intervals*  Profiling in Python  Deterministic profiling -  cProfile (C extension)-  profile (pure Python module)               cProfile  C extension  Based on lsprof  Deterministic profiling   cProfile is built to trace a single thread- multi-threaded application ([Yappi](https://github.com/yousong/yappi)).  Example CProfile<jupyter_code>from crypt import crypt
import sqlite3

salt = <PASSWORD>'  # Bad security, store safely
db = sqlite3.connect('./passwords.db')
db.row_factory = sqlite3.Row  # Access columns by names
def user_passwd(user):
    """Get user password from db"""
    cur = db.cursor()
    cur.execute('SELECT passwd FROM users WHERE user = ?', (user, ))
    row = cur.fetchone()
    if row is None:  # No such user
        raise KeyError(user)
    return row['passwd']


def encrypt_passwd(passwd):
    """Encrypt user password"""
    return crypt(passwd, salt)
def login(user, password):
    """Return True is user/password pair matches"""
    try:
        db_passwd = user_passwd(user)
    except KeyError:
        return False

    passwd = encrypt_passwd(password)
    return passwd == db_passwd
from random import random


def gen_cases(n):
    """Generate tests cases"""
    for i in range(n):
        if random() > 0.1:  # 90% of logins are OK
            yield ('daffy', 'rabbit season')
        else:
            if random() < 0.2:
                yield ('tweety', 'puddy tat')  # no such user
            else:
                yield ('daffy', 'duck season')


def bench_login(cases):
    """Benchmark login with test cases"""
    for user, passwd in cases:
        login(user, passwd)
n = 1000
cases = list(gen_cases(n))
import cProfile

cProfile.run('bench_login(cases)', sort="cumtime")
cProfile.run('bench_login(cases)', filename="profiling.out")
import pstats
from pstats import SortKey

sortby = SortKey.CUMULATIVE

ps = pstats.Stats("profiling.out").strip_dirs().sort_stats(sortby)
ps.print_stats(10)<jupyter_output>Fri Nov  5 09:58:43 2021    profiling.out

         8912 function calls in 0.040 seconds

   Ordered by: cumulative time
   List reduced from 13 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.040    0.040 {built-in method builtins.exec}
        1    0.000    0.000    0.040    0.040 <string>:1(<module>)
        1    0.000    0.000    0.040    0.040 2615380777.py:16(bench_login)
     1000    0.002    0.000    0.040    0.000 2293933993.py:1(login)
     1000    0.001    0.000    0.027    0.000 1035362234.py:1(user_passwd)
     1000    0.021    0.000    0.021    0.000 {method 'execute' of 'sqlite3.Cursor' objects}
      977    0.001    0.000    0.010    0.000 1035362234.py:11(encrypt_passwd)
      977    0.001    0.000    0.009    0.000 crypt.py:70(crypt)
      977    0.009    0.000    0.009    0.000 {built-in method _crypt.crypt}
     1000    0.004    0.000    0.004    0.000 {method 'fetchone' of 'sqli[...]<jupyter_text>snakeviz<jupyter_code>%load_ext snakeviz
%snakeviz bench_login(cases)<jupyter_output>*** Profile stats marshalled to file '/<KEY>'. 
Embedding SnakeViz in this document...<jupyter_text>Example line_profiler<jupyter_code>%load_ext line_profiler
%lprun -T lprof0 -f login bench_login(cases)<jupyter_output>*** Profile printout saved to text file 'lprof0'.<jupyter_text>Tracing Memory Allocation - Programs contain data.- They are stored in the memory. - Accessing memory becomes important.- In modern computers it is done in layers.  Latency                          tracemalloc & memory_profiler  - The tracemalloc module is a debug tool to trace memory blocks allocated by Python. (New in 3.4)  - memory_profiler - tracemalloc nos permite entender como esta asignando python la memoria en nuestro programa.- memory profiler nos permite ver linea por linea donde se esta yendo la memoria.  Example memory_profiler<jupyter_code>%%file memory.py

import numpy as np

@profile
def allocate():
    vector_list = [float(i) for i in range(1_000_000)]
    vector_np = np.arange(0, 1_000_000, dtype="d")
    
allocate()
! python -m memory_profiler memory.py
! mprof run memory.py
! mprof plot -o image.png<jupyter_output>Using last profile data.
StarcoderdataJupyterStructuredCleanDedup