<jupyter_start><jupyter_text>Draw all bounding boxes for detected object<jupyter_code>from glob import glob
from pathlib import Path

import pandas as pd
import numpy as np
import cv2

from PIL import Image
import matplotlib.pyplot as plt

from scipy.optimize import linear_sum_assignment

data_path = "../data"
img_width = 1920
img_height = 1200
def from_yolo_to_cor(box, shape):
    img_h, img_w, _ = shape
    # top left
    x1, y1 = int((box[0] - box[2]/2)*img_w), int((box[1] - box[3]/2)*img_h)
    # bottom right
    x2, y2 = int((box[0] + box[2]/2)*img_w), int((box[1] + box[3]/2)*img_h)
    return {"x1": x1, "y1": y1, "x2": x2, "y2": y2}
    
def draw_boxes(img, boxes):
    for box in boxes:
        coords = from_yolo_to_cor(box, (1200, 1920, 3))
        cv2.rectangle(img, (coords['x1'], coords['y1']), (coords['x2'], coords['y2']), (0,255,0), 3)
    plt.figure(figsize=(10,10))
    plt.imshow(img)
for img in glob(f"{data_path}/*.jpg"):
    image = np.array(Image.open(img))
    boxes = pd.read_csv(f"{data_path}/{Path(img).stem}.txt", delimiter=" ",  names=["category", "x", "y","width", "height"])
    draw_boxes(image, boxes[["x", "y","width", "height"]].values)<jupyter_output><empty_output><jupyter_text>SORT steps1. Get bounding box detection at t0, predict bounding box using Kalman filter for t1'2. Using Hungarian/Munkres algorithm get the assignments using IoU score between detected bounding boxes t1 and predicted t1' bounding boxes3. Update the kalman filter with the assigned prediction t1 and the Kalman model for that bounding box.  Load two consecutive frames t0 and t1<jupyter_code>frame_t0 = pd.read_csv(f"{data_path}/1478019953689774621.txt", delimiter=" ",  names=["category", "x", "y","width", "height"])
frame_t1 = pd.read_csv(f"{data_path}/1478019954186238236.txt", delimiter=" ",  names=["category", "x", "y","width", "height"])<jupyter_output><empty_output><jupyter_text>Select only cars for now to run tracking on and select only: x, y, w, h.<jupyter_code>frame_t0 = frame_t0[frame_t0['category'] == 0][["x", "y", "width", "height"]]
frame_t1 = frame_t1[frame_t1['category'] == 0][["x", "y", "width", "height"]]<jupyter_output><empty_output><jupyter_text>Kalman filter using constant velocity modelUsing the bounding box, predict the bounding box for the next timestep using constant velocity model<jupyter_code>#import sys
#!{sys.executable} -m pip install pykalman
from pykalman import KalmanFilter
cx, cy, box_width, box_height = frame_t0.values[0]<jupyter_output><empty_output><jupyter_text>Velocities initialized to 0 in the state vector, our filter will optimize these.<jupyter_code>vy, vx ,v_width, v_height = 0, 0, 0, 0
frame_0_state_vector = [cx*img_width, cy*img_height, box_width*img_width, box_height*img_height, vx, vy, v_width, v_height]
frame_0_state_vector<jupyter_output><empty_output><jupyter_text>Initialiaze the covariance P matrix, bigger the value bigger the expected noise, will be adjusted by the filter. Same size as the state vector, one covariance per variable.Called `initial_state_covariance` in pykalman (default identity matrix)<jupyter_code>p = np.diag(np.ones(8))*10
p<jupyter_output><empty_output><jupyter_text>Set up the transition matrix FCalled `transition_matrices` in pykalman, `A`<jupyter_code>dt = np.ones(4) * 0.01
A = np.diag(dt, 4)
np.fill_diagonal(A, 1)
A<jupyter_output><empty_output><jupyter_text>Process covariance, noise matrixPykalman `transition_covariance` also `Q`<jupyter_code>Q = np.diag(np.ones(8)) * 0.01
Q<jupyter_output><empty_output><jupyter_text>Measurement noise, the pixel error we can expect from our Yolov3 predictions in x and ycx and cy noise: 1 pixelwidth and height noise: 10 pixels`observation_covariance` in Pykalman `R`<jupyter_code>R = np.diag(np.ones(4))
R[2][2] = 10
R[3][3] = 10
R<jupyter_output><empty_output><jupyter_text>Pykalman can be initialized with [n_dim_obs, n_dim_obs] for observation, and we have 8 observations. This does not need to be explicitly stated when transition matrix etc are defined.<jupyter_code>n_timesteps = 2
n_dim_state = 8
n_dim_obs = 4
states = np.zeros((n_timesteps, n_dim_state))
measurements = np.zeros((n_timesteps, n_dim_obs))
states.shape
measurements.shape
means, covariances = kf.filter(measurements)
kf = KalmanFilter(transition_matrices=A, transition_covariance=Q, initial_state_covariance=p)
np.shape(frame_0_state_vector)
filtered_state_means, filtered_state_covariances = kf.filter(frame_0_state_vector)
filtered_state_means.shape<jupyter_output><empty_output><jupyter_text>Results are `[cx, cy, box_width, box_height, vx, vy, v_width, v_height]`<jupyter_code>filtered_state_means<jupyter_output><empty_output><jupyter_text>Hungarian algorithmhttps://towardsdatascience.com/computer-vision-for-tracking-8220759eee85<jupyter_code>frame_t0.values
frame_t1.values<jupyter_output><empty_output><jupyter_text>Create a matrix with pairwise comparison between the bounding boxes in frame t0 (already tracking) and frame t1 (new detections)<jupyter_code>h_mat = np.zeros((len(frame_t0.values), len(frame_t1.values)))
h_mat<jupyter_output><empty_output><jupyter_text>Calculate the IOU for bounding box each pair in the matrix.<jupyter_code>def bb_intersection_over_union(box_a, box_b):
    #assert box_a['x1'] < box_a['x2']
    #assert box_a['y1'] < box_a['y2']
    #assert box_b['x2'] < box_b['x2']
    #assert box_b['y2'] < box_b['y2']
    
    # determine the (x, y)-coordinates of the intersection rectangle
    x_1 = max(box_a['x1'], box_b['x1'])
    y_1 = max(box_a['y1'], box_b['y1'])
    x_2 = min(box_a['x2'], box_b['x2'])
    y_2 = min(box_a['y2'], box_b['y2'])
    
    if (x_2 - x_1) < 0 or (y_2 - y_1) < 0:
        return 0.0

    # compute the area of intersection rectangle
    intersection_area = (x_2 - x_1 + 1) * (y_2 - y_1 + 1)
    print(intersection_area)

    # compute the area of both the prediction and ground-truth
    # rectangles
    box_a_area = (box_a['x2'] - box_a['x1']) * (box_a['y2'] - box_a['y1'])
    box_b_area = (box_b['x2'] - box_b['x1']) * (box_b['y2'] - box_b['y1'])
    
    print(box_a_area)
    print(box_b_area)

    # compute the intersection over union by taking the intersection
    # area and dividing it by the sum of prediction + ground-truth
    # areas - the interesection area
    iou = intersection_area / float(box_a_area + box_b_area - intersection_area)
    # return the intersection over union value
    if iou >= 0.0 and iou <= 1.0: 
        return iou
    else:
        return 0.0
    
for idx, box_t0 in enumerate(frame_t0.values):
    for idy, box_t1 in enumerate(frame_t1.values):
        if np.array_equal(box_t0, box_t1):
            continue
        box_0 = from_yolo_to_cor(box_t0, (1200, 1920, 3))
        box_1 = from_yolo_to_cor(box_t1, (1200, 1920, 3))
        print(box_0)
        print(box_1)
        iou = bb_intersection_over_union(box_0, box_1)
        h_mat[idx][idy] = iou

h_mat
linear_sum_assignment(-h_mat)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup