<jupyter_start><jupyter_text>What I wanted to do is to map each player from tracking data to the camera data like this: The same data can be normalized and viewed on the same plot:<jupyter_code>k = get_kp('57583_000082', 12)
get_kp.plot(True)<jupyter_output><empty_output><jupyter_text>Point Cloud Registration So, a good portion of this comp comes down to point-cloud registration. There are plenty of methods that perform this taks. For example, OpenCV have `cv.findHomography`. As you can see below, this doesn't perform well. The main reason is that although the function uses Random sample consensus (RANSAC) to filter outliers, the clouds of points must be roughly alinged already.<jupyter_code>srcPoints = k['Tracking'].astype('float32').reshape(-1,1,2)
dstPoints = k['Sideline'].astype('float32').reshape(-1,1,2)
M, mask = cv.findHomography(srcPoints, dstPoints, cv.RANSAC)
print(mask.sum())
tfmdPoints = cv.perspectiveTransform(srcPoints,M)
plt.scatter(srcPoints[:,0,0], srcPoints[:,0,1], marker = 'o', color = 'red', label = 'source')
plt.scatter(dstPoints[:,0,0], dstPoints[:,0,1], marker = '^', color = 'green', label = 'target')  
plt.scatter(tfmdPoints[:,0,0], tfmdPoints[:,0,1], marker = 'o', color = 'blue', label = 'result')
plt.legend();<jupyter_output><empty_output><jupyter_text>Pure Pytorch implementation (using gradient descent)So, isntead of using open CV, I decided to do this my won using pytorch. The idea here is to solve the folowing problem:\begin{equation}\begin{bmatrix} x^{'} \\ y^{'} \\ 1 \end{bmatrix} = H \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}\end{equation}where $H$ is a transformation matrix (that I want to find) and $(x', y')$ is the transformed coorinates of a point $(x, y)$.The optimization problem is to find $H$ such as $(x', y')$ is as close as possible to some ground truth $(x_t, y_t)$ cloud of points.<jupyter_code>import torch
def min_mse(preds, targets):
    d = torch.cdist(preds.squeeze(2), targets.squeeze(2))
    loss = (d.min(dim = 1).values**2).mean().sqrt()
    return loss
def step(src, trg, m, lr = 3e-3, prt = True):
    preds = torch.matmul(m, src) # Homography transform
    loss = min_mse(preds, trg)   # mse between the closes pair of points
    if prt: print(f'loss: {(loss.item()):.5f}')
    loss.backward()
    m.data -= lr * m.grad.data
    m.grad = None
def fit_predict(src, trg, init_rot = 0, lr = 3e-3, n_steps = 10000, verbose = True):
    t = np.pi * init_rot / 180
    m = torch.tensor([[np.cos(t),-np.sin(t), 0],
                      [np.sin(t), np.cos(t), 0],
                      [        0,         0, 1]], dtype = torch.double)
    m.requires_grad_()
    for i in range(n_steps): 
        if i % (n_steps//10) and verbose:
            step(src, trg, m, lr=lr, prt=False)
        else:
            step(src, trg, m, lr=lr)
            
    with torch.no_grad():
        tfm = torch.matmul(m, src)
        
    if verbose:
        plt.scatter(src[:,0], src[:,1], marker = 'o', color = 'red', label = 'source')
        plt.scatter(trg[:,0], trg[:,1], marker = '^', color = 'green', label = 'target')  
        plt.scatter(tfm[:,0], tfm[:,1], marker = 'o', color = 'blue', label = 'result')
        plt.legend();
        
    return tfm
k = get_kp('57583_000082', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg)
k = get_kp('57597_000658', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg)
k = get_kp('57781_000252', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg, 0)
spwt('57781_000252', 1)
spwt
video_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_baseline_helmets.csv')
video_df = add_video_features(video_df)
video_df.query(f"game_play == '57781_000252' and frame == '1' and camera == 'Sideline'").head(10)
video_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_baseline_helmets.csv')
video_df = video_df.query('conf > 0.8')
video_df = add_video_features(video_df)
get_kp_highconf = get_keypoints(video_df)
k = get_kp_highconf('57781_000252', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg, 0)
k = get_kp('57583_000082', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Endzone']), torch.ones(len(k['Endzone'])).unsqueeze(1)], axis = -1).unsqueeze(2)
print(src.shape, trg.shape)
tfm = fit_predict(src, trg)
k = get_kp('57583_000082', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Endzone']), torch.ones(len(k['Endzone'])).unsqueeze(1)], axis = -1).unsqueeze(2)
print(src.shape, trg.shape)
tfm = fit_predict(src, trg, 90)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup