<jupyter_start><jupyter_text>Setup<jupyter_code>#download the image data
%cd /content/
!gdown https://drive.google.com/uc?id=1HpFpsQ7VojeJbXj8i9i8IeIzXrzQ5zOs
!unzip "nerf_data.zip"; rm "nerf_data.zip"
# !gdown https://drive.google.com/uc?id=1ahp0Ac3I_7Fm1usqPT_Rvgww5KaLvo6Y
# !unzip "rollerblade.zip"; rm "rollerblade.zip"
%cd /content/
!git clone https://github.com/rohaldb/NSFF-custom
!mv NSFF-custom Neural-Scene-Flow-Fields
%cd /content/Neural-Scene-Flow-Fields
!chmod -R +x ./
!pip install -r requirements.txt
%cd /content
!git clone https://github.com/rohaldb/multi-mono-sf.git
%cd multi-mono-sf
!chmod -R +x ./
!pip install -r requirements.txt
#download nsf depth prediction model and move to appropriate place
!gdown https://drive.google.com/uc?id=1ROcvQzxtiBJOFY0MoLWk32emmzNOIAA4
!mv model.pt /content/Neural-Scene-Flow-Fields/nsff_scripts<jupyter_output><empty_output><jupyter_text>ExecutionFirst run the standard preprocessing pipeline of the neural-scene-flow paper. This will:1. Convert the intrinsics/extrinsics to a special readable format2. Resize the images and run a single view depth model to produce disparity estimates3. Produce optical flow estimates4. Use the optical flow estimates to obtain motion masksWe do not use the depth map or the optical flow for custom experiments.<jupyter_code>%cd /content/Neural-Scene-Flow-Fields/nsff_scripts
# create camera intrinsics/extrinsic format for NSFF, same as original NeRF where it uses imgs2poses.py script from the LLFF code: https://github.com/Fyusion/LLFF/blob/master/imgs2poses.py
!python save_poses_nerf.py --data_path "/content/nerf_data/kid-running/dense/"
# Resize input images and run single view model
!python run_midas.py --data_path "/content/nerf_data/kid-running/dense/" --input_w 640 --input_h 360 --resize_height 288
# Run optical flow model (for easy setup and Pytorch version consistency, we use RAFT as backbond optical flow model, but should be easy to change to other models such as PWC-Net or FlowNet2.0)
!./download_models.sh
!python run_flows_video.py --model models/raft-things.pth --data_path /content/nerf_data/kid-running/dense/ --epi_threshold 1.0 --input_flow_w 768 --input_semantic_w 1024 --input_semantic_h 576<jupyter_output><empty_output><jupyter_text>Next we run the mono-sf code, which will produce:1. Disparity maps2. Scene flow estimates3. Induced optical flow estimatesBy default, it will not produce visualisations to acompany the stored tensors. To include them in the output, add `--save_vis=True` to `eval_kid_running.sh`. The results are stored in various folders within /content/nerf_data/kid-running/dense<jupyter_code># Run mono-sf to obtain scene flow, induced optical flow and disparity estimates
%cd /content/multi-mono-sf/scripts
!./eval_kid_running.sh<jupyter_output><empty_output><jupyter_text>Next we need to undo the scene flow due to camera ego-motion from mono-sf estimates. The results are stored in /content/nerf_data/kid-running/dense/sf_corrected and /content/nerf_data/kid-running/dense/sf_bw_corrected<jupyter_code>%cd /content/Neural-Scene-Flow-Fields/nsff_scripts
!python ./account_for_camera_motion.py --datadir "/content/nerf_data/kid-running/dense" --final_height 288
#save to gdrive
%cd /content/
!zip -r custom_nerf_data.zip ./nerf_data
!cp -r custom_nerf_data.zip /content/drive/MyDrive/Thesis/Data
!ls /content/drive/MyDrive/Thesis/Data<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup