Dustin Hoffman as Master Shifu , an old red panda who is the stern kung fu master to the Five and ( unwillingly ) to Po ; he is troubled over mistakes in the past and is the former - master and adoptive father of Tai Lung .