Will Ferrell has revealed why he refuses to return as Buddy the Elf, despite the role becoming one of his most beloved. The 2003 holiday hit Elf remains a Christmas favorite, but behind the scenes, not everyone loved Buddy’s quirky charm.

Ferrell shared on the Messy podcast that his co-star, the late James Caan, wasn’t a fan of his performance during filming. “He’d say, ‘I don’t get you. You’re not funny,’” Ferrell recalled. Although Caan later admitted the performance was “brilliant” after seeing the final cut, the tension was real. “He was truly annoyed with me,” Ferrell joked.

Talks of a sequel fizzled out, and Ferrell turned down a $29 million offer. Why? “The script wasn’t good. I couldn’t promote it honestly,” he said.

Caan also claimed Ferrell and director Jon Favreau clashed on set, which reportedly blocked the sequel. “Will didn’t want the director,” Caan explained in an interview.

Despite the laughs Buddy brought to millions, Ferrell has firmly closed the door on revisiting the North Pole.