Rob Lowe is departing from ABC's family drama Brothers & Sisters after four seasons, according to The Hollywood Reporter.

Lowe joined Brothers & Sisters shortly after the start of the series in the fall of 2006 as a recurring and was quickly promoted to regular, playing Sen. Robert McCallister.

While he has played a key role in several story lines on the show, mostly through his romantic relationship with Kitty (Calista Flockhart), McCallister has never risen to the level of prominence of the Walker family members, who are at the center of the show.

Lowe may continue on ABC.

"We would like to keep Rob at ABC," the network's entertainment president Steve McPherson said in a statement. "We feel he should be anchoring his own franchise. We are in talks now."