Bernie Madoff Defends $60 Billion Ponzi Scheme to ABC News' Barbara Walters (Video)

Convicted ponzi schemer Bernie Madoff defends swindling his investors of billions of dollars in an interview with ABC News' Barbara Walters that aired on Good Morning America Thursday.

our editor recommends

'60 Minutes': Ponzi Scheme Perpetrator Bernie Madoff and Wife Attempted Suicide

Bernie Madoff Whistleblower Earns Standing Ovation at Toronto Screening

"I understand why clients hate me," he said. "The gravy train is over. I can live with that."

PHOTOS: The Most Talked-About TV News Faces

But, "the average person thinks I robbed widows and orphans. I made wealthy people wealthier," he added.

The worst thing about his 150-year prison sentence, being served at the Federal Correction Complex at Butler, N.C., is "not seeing my family and knowing they hate me. I betrayed them."

His wife Ruth, "asked me to let her go, which I understood," says Madoff. Ruth recently said on 60 Minutes that she attempted suicide with her husband after the ponzi scheme was revealed.

VIDEO: Bernie Madoff's Daughter-in-Law Speaks Out

"Ruth not communicating is the hardest thing," he went on. "Ruth doesn't hate me. She has no-one. It's not fair to her. She lost her first son. ... She is a devoted wife and didn't care about the money."

When asked what he'd tell his grandfather, Madoff says, "I am sorry to have caused them pain." Walters -- who sat with Madoff for two hours on Oct. 14 to conduct the interview -- notes that he had no apparent emotion when he made that statement.

Madoff says he makes about $170 a month doing various prison jobs. He passes time by reading and recently finished a book about Wall Street robber barons.

VIDEO: Bernie Madoff's Daughter-in-law Reveals She'd 'Spit in His Face' During '20/20' Interview

"I feel safer here than outside," Madoff said. "Days go by. I have people to talk to and no decisions to make. ... I know that I will die in prison. I lived the last 20 years of my life in fear. Now I have no fear -- nothing to think about because I'm no longer in control of my own life."

He admits he suffers from "terrible nightmares."

Stephanie Mack, Mark Madoff's widow, recently released a book about her own ordeal, The End of Normal: A Wife's Anguish, A Widow's New Life. In an interview with ABC News' Chris Cuomo that aired last week on 20/20, Mack said that her late husband had previously unsuccessfully attempted suicide by swallowing 60 anti-anxiety and sleeping pills.