Three other people, a woman and her two children were taken from the house by witnesses and transported to an area hospital for treatment.
Three neighbors pulled out the three people, a 40-year-old woman, a 12-year-old boy and a 3-year-old girl.
One of the men, who gave his name as Dee, said that they could not get to the elderly woman in the fire.
"I ran over and when I got over here there was a lady on the ground, he said. "There was three people still in the house. So me and two other guys went in the house through the back door and pulled them out. We pulled the young lady out and put her on the ground.
"We tried to get the old lady out, we couldn't."
The mother and her two children are in stable condition, hospitalized for smoke inhalation.
The fire started at about 2:15 p.m. at the house, located just west of Mt. Elliot on the city's east side.
Neighbors tell FOX 2 that the 97-year-old woman was known as Ms. Mack, "a very sweet woman who has lived in that house for years."
The 40-year-old is said to be Ms. Mack's granddaughter. The cause of the fire is still under investigation.
One of the fire trucks responding was involved in a crash nearby at the time of the fire.