The fire erupted early Saturday and raced from row home to row home, leaving behind only charred frames. One witness said the blaze may have begun on a couch on a porch.
Twin sisters Maria and Marialla Bowah were among the dead. Their father, Pennoh Davis, told WPVI-TV that they loved to attend Sunday school.
"My little girls, they loved to play, loved to read, loved to draw and loved to sing," Davis said.
The girls' mother, Dewen Bowah, told police she was home with seven children and managed to get her three other daughters out before jumping from a second-floor window. But she couldn't save the twins, 1-month-old Taj Jaque and 4-year-old Patrick Sanyeah. The boys' mother wasn't in the home at the time.
"We lost four precious lives," Mayor Michael Nutter said Saturday. "Four little innocent children, in a horrific tragedy. I can only pray that their pain was not long and they did not truly feel and experience the intensity of this fire and flames."
Bowah and her three surviving daughters were taken to the hospital, but the extent of their injuries was not known.
Jeff Boone told The Philadelphia Inquirer that he saw a couch on fire on the porch of a house about five doors down from his residence and heard children screaming.
The flames spread across porches so fast, he said, that "it looked like someone had a flamethrower and just shot it all across."
Boone said he called 911 and bolted out of his house to try to save his neighbors.
"I was running, screaming, telling everybody, 'Get up! Yo, get yo! Go!'" he said.
Milton Musa told the newspaper his roommate woke him up and said their home was on fire. Once outside, Musa said, he saw two children hanging from a neighbor's window.
"I could see they weren't strong, and I was afraid they'd fall to the cement," Musa said. "So I went underneath them, let them fall on my back and carried them away."
He wiped away tears as he recalled the intense moments as the blaze swept through the neighborhood.
"Everyone was running for their lives," he said. "I've lost everything. My paperwork, my documents, my house. Everything."
About 100 firefighters responded to the blaze. Fire Commissioner Derrick Sawyer called it a "tragic, tragic day for the city of Philadelphia."
The Red Cross said 32 people were displaced by the fire.