The hosts missed three of their spot-kicks, with Lehlohonolo Majoro shooting wide to give Mali victory.
The match promised much, with South Africa playing well under the weight of expectation, and took the lead in the first half, with striker Tokelo Rantie firing home from Thuso Phala's cross.
Mali captain Seydou Keita's equaliser saw his side draw level, heading past South Africa keeper Itumeleng Khune from six yards.
Coach Gordon Igesund said Bafana made "huge strides".
"My boys lost with dignity, pride and passion," he said.
"There is only so much any coach can ask from his team and my boys gave everything they had.
"We have proven that we can match anybody. It's just a pity with penalties - it's the luck of the draw."
South Africa now turn their attentions to qualifying for the 2014 World Cup in Brazil. They have drawn their opening two qualifiers and play Central African Republic next month.
Copyright is owned by Africa Outlook and/or Outlook Publishing. All rights reserved.