Bloomberg — A confluence of computer mishaps culminating in the failure of Nasdaq OMX Group Inc. backup systems led to last week’s three-hour stock halt, the second- biggest U.S. exchange operator said.
Exchange computers were flooded Aug. 22 with inaccurate data before flawed software disabled systems that should have prevented the malfunction from snowballing, according to a statement released yesterday. The challenges were “clearly within the control of Nasdaq OMX,” the company said.
Thousands of stocks stopped trading around the country and officials from President Barack Obama to Treasury Secretary Jacob J. Lew were alerted as the exchange worked to address the breakdown. The disruption underscored how quickly the integrity of the U.S. market, which has a value of about $20 trillion, can be subverted as orders to buy and sell shares are matched on more than 50 exchanges and alternative electronic venues.
“As an institutional investor, it shakes your confidence,” Daniel Genter, who oversees about $4 billion as president of Los Angeles-based RNC Genter Capital Management, said in a phone interview. “It also shows how crippling it can be if you have a major flaw in technology.”
Nasdaq is working with NYSE Euronext, owner of NYSE Arca, to provide a timeline to regulators detailing how events unfolded in one of the biggest U.S. market outages in decades. Nasdaq’s statement yesterday showed how faulty connectivity between a feed it oversees to disseminate prices to the market and Arca set off events that caused the shutdown.
Information Overload
Cascading data spurred by the Arca connectivity issue created a deluge that “vastly exceeded” the capacity of the company’s price conduit known as the securities information processor, Nasdaq said. The feed’s failure “then revealed a latent flaw in the SIP’s software code,” Nasdaq said.
The malfunction began when NYSE Arca sent more than 20 “connect and disconnect sequences” as well as a stream of quotes for inaccurate stock symbols, according to the summary. At one point, Nasdaq received over 2 1/2 times more data per second than the system’s tested capacity and about 26 times the average amount.
The report highlights a flaw where Nasdaq’s systems don’t delete stale market data and processing times “dramatically increase” as the messages build up, according to Haim Bodek, a managing principal at Decimus Capital Markets LLC and a former trader at UBS AG and his own firm, Trading Machines LLC.
“Both are no-nos in electronic trading systems design,” he wrote in an e-mail. “In other words, the SIP can enter a ‘death spiral’ as it is currently implemented.”
‘Failing Over’
After being inundated with data, the flaw in the SIP software prevented redundancy that is built into the system from “failing over cleanly” to a backup program, Nasdaq said.
“Any piece of software, even if it’s run for 100 percent for 10 years, there’s still flaws in it,” Nasdaq CEO Robert Greifeld said in an interview Thursday with Bloomberg News. “Whatever you have, it’s a question of what unique set of circumstances happen to reveal that.”
Several of the issues that led to the halt were within Nasdaq’s control, the New York-based exchange operator said in the report.
“We are responsible for them, regret them, and intend to take all steps necessary to address them to enhance stability and functionality of the markets,” the report said.
It said, “Other issues contributing to the halt are more endemic to technology issues across today’s complex markets and will require a broader industry-wide effort to resolve.”
‘Our Responsibility’
“That failover, that is our responsibility and our system did not handle that well,” Greifeld said.
The report said a “combination of large system inputs and delayed outputs ultimately degraded the ability of the SIP system to process quotes to an extent that a shutdown of the system was in the broader public interest, to prevent information asymmetry and ensure fair conditions for all market participants.”
The shutdown was the latest in a series of failures to disrupt markets, prompting the Securities and Exchange Commission to push for rules requiring executives to boost the reliability of their technology. Faulty software caused Nasdaq’s mishandling of Facebook Inc.’s public debut last year.
In a separate message sent to clients yesterday, Nasdaq said that it’s developing plans to better communicate with companies listed on its exchange and identify systems that need improvement after last week’s technical malfunction.
Forensic Analysis
Nasdaq will be working with regulators, members of the exchange and partners to conduct a “thorough forensic analysis on what happened” and identify areas for improvement, the market operator told customers yesterday. The message was sent by Bruce Aust, executive vice president of Nasdaq’s global corporate client group, and Robert McCooey, senior vice president of the global corporate client group, and obtained by Bloomberg News.
“It is clear that these systems could be more robust in their support of markets given the complex ecosystem in which we operate,” the client letter said. “We are reviewing additional enhancements and potential design changes to further strengthen the SIP resiliency,” it said.
Richard Adamonis, a spokesman for NYSE Euronext, declined to comment.
