It was coincidence. But the scheduling couldn’t have been better.
The securities industry had been set for more than three months to carry out its annual test of how trading firms, market operators and their utilities could operate through an emergency using backup sites, backup communications and disaster recovery facilities.
The date: Oct. 27.
The Business Continuity Test coordinated by the Securities Industry and Financial Markets Association along with the Financial Information Forum would take place two days ahead of the arrival of Hurricane Sandy, the largest storm ever spawned in the Atlantic Ocean.
“It was very timely, said Karl Schimmeck, vice president of financial services operations at SIFMA.
Participants in the 11th annual edition of the event included brokers, market data suppliers and technology service bureaus, as well as global banks, regional banks and financial advisers, according to Schimmeck.
The practice: Allow market participants to transmit dummy orders to markets, receive simulated execution reports, and conduct settlement and payment tests. Equities, options, futures, fixed-income, foreign exchange and commercial paper market operators all were “open” for business.
Before Hurricane Sandy-or any other storm-hit.
The testing window was from 9 a.m. to 1 p.m. Eastern time Saturday. The broad industry effort only uncovered “little issues,” Schimmeck said. Nothing that would preclude markets from opening two days later, with backup systems, if they so chose.
But they still chose not to.
By Sunday night, after as many as 14 hours of “epic” phone calls involving as many as 800 market participants at a time, the nation’s stock exchanges, led by the New York Stock Exchange, chose to shut down. Before Sandy reached landfall.
“It was a gutsy call,” said Tom Price, managing director and head of the Technology, Operations and Business Continuity Planning Group at SIFMA. “The markets always want to be open.”
NYSE chief executive Duncan Niederauer, in a post-storm interview on the floor of the exchange, said it would have been “irresponsible” to have opened for trading because its members and other articipants would have “a lot of people in harm’s way,” even if markets operated only electronically.
The storm caused $6 billion in business losses and more than $20 billion in physical damage, according to IHS Global Insight, a Boston analytics firm. Morgan Stanley and other Wall Street firms’ offices near the Hudson waterfront got shut down for weeks.
And, of course, all national exchanges chose to shut down for two days, an unprecedented move in an era when trading can take place entirely electronically. Particularly so, since the markets themselves are all housed in highly secure facilities, across the Hudson River in New Jersey.
The choice led to widespread back-channel criticism. Brokers and other exchange operators complained that the NYSE did not hold proper advance testing of its backup plan, which was to route all orders through Arca, its all-electronic exchange. Many of the NYSE’s own members, one broker and one operator told Traders Magazine, would have had to scramble Sunday evening to check their connections with Arca and make sure all ancillary services, such as market data feeds, were in place and operating properly.
The fieriest criticism came from former Securities and Exchange Commission chairman Arthur Levitt.
“People look to the New York Stock Exchange as being the symbol of American capitalism, and to see the exchange go down for two days without an adequate backup plan is very, very unfortunate,” Levitt said on a Bloomberg Radio interview. “To see the New York Stock Exchange crippled is a body blow that will really shake the image of that institution for a long time to come.”
Which met with a fiery response from the exchange.
“Levitt is appallingly and irresponsibly wrong. Our backup plan to trade on a fully electronic basis was ready to be implemented,” said senior vice president Robert Rendine, the exchange operator’s global head of communications. The industry, as a whole, decided to shut its markets, for non-electronic reasons: the safety of employees.
NYSE Euronext’s $250 million data center in Mahwah, N.J., which opened in 2010, is built to withstand hurricanes and can operate on backup power for a week. The exchange operator maintains a backup facility in the Midwest, as well.
The “scramble” on Sunday evening also could easily have been avoided, if NYSE members had gotten prepared before the arrival of Sandy was imminent.
NYSE’s emergency backup plan, to route all orders for NYSE-listed stocks to NYSE Arca, was approved in December 2009, and a formal “information memo” with details was published by NYSE Euronext on March 14, 2010.
That memo advised NYSE members that did not have connections with Arca to set up some sort of connection, for business continuity purposes.
Industry testing on the plan was conducted as recently as March. But a large number of NYSE brokers did not participate.
The Business Continuity Test carried out by SIFMA and FIF the weekend before Sandy hit gave one last chance for testing. The dummy operations tested systems put in place by brokers. Brokers could test trading in equities and options on the NYSE Arca platform, if they chose to do so.
But many brokers were not ready for the switchover, according to many market participants, which argued for closing the New York Stock Exchange on Monday and Tuesday, when the storm arrived.
“People have to be responsible for their own systems,” said Jim Toes, president of the Security Traders Association, which represents 4,200 individuals involved in trading equities and equity options.
In this case, the NYSE signaled at 4:21 p.m. Sunday that it had “decided to suspend physical floor operations and invoke its contingency plan” to trade all NYSE-listed stocks on NYSE Arca. Operations of NYSE MKT, once known as the American Stock Exchange, would be suspended.
That was the apparent trigger for firms not ready to send orders to NYSE Arca to start scrambling.
“Some individuals would have to go to their boss and say we have no connectivity to the backup and we will not be able to open up for business,” said Toes. “Reputations are made and broken on days like this, which makes it more than a one-day event.”
The NYSE dealt with criticism that it did not stage adequate testing of its backup plan immediately before Sandy hit.
“The forecast at the end of last week for the storm was certainly very ominous, but had a lot of uncertainty around it,” Joseph Mecane, co-head of U.S. listings and cash execution for NYSE Euronext, said during an online discussion of the industry’s response to Sandy. “I think the exchanges all agreed that the right way to proceed was to check in periodically throughout the weekend and assess things as they evolved.”
Price said the exchanges decided to put “people before profit.”
And if brokers or NYSE members with the same information about a weather-related emergency don’t get prepared by updating their connections to a known backup plan long in advance, it’s not necessarily an exchange’s fault if its members are not ready, at the last moment.
“Whose responsibility is that?” asked Toes.
In fact, there may be some trading firms that do not want to invest in backup systems, said Alex Tabb, a partner at industry consultancy Tabb Group.
High-frequency trading firms, for instance, “co-locate” their servers as close to an exchange’s matching engines as possible, under the same roof.
So setting up shop in a backup site, such as NYSE Arca, can be as expensive as setting up in a primary site. And each backup site required adds cost.
“So I pay my dollar to get a site in Mahwah or Carteret,” Tabb said, speaking of the primary data centers of the New York Stock Exchange and the Nasdaq Stock Market. “I then have to pay another dollar to get the site in Pennsylvania or Virginia or the Midwest or wherever (a backup site) may be. And not only do I have to pay my dollar at the first site and the second site, I’ve got to pay all the dollars to have the technology that is in the first site, I’ve got to pay all that technology for the second site, which could sit there for years, and never be activated. And every time I update my technology on the first site, I’ve got to update it on the second site. And it still sits there for years and never is activated. And all the market data feeds and all the algos have got to go in both locations, and I’ve got to manage all that. You start talking about a pretty penny that may never get used or get used once every 10 years.”
Which is a cost that a high-frequency firm, operating on a margin of less than a penny a trade, may not want to pay, Tabb said.
But, if you are set up, adjusting to the NYSE’s backup plan would have or should have been easy, said Jeff Bell, chief executive officer of Lime Brokerage, a unit of Wedbush Securities that develops high-speed technologies in order to provide “direct access catering” to traders, hedge funds, and institutions. Lime connects to all the stock exchanges operated by NYSE Euronext, Nasdaq OMX, BATS Global Markets and Direct Edge.
“All that would have happened, in my view of things, was we would have taken down the routes to the New York Stock Exchange proper” only, Bell said.
“What is different, at some other firms, is they maybe don’t trade all the markets,” he said. “They really just need to get an execution done and they like to go to the New York (exchange) because they’ve been doing it for decades. But if all of a sudden that execution venue is gone, they’ve got to find another way to work around” that.
The storm over the storm is likely to blow away. “Hindsight is 20/20,” said Sang Lee, managing partner of Aite Group, a financial technology advisory firm. “At the end of the day there are so many moving parts in our market infrastructure right now that it wasn’t worth the risk (that) all hell breaks loose,” if markets were to blow up on the day Sandy arrived.
The industry didn’t need another black eye, he said, in a year when one exchange operator, BATS, couldn’t take its own stock public on its own technology in March; when another exchange operator, Nasdaq, flubbed the first offering of stock to the public by Facebook; and when a major market maker, Knight Capital Group, nearly destroyed itself by ending a flood of erroneous orders that cost it $457.6 million and wiped out 70 percent of its shareholders’ equity.
In the end, the decision to shut down came once New York City’s transportation authority said at 7 p.m. Sunday it was shutting down mass transit. And with the realization that Sandy would actually hit Monday evening, not morning.
That could mean that employees who either stayed in the city or came into the city couldn’t get out after the close of trading.
That would “potentially create a disastrous situation right at the time when people would be feeling the need to be with their families and be with their own personal situations,” Mecane said.
The NYSE was not alone. “From my perspective, this is a really easy decision and one I think I would make again, every single time,” said William O’Brien, the chief executive officer of the Direct Edge exchanges. The shutdown “was just about putting people first and ensuring a fair and orderly market second. I think that’s the right priority.”
Nonetheless, Mecane said the NYSE will use the two-day shutdown as “a learning experience.”
There are “clearly some things” such as “specific” disaster recovery plans and communications protocols that the exchange operator will look at improving “for the next time,” Mecane said.
“I think we all plan to do a little bit of a deeper dive and more outreach around that,” he said, before the next superstorm arrives.
