Such glitches are fairly rare at market centers. But they do occur. The London Stock Exchange went down for seven hours in September; three months earlier, futures and options trading was halted for four hours at the Tokyo Stock Exchange, which also saw problems in November 2005. After a June 2001 software upgrade, the New York Stock Exchange stopped trading for about 90 minutes.
“It can happen to anyone,” said Karl Ottywill, the chief information officer at Alpha Trading Systems, an alternative trading systems that launched Nov. 7; “It’s horrible that it happened to [them] and I feel for them. When I look at the number of connection and integration points in the market, it’s a tremendous amount of work to keep that all going.”
Indeed, itÂ’s just not exchange executives who should be taking precautions. All those responsible for the smooth running of securities operations need to safeguard their systems and prepare recovery plans in case of sudden glitches or outages.
At TSX, trading was stopped when the system that supplies the exchangeÂ’s Level One data feed began spewing out zeroes rather than accurate prices, volumes and bid-and-ask data. When technicians tried to activate a backup, they discovered the system was not updating properly. Ultimately a series of tests pinpointed the culprit as malfunctioning firmware, programming code written into a computerÂ’s read-only memory at the time of manufacture. The device acts on the instructions during daily operations.
In this case, said TMX, the Toronto Stock Exchange’s parent organization, a network firmware problem “resulted in complications with data synchronization between the primary and redundant data feed generators, which impacted the delivery of the Level One data feeds.”
The day after the event, TMX chief executive Thomas Kloet apologized for the “disruption” in a prepared statement, adding that “all efforts are being made” to prevent a reoccurrence.
Data feed failures generally can be prevented by establishing and maintaining backup data centers, installing sufficient servers to switch over to fresh data, and constantly monitoring delays in data transmission on a network, particularly in response to microbursts — 5 to 10 millisecond periods of heavy traffic — according to research firm Celent.
Those moves are among the 10 steps experts agree firms should take to ensure that what happened in London, Tokyo and Toronto doesnÂ’t happen to them:
• Mirror all systems. Maintain twin data centers at all times. Also consider remote replication of transaction data. And place the centers in different cities, far apart.
• Keep both active at all times. Don’t switch over — operate both facilities constantly, feeding data or making trades.
• Maintain open communications. Have two or more pathways open at all times. Don’t use the same carrier.
• Double your power. Have two power supplies.
• Then, triple your protection. Install uninterruptible power systems and emergency generators.
• Mirror every box. Have multiple network cards.
• Install new firmware in a test network before migrating any changeover into the production network. This will flush out issues in getting devices to work together, identify performance problems and ensure that the system can withstand the change.
• Go first-class. Build a subsystem for transferring data using Fibre Channel or Serial Attached SCSI technologies, enabling around-the-clock, enterprise-class service.
• Know what to expect. Baseline your network’s infrastructure. Create metrics for monitoring how much of the network is being consumed by traffic and where and when the bandwidth is being used.
• Automate alerts. Install network management systems that send alerts when issues arise — and when an outage is imminent. The key, of course, is to do whatever it takes to avoid an outage. “Whenever you have mishaps,” said Sang Lee, managing partner of Boston-based Aite Group, “you give people an excuse to look at other venues.”