Wipe Out! Town Computer System Fails [UPDATE]

Updated March 22, 11:45 p.m.:

Greenwich officials said Thursday that hardware failures and human error are to blame for the massive loss of data in the town's Parks and Recreation Department.

The hardware failures involved backup systems causing the loss of all data regarding recreational permit applications and registrations for all of the department's programs, as well as an estimated 6,400 beach card applications. Parks and Rec staff will begin reconstructing the data base over the weekend, according to a statement released by Town Administrator John Crary.

"During the most recent reconfiguration of our backup system an IT technician failed to assure that all Parks and Recreation data was selected to be backed up. This error led to the loss of data for Parks and Recreation," Crary said in the statement.

Here is Crary's statement:

"As of today (March 22) the entire Town’s IT software systems are back in operation. Town departments, with the exception of Parks and Recreation, are operating on a normal schedule. Parks and Recreation data since February 26 is in the process of being reentered from other backup that is in the Town of Greenwich’s possession. The department plans to work over the weekend to reenter all permits and applications that had been previously received. Data for Golf, Recreation programs and permits for slips for the Town’s Marina’s are being entered first.  Once that is completed approximately 6,400 beach cards will be reentered from digital images that the department retains for internal use. The department will continue in this fashion until all data has been reentered.   

"It is important to note that while the main culprit of the loss of data were two simultaneous hardware failures there was an element of human error that caused the specific loss of data for the Parks and Recreation department. Specifically, during the most recent reconfiguration of our backup system an IT technician failed to assure that all Parks and Recreation data was selected to be backed up. This error led to the loss of data for Parks and Recreation but fortunately did not affect any other departments. 

"The Town apologizes for any inconvenience this may have caused any of our citizens and is working diligently to learn from this experience. The Town’s IT department supports over 130 different software packages that provide the technological backbone of our local government. The IT department has had for many years a Disaster Recovery plan to prepare for and respond to the many possible disaster scenarios that could occur. During the past four days we have had an opportunity to utilize the plan and to learn of all of its good and bad points. We are happy to report that with the exception of a few elements it has worked very well. One clear lesson learned from this experience has been the need to have a more in depth and redundant backup system. While simultaneous hardware failures are rare we can do more to assure that even in these circumstances the danger of the loss of data and the its impact on our daily operations are lessened.   

"Further details will be released as they become known."              

Original story, March 21:

Greenwich town officials are finding out that the computer age isn't always what it's cracked up to be.

The town's computer system crashed over the weekend and so did some of the backup systems, leaving town employees in an internet version of the Bermuda Triangle.

Word of the problem first surfaced Monday evening during the monthly meeting of the Board of Estimate and Taxation. Comptroller Peter Mynarski told the 12-member board—which is in its critical week of the budget process—"if you e-mail us and don't hear back in 30 to 60 minutes, call us." Mynarski added, "The system crashed over the weekend." The BET is in the midst of its budget deliberations, having held its public hearing Tuesday night and its budget vote scheduled for Thursday night.

Following the BET's budget public hearing at Central Middle School, First Selectmen Peter Tesei said, some elements of the computer backup system—housed at The Nathaniel Witherell nursing home—failed.

So what's the impact on Greenwich residents?

Your beach card or golf permits may be delayed. "We lost 3 weeks worth of data," Tesei said. "We have to go back and manually recreate the files." Tesei estimated the Parks & Recreation Department lost about 15,000 pieces of data—all related to beach and golf card applications.

Tesei said individual town employees also experienced computer woes, including himself. "I sent out a couple e-mails this afternoon ..." and the computer quit.

The town's IT department is continuing to work on fixing the problem.