Saturday, March 31 2007
I received a call from the tech on call for the weekend at 9 p.m. seeking assistance, the email server was down. Drive E containing the information store and logs was full. He had deleted all log files to create 6G of empty space, but the information store could not be mounted. I connected to the server remotely and investigated the issue. The current log file required to mount the information store database was also deleted, this rendered the database inaccessible. All articles in Microsoft’s knowledge base indicated that the only recourse was to restore the data from backup. By now it was 1 a.m. April 1, I wished it was a cruel April fool’s joke someone was playing on me, but my wish was not granted.  I then tried to restore Friday night’s backup from tape, but the tape was inaccessible. I called the tech on, but his phone was off. I sent out an email to all IT personnels via blackberry asking for volunteers to go into the office to load the tape. At this point I was exhausted and decided to get some rest, it was close to 2 a.m.Â
Sunday, April 1, 2007
I might as well have gone to the office the night before because I had terrible sleep, the issue nagged me all night. I drove to the office at 7 a.m., loaded the tape and ran the restore of databases. The restore took over an hour to complete and failed at 99%, Arrrrrrggghhhhhhh. It seems that the databases were restored but could not be mounted. I tried running the Eseutil /p to fix corrupt database, that requires 10% of database size as free space. The free space was very low, less than the recommended 10%, having no choice I proceeded with the utility. It took over 2 hours to complete. I then proceeded with initiating the Eseutil /d command, that defrags and clear all white spaces in the database. This command would not run as it required %110 free disk space, big Arrrggghh again. I decided to skip the defrag and runt the Isinteg utility to fix the integrity of the log file vs the database. That utility would not initiate at all, resulting in numerous errors. At this point I decided to copy the database over to a different server that had enough free space. The database was huge and took forever to copy. I then proceeded to run the eseutil /d command, at the end of over 2 hours it completed. Terrible result, the database was reduced to 10% of it’s original size, that means 90% of information was lost. Having no other recourse, I called Mircrosoft for assistance. I charged the fee of US $490 to my personal credit card and I believe there is an additional charge for weekend support. The Microsoft rep talked me through the fix to get the Isinteg running. That completed with many fixes, the goal was to run it many times until there were no more reported errors fixed. Each run took close to 3 hours to complete. Meantime I made the executive decision to create a fresh database in order to get email services up. However, I had to stop send and receive of all old emails because of low free disk space. The old database was possible unstable and mounted. The idea was to merge the old database to the new, but more than double the free disk space was required. We needed the SAN, but I was unfamiliar with how to connect to Windows server. It was close to 9 p.m., I was starving and exhausted due to lack of food. I ended the call with Microsoft and decided to go home and work remotely. It was about midnight before the final run of Isinteg completed without any fixes.Â
Monday, April 2, 2007
I connected with the vendor for Equallogic  and the rep talked me though connecting the SAN volume to the mail server, now I had the free space to work with. The new database was migrated over to the SAN volume. Full send and receive email services was restored. I initiated the merger of old emails to the new database alphabetically by firstname, it was a very slow process. By end of day I was still on letter E. I then passed the process over to my backup. He worked through the night to complete the remaining mailboxes.
Tuesday, April 3, 2007
Fixed some problem mailboxes and ignored the ones that were full and belonged to the truck drivers. I don’t believe they ever check their emails and it would be wasted resources, if they were full to being with why bother.
