One of the feature of Lotus Domino that I can’t live without (as Lotus Domino administrator) is Transaction Log. Transaction Log help me a lot when the server crash (very rarely) and backup time.
Without the transaction log, when the server crash, it will take hours for the server to recovery. It will check every databases one at a time and try to check and fix if it found any problem. And when the database if being fix, the user can’t access it. If the database is big, more than 10 GB, it will take hours to complete it.
With transaction log, after the server crash, the server will ready in less than 15 minutes.
Without the transaction log, the backup procedure will become a problem. In my case the total databases size is more that 1 TB, and it will take almost 24 hours to complete the full backup. And it’s almost impossible to me to run full backup everyday.
With the transaction log, I only to run the full backup every week, and I only need to backup the transaction log for daily backup. Yes the full backup will take up to 24 hours for 1 TB data, for for daily backup, it take less than 1 hour backup.
So, if you have a medium size or large lotus domino server, I highly recommend that you to run it with transaction log enable.
Before that, these are a few tips.
- Use the fastest and dedicated hard drive that you can use for transaction log. I’m using IBM SAS drive with 15000 RPM, and that still too slow to me.
- Use raid 1 for transaction log, it recommended by IBM. So if one drive is failed, you still have one to run the transaction log.
- Backup, backup, backup. Transaction log rely on good backup that support transaction log, it will only clear up the old TXN files after the backup is complete. Without a good backup, your hard drive will filled up by the TXN files.
- If possible, use the latest Lotus Domino version. I have problem with Lotus Domino 8.5.2 FP1 that crash because the size of the transaction log, even the hard disk still have many free space. After consult with IBM support, they suggested me to upgrade the fixed pacth to FP2, after that the problem is gone.
Popularity: 3% [?]
Steps to upgrade Domino 32-bit to Domino 64-bit in windows platform
Just did the upgrade both Windows server 2003 32-bit and domino 8.5.2 32-bit to 64-bit version.
After all it run well, then I found this technote.
Duh, I hope I found that technote much more earlier, before I upgrade my windows and domino into 64-bit version, that will make it a lot easier.
Please take a special attention on point seven (or six), “7. Before restarting the Domino server, run the offline maintenance on the following system databases from command prompt.”
If you have a large domino databases, and you have only limited down time, than copy entire domino data directory may be is not an option. In my experience, copy 1,2 TB of domino data will take more than a day. And after we start the new server, it give a lot of error messages and the fixup process run almost on all databases.
So we did it with another way, we create a additional server, create replica of all databases to the new server, and after that we only need to switch server id. It take a longer time to replicate (almost a week), but the down time is minimum, less than a hour.
So, which way is your choice, either copy the data all create replicas.
Popularity: 3% [?]
Do The IBM Hardware Support reduce in quality?
We just received a brand new xSeries 3620 M3 less than a month ago, and after configured and installed the operating system, we noticed that one of the hard disk light (from five hard disks) is not working. It mean not blinking at all.
Because this is a brand new server, and our first though this is only a hardware problem, on Friday we called the IBM support and hope that they can change the hard disk.
After we received the ticket ID, one of IBM support person called us and I explained the problem to him. After discussed, he want me to run the diagnostic tools the check the entire system. I though “What?”, but I followed him anyway, download the diagnostic tools, run it and send the result to his email address.
On Monday, we still do not received any news or any update, and on Tuesday I called the IBM support again and ask about my ticket status. Then I’ve got another call from another IBM support person, mentioned that the first IBM support person is not available so he will take over while he away.
Again, I explained the problem and re submitted the IBM diagnostic tools result to him, and he promised to update me about the status.
About one or two hours, I received his email, mentioned that the result that I just send to him is not complete because my server is not have a IPMI driver. So he gave me the link to download the IPMI driver and ask me to installed the driver. After that, I re run the diagnostic tools, and again re submit the result.
On Wenesday, there no update from them.
On Thursday, the IBM support person called me again and updated the status. And he said that the firmware in my server is not up to date, and require me to update the firmware. I though “What?”.
The problem is simple, the light activity in one the hard disk is not working (while in another four hard disks is working well), thats all. I really sure this only a small problem, maybe only something wrong with the bracket. Why update the firmwares?
On the phone, He mentioned that I need to update several firmwares, I said to him, just give me the list in the email, and I will look into it
When I received the email, I’m shocked.
He mentioned that I need to upgrade not only one, or two but four firmwares..
- Integrated Management Module
- IBM System x uEFI update for 3620 M3/3630 M3
- ServeRAID M1015 SAS/SATA Controller Firmware
- IBM FW HDD SAS-1.09 Windows 32-64
And not forget, he also add a little warning at the end of his email
“Please backup your data before updating the firmwares”
Great.
I consulted with IBM BP, and hope that somehow they can do something about it.
I used to recommend IBM xSeries to many people, and one of the reason because they have a very good support, but now, I have doubts, BIG doubts.
Update:
On monday, wrote them email mention that I will not do the firmware upgrade and they responded that they will their support to us to upgrade the firmware.
On Tuesday, one support come and a good thing is that he also bring a new hard disk. He upgraded all the firmware and it took more than one hour, but the light still not blinking. Then he replaced the hard disk with the new one, but the light still not blinking. Then he tried to plug the old hard disk into another slot, and the light now blinking. By that, the support engineer said that the problem is in the backplane of the server. And he will come back again tomorrow with a new backplane.
On Wenesday, he came again with a new backplane. After he switch the backplane with a new one, now the light is blinking fine.
So, the problem was in the backplane, not in the firmware or the hard disk. The diagnostic tools from IBM unable to detect the malfunction in the backplane. And it took more that a week for IBM to identify and fix the problems, that a big disappointing for me.
But the good thing is, that now the server working fine without any problems.
Popularity: 1% [?]
Three things that need to consider for new server
You need to buy a new server for your most critical data, the server should be able to run 24 hours a day, 7 days a week.
What are to consider?
Brand of the servers? Of course. The softwares? Sure.
Anything else?
These are three things that cross my mind.
Raid 5
If your data are important, and mostly are, than use server that support Raid, at least Raid 5.
With Raid 5, we use at least three hard disks with the same capacity (and better with the same brand and model) and configure it as one drive. If one hard disk failed, the data still can be accessible.
Without Raid 5, if you have three hard disk, and one hard disk is failed, that all the data in that hard disk will be lost. You need to rely on your latest backup and pray that restoring the data won’t be a problem.
Minor side with Raid 5 is that Raid 5 will use one hard disk capacity as it parity, so if you have three 1 TB hard disks, than the total of your capacity is only 2 TB.
And also with Raid 5 , the performance is slower.
With at least Raid 5, you will have redundancy and more capacity compare to Raid 1. There is also Raid 6 that extend the Raid 5.
More about Standard Raid Levels
Hot Swap
Ok, now you use Raid 5 for your critical servers that should run 24 hours a day and 7 days a week.
If one hard disk failed, at least your server still able to run with slower performance.
Now, you still need to replace the failed hard disk with a new one. If your server support Hot Swap capabilities like most IBM xseries, than it should not be a problem. You just need to unplug the failed hard disk, wait for 30 seconds and plug in the new one. Total time should be less than one minute.
But, what if your server do not support Hot Swap?
You need to shutdown your server (when? ), open the server case if necessary, detach the cables, unplug the failed hard disk, then plug in the new disk, attach the cables, close the case then power it on again.
How much time do you need for that?
And you need a spare time to do it, you can’t just shutdown the server on office hours, you need to do it after office hours, or maybe before office hours or maybe on weekend?
Warranty
Make sure that your servers have good support and spare parts availability.
One thing that I like the most from IBM server is their warranty and support.
While I’m working with the IBM servers, I have two mainboards problem and four failed hard disks.
The longest time it need is to replace one mainboard from the old server (very old), because they need to ship it from Singapore.
Recently I have one failed hard disk (one of the reason why I wrote this), call IBM support and describe the problem, and all they need is our machine type, serial number and the FRU of the hard disk. Then hard disk replacement already in the office in the next morning.
That what I called a good support.
If they need a month to replace a failed a broken part, that the support, the warranty is useless.
If your warranty going to expired, or already expired, you should, you must extend it. It’s not cheap, but for me, very worthed.
With IBM, you can choose 5 days support, or 7 days support. With 7 days support, if you call them on weekend, they still come.
Not sure for other brand, if they support as good as IBM. I only works with IBM brand.
Note, I’m not an employee of IBM.
Popularity: 1% [?]
Connection without Connection
This is really happen. 
I just have an electrical problem, the power from electric company was down, so we have to run our servers from ups.
After a while, the ups start to drop the battery, so we start to shutdown unnecessary servers, so it give the ups more power to run.
But, it not for long, the electriciy still down and the ups battery start to drop quickly, so we decided to run ONLY the smtp server, so at least we still can receive emails.
But someone in the IT department take it very literally, he or she also pull the plug of the switches, including the main switch that connect to the servers (including smtp server)
And the smtp server is up and running.
But without the connection to the main switch, how it can receive any emails? 
No one say that he or she that pull the plug of the switches, it still remain a mystery. 
Popularity: 1% [?]