Exchange Server 2007 Down due to Insufficient System Resource

Environment: Exchange Server 2007 SP1 with BES (BlackBerry Enterprise Server)
VMWare ESX Server 3.5.0

BES User was unable to send/receive mail from their BB.

Investigation:

- check BES server = OK
- check OWA/WebMail = inaccessible
- check Exchange Management = OK (services UP)
- telnet to exchange server port 25 = NOK --> Insufficient Resource
- check Exchange Server resource = Memory OK, but DISK FREE SPACE 'only' 140MB

discovered that Exchange Server 2007 has a 'Back Pressure'

In Exchange Server 2007, the Transport service monitors system resources such as disk space and memory on Transport servers (the Hub Transport and the Edge Transport servers), and stops message submission if it is running low on these resources. It continues to deliver existing messages in the queue. When resource utilization returns to normal, it resumes message submission. The feature is called Back Pressure.

The Back Pressure settings in Exchange Server 2007 RTM stop inbound mailflow if free disk space is below 4 Gigs. This static threshold has been lowered in SP1 to a more realistic 500 Mb.

Resolution:

- purge/delete some unused files, but free space still < 500Mb.
- Create new disk partition (idea to change mail.queue & logs to new path):
1. Stop BES
2. Shutdown Exchange Server
3. Add New Disk (10GB) from VMWare Infrastructure Client
4. Startup Exchange Server, format new disk from DiskManagement (control panel)
4. Re-configure Queue & Log path location, do backup original config (safety first!)
C:\Program Files\Microsoft\Exchange Server\Bin\EdgeTransport.exe.config

use notepad, change LocalPath value:

add key="QueueDatabasePath" value="LocalPath"
add key="QueueDatabaseLoggingPath" value="LocalPath"

i.e:
from add key="QueueDatabasePath" value="C:\original_path"
add key="QueueDatabaseLoggingPath" value="C:\original_path"
to add key="QueueDatabasePath" value="E:\Queue\QueueDB"
add key="QueueDatabaseLoggingPath" value="E:\Queue\QueueLogs"



5. Restart Exchange Transport Services (new mail.queue, log, chk, jrs created on new location)
6. Verify that the new Mail.que and Trn.chk files are created at the new location.
7. Verify that the new Trn.log, Trntmp.log, Trnres00001.jrs, Trnres00002.jrs, and Temp.edb files are created at the new location.
6. Move old/unused Mail.queue, Log, jrs, chk from original location to new disk (for backup only, in any case you will need it, actually you can delete it permanently)
7. Start BES service
8. Check that Exchange OWA/POP/SMTP & BES back to normal

BES Services Failed to start - ERROR Windows 1069

Hal tersebut berawal dari bermasalahnya Firewall, FW mem-blocked port 3101 with some reasons.


BES tidak bisa connect ke srp.id.blackberry.net port 3101, bahkan dengan manual telnet pun (DOS Command Line), connection refused.

Setelah firewall issue solved, masalah lain pun muncul di BES - Blackberry Manager.

SRP tidak bisa start-up tanpa alasan yg jelas, STARTING...lalu STOPPED dengan SRP status UNKNOWN. bahkan tidak ada log yang tertulis :(

Well, is it something like Windows behavior ? RESTART!

BES tak jua bisa 'naik', malah tambah parah...tidak ada log files yang generated. hanya log *MNGR* dengan last line: [DIAG] EVENT=Registered_thread ....... "Unnamed"

lalu, ditemukan semua services STOPPED kecuali BB Attachment Service. coba manual start services dari windows-services (administrative tools), and criiing.... pop-up error 1069 ...something like: "failed to start up due to "LOGON FAILURE".....

:D got the clue!

and then next what?

Seperti diketahui, semua service BES Server (kecuali BB Attachment Service) adalah 'Logon as DOMAIN/BESADMIN'.

Mungkin ada masalah dengan AD? BESADMIN user password reset, BES Server restart, re-join domain. tapi tetap tidak bisa men-startup satupun services BES!

well..well..well... gotcha!!!

BUG Windows? BUG BES? atau apa ini?

finally, ini yg dilakukan:
1. right click BES Services (yang logon as domain) satu-persatu
2. properties
3. Logon menu tab
4. ketik ulang password BESADMIN (domain user)
5. lalu, start BES Services secara berurutan: Controller - Dispatcher - Router
6. Open BES Manager
7. Start BES
8. Start MDS

DONE! SRP Connected and BES works fine....