You are on page 1of 273

Exchange Server 2007 SP1 Core Roles Troubleshooting

Joo Bravo joao.bravo@microsoft.com

Agenda
Introduction to Exchange Server 2007 Troubleshooting Troubleshooting Client Access Server (CAS) Troubleshooting Hub Transport Server (HT) Troubleshooting Mailbox Server (MBX)

Microsoft Confidential

Introducing to Exchange Server 2007 Troubleshooting

Introduction
Exchange Services Overview Active Directory Provider

Microsoft Confidential

Exchange Services shared by all Server Roles

Microsoft Exchange Active Directory Topology


(MSExchangeADTopology)

Provides Active Directory topology information to several Exchange Server components This service does not have any dependencies EDGE not dependent

Microsoft Exchange Monitoring

(MSExchangeMonitoring)

Provides a remote procedure call (RPC) server, used to invoke diagnostic cmdlets.

Microsoft Exchange Transport Log Search


(MSExchangeTransportLogSearch)

Provides message tracking and transport log searching UM not dependent

Microsoft Confidential

Exchange Client Access Role Services

Microsoft Exchange File Distribution Service (MSExchangeFDS)


Used to distribute offline address book Dependent upon AD Topology Service

Microsoft Exchange IMAP4 (MSExchangeIMAP4)


Provides IMAP4 services to IMAP clients Dependent upon AD Topology Service

Microsoft Exchange POP3 (MSExchangePOP3)


Provides POP3 services to POP3 clients Dependent upon AD Topology Service

Microsoft Exchange Service Host (MSExchangeServiceHost)


Config of RPC virtual directory in IIS Registry Config or ValidPorts, NSPI Interface Protocol Sequences, and AllowAnonymous for Outlook Anywhere Dependent upon AD Topology Service
6 Microsoft Confidential

Exchange Hub Transport Role Services Microsoft Exchange Transport (MSExchangeTransport)


Provides SMTP server and transport stack Dependent upon AD Topology Service

Microsoft Exchange EdgeSync (MSExchangeEdgeSync)


Connects to ADAM instance on subscribed Edge Transport servers over secure LDAP If there are no Edge Subscriptions configured, this service can be disabled Dependent upon AD Topology Service

Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)


Auto download anti-spam filter updates

Microsoft Confidential

Exchange Edge Role Services

Microsoft Exchange Transport (MSExchangeTransport)


Provides SMTP server and transport stack

Microsoft Exchange ADAM (ADAM_MSExchange)


Stores configuration and recipient data on the Edge Transport server

Microsoft Exchange Credential Service (EdgeCredentialSvc)


Monitors credential changes in ADAM and installs the changes on the Edge Transport server

Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)


Auto download anti-spam filter updates

Microsoft Confidential

Exchange Mailbox Role Services Microsoft Exchange Information Store (MSExchangeIS)


Manages Exchange Server databases Provides data storage for messaging clients Dependent upon : Event Log, NT LM Security Support Provider, Remote Procedure Call (RPC), Server, and Workstation

Microsoft Exchange System Attendant (MSExchangeSA)


Provides monitoring, maintenance, and directory lookup services Dependent upon: Event Log, NT LM Security Support Provider, Remote Procedure Call (RPC), Server, and Workstation

Microsoft Exchange Mail Submission Service


(MSExchangeMailSubmission)

Notifies Hub Transport server located in the Mailbox server's Active Directory to pickup from a sender's outbox Dependent upon AD Topology Service

Microsoft Confidential

Exchange Mailbox Role Services


(MSExchangeMailboxAssistants)

cont

Microsoft Exchange Mailbox Assistants


Provides functionality for Calendar Attendant, Resource Booking Attendant, Out of Office Assistant, and Managed Folder Mailbox Assistant. Dependent upon AD Topology Service

Microsoft Exchange Replication Service (MSExchangeRepl)


Provides log shipping functionality for LCR, CCR and SCR. Dependent upon AD Topology Service

Microsoft Exchange Service Host (MSExchangeServiceHost)


Config of RPC virtual directory in IIS Registry Config or ValidPorts, NSPI Interface Protocol Sequences, and AllowAnonymous for Outlook Anywhere Dependent upon AD Topology Service

10

Microsoft Confidential

Exchange Mailbox Role Services

cont

Microsoft Exchange Search Indexer (MSExchangeSearch)


Provides content to the Microsoft Search (Exchange Server) service for indexing. Dependent upon AD Topology Service and the Microsoft Search (Exchange Server) service.

Microsoft Search (Exchange Server) (MSFTESQL-Exchange)


Provides full-text indexing of mailbox data content Exchange-customized version of Microsoft Search Dependent upon the Remote Procedure Call (RPC) service

11

Microsoft Confidential

Exchange Unified Messaging Role Services

Microsoft Exchange File Distribution Service (MSExchangeFDS)


Distribute custom Unified Messaging prompts Dependent upon AD Topology Service

Microsoft Exchange Speech Engine (MSS)


Provides speech processing services Dependent upon Windows Management Instrumentation service

Microsoft Exchange Unified Messaging (MSExchangeUM)


Provides Unified Messaging features: Storing inbound faxes and voice mail messages Access to mailbox via Outlook Voice Access Dependent upon AD Topology Service and the Microsoft Exchange Speech Engine service

12

Microsoft Confidential

Active Directory Provider

What is the AD Provider? Components Who uses the AD Provider? Exchange Active Directory Topology Service AD Topology vs Exchange Topology

13

Microsoft Confidential

Active Directory Provider

The majority of components and services in Exchange 2007 are built on managed code:
Replay Service Exchange Transport Service Mailbox Assistants Search Unified Messaging

Unmanaged components
Information Store Service System Attendant DAV

14

Microsoft Confidential

Active Directory Provider: What is the AD Provider?

What is the AD Provider?


New component in Exchange 2007 that leverages the advantages of being built on managed code Provides an efficient, robust mechanism for managed applications to communicate with Active Directory Loaded by all managed applications Unmanaged applications continue to load DSAccess.dll

Improvements over DSAccess include


Support for paged queries Support for very large multivalued attributes Does not access and store directory information in a cache

15

Microsoft Confidential

Active Directory Provider: Components


AD Recipient Objects AD System Configuration Objects AD Driver
The engine inside the AD Provider Provides the following functions: Determines which server to connect to Maintains connections pools Performs connection failover Manages Exchange and AD topology discovery CRUD operations against AD Provides interfaces to recipient and system configuration objects

16

Microsoft Confidential

Active Directory Provider: Who uses the AD Provider?


Who uses the AD Provider?
Microsoft Exchange EdgeSync Microsoft Exchange File Distribution Microsoft Exchange Service Host Microsoft Exchange Transport Microsoft Exchange Transport Log Search Microsoft Exchange Replication Service Microsoft Exchange Mail Submission Microsoft Exchange Mailbox Assistants Microsoft Exchange Search Indexer Microsoft Exchange Unified Messaging Exchange Management Console Exchange Management Shell Setup
17 Microsoft Confidential

Active Directory Provider: Exchange AD Topology Service


Unmanaged Windows service that provides an RPC server interface in order to allow managed code processes access to AD topology information maintained by DSAccess Effectively a wrapper for DSAccess that makes specific DSAccess functions available via RPC to the AD Driver running within managed code process Dependency service for all managed applications

18

Microsoft Confidential

Active Directory Provider: AD Provider and AD Topology Service


AD Provider and AD Topology Service

19

Microsoft Confidential

Active Directory Provider: AD Topology vs Exchange Topology


Active Directory Topology
List of DCs and GCs in the local site and closest sites Details about each server: Is it reachable via ping? Is it a DC or GC? Is it synchronized? Which domain it lives in? OS version SACL Right This data is used in failover and load balancing

Exchange Topology
AD sites, site links and costs Subnets VDirs Location of Exchange servers Examples of use: Mail Routing Mapping of Client Access, Hub Transport and Unified Messaging server to the appropriate Mailbox server PF referrals

20

Microsoft Confidential

Troubleshooting Client Access Server (CAS)

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

22

Microsoft Confidential

Introduction What CAS is used for?

Outlook 2007 Clients only

AutoDiscover Service Availability Service Calendaring/Scheduling Assistant OOF configuration UM configuration

Previous Version Clients

Outlook Anywhere Exchange ActiveSync POP3/IMAP4 Outlook Web Access

23

Microsoft Confidential

Introduction CAS Role


At least one in each AD site that contains Mailbox server role
It is NOT supported to work with no CAS server

CAS role can be combined with any other role except:


Edge Transport Server role A server in a cluster

25

Microsoft Confidential

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

26

Microsoft Confidential

Client Access Server - Overview


Outlook Web Access Outlook Express Windows Mobile Outlook 2007 (via Outlook Anywhere) Outlook 2003/2007 (via MAPI)

IMAP4/POP3

HTTPS HTTPS HTTPS

Outlook Anywhere Encrypted RPC (MAPI)

RPC (MAPI)

SMTP

Client Access server roles

Site A

Encrypted RPC (MAPI) Encrypted RPC (MAPI) Mailbox server roles SMTP over TLS

Client Access server roles

Mailbox server roles Encrypted RPC (MAPI)

Site B

Hub Transport server roles

Hub Transport server roles

27

Microsoft Confidential

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

29

Microsoft Confidential

Locating CAS Configuration and Topology data


Critical Data Microsoft Office Outlook Web Access Web Site and Web.config file IMAP4 and POP3 protocol settings Availability service Location File System \Client Access\Owa

File system \ClientAccess\PopImap Active Directory configuration container and file system, including the Web.config file \ClientAccess\exchweb\ews IIS metabase Active Directory configuration container Active Directory configuration container File system, including the Web.config file in the \ ClientAccess\Sync folder IIS Metabase Active Directory configuration container and file system \ClientAccess\ IIS metabase
Microsoft Confidential

Autodiscover Exchange ActiveSync

Outlook Web Access virtual directories Web services configuration


30

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

31

Microsoft Confidential

Autodiscover
What Autodiscover does:
Automatically configure Outlook profiles without knowing where the mailbox is located Provide Web Service URLs to Outlook 2007 Clients Use both RPC and HTTPS connection

What Autodiscover doesnt configure:


Cache mode vs. Online mode Security Settings Remote Mail Settings Anything under Tools/Options in Outlook

32

Microsoft Confidential

Service Connection Point


CAS Installation will create:
A new virtual directory called AutoDiscover in IIS Used by Outlook 2007 Clients only Service Connection Point (SCP) in Active Directory which contains authoritative list of AutoDiscover Service URLs CN=<CAS_server>,CN=AutoDiscover,CN=Protocols,CN=<CAS_Server>,CN=Ser vers,CN=Exchange Administrative Group,CN=AdministrativeGroup, CN=<Organization>,CN=Services,[Configuration Naming Context]

SCP Objects are accessed by domain joined Outlook 2007 clients to locate AutoDiscover service Non domain joined clients rely on DNS to locate AutoDiscover service

33

Microsoft Confidential

Autodiscover
Configuring Outlook 2007 profiles and Web services URLs
1
Outlook uses e-mail address to locate an Exchange Client Access servers at a pre-defined location (autodiscover.domain.com)

Outlook 2007 John@contoso.com

HTTP Request

XML Config

Configuration settings are downloaded by Outlook and applied to profile

autodiscover.contoso.com AutoDiscover Service


Exchange captures Outlook request, and builds specific connection settings for Outlook

Client Access server role

Outlook Anywhere settings Server locations Web service URLs Authentication information OAB download location

AD Lookup 0

34

Microsoft Confidential

AD

Configuration Information

If domain joined, Outlook automatically fills out the users email address and password

Locating Autodiscover
To locate AutoDiscover:
Internal domain joined clients use SCP Non domain joined or external clients use DNS

For Outlook Anywhere and remote clients an host record for Autodiscover server should be created on an external DNS Without AutoDiscover access client can access mailbox but certain functions like F/B, OOF, OAB and UM will not be accessible If AutoDiscover is located via DNS, Outlook will try predetermined order of URLs to connect to AutoDiscover Server. For example:
https://domain.com/autodiscover/autodiscover.xml https://autodiscover.domain.com/autodiscover/autocover.xml

36

Microsoft Confidential

New DNS SRV Record for Locating Autodiscover Service


Predefined URL method requires valid SSL certificate for URLs being used Generally different DNS names are used for Outlook Anywhere and OWA HTTP Redirect needs additional Web Site in IIS and two Public IP Addresses New Software for Outlook performs and additional check for DNS SRV record for Autodiscover Service This feature is available as part of following update rollup for Outlook : (http://support.microsoft.com/kb/939184/) Description of the update rollup for Outlook 2007: June 27, 2007

38

Microsoft Confidential

Troubleshoot Autodiscover

Client side
Test E-mail Autoconfiguration Outlook Logging

Server side
Test-OutlookWebServices Event Logs Exchange Management Shell

40

Microsoft Confidential

Troubleshoot Autodiscover Client.

Results tab: Web service URLs Log tab: URLs used and error codes Popular error codes
80072EE7 ERROR_INTERNET_NAME_NOT_RESOLVED 80072EFD ERROR_INTERNET_CANNOT_CONNECT 80072F17 ERROR_INTERNET_SEC_CERT_ERRORS

Outlook logging
OLKDISC.log in temp directory OLKAS directory
41 Microsoft Confidential

Troubleshoot Autodiscover Test-OutlookWebService


Examples of failures:
When using self signed certificates

A DNS issue or a general server performance problem

The AutoDiscover XML file is missing.

42

Microsoft Confidential

Troubleshoot Autodiscover Server Event Logs

Event logging
Three MSExchange AutoDiscover event categories
\Core \Provider \Web Set-EventLogLevel "MSExchange AutoDiscover\Core" Level:Expert

43

Microsoft Confidential

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

44

Microsoft Confidential

Availability Service
Calendaring functionality for free/busy, meeting suggestions and Out-of-Office (OOF) depends on Availability Web Service Availability Service is used only by Exchange 2007 Mailboxes For Exchange 2007 Mailboxes, Calendar data will be read from users mailbox directly OL 2007/Exchange 2007 users access Exchange 2003 mailbox free/busy data by using Availability Service to look up free/busy Public Folders on Exchange 2003 Servers

45

Microsoft Confidential

Availability Service Cross Forest Access


To access cross-forest free/busy data, make sure free/busy information is replicated between forests To see free/busy data of Exchange 2003 mailboxes in the other forest, configure the availability service by running the following command on any server in the Exchange 2007 forest: Add-AvailabilityAddressSpace -ForestName:<forest name e.g. msft.com> -AccessMethod:PublicFolder

46

Microsoft Confidential

Troubleshooting Availability Service


In Outlook 2007, Ctrl-Right-click on the Outlook system tray icon. Enter your email address and password

47

Microsoft Confidential

Troubleshooting Availability Service, cont.


There are two ways to check if Availability service is functioning correctly: Event Log
4001 The Availability Service could not discover an Availability Service in the remote forest 4003 Public Folder Request Failed 4004 Unable to find a public folder server for the organizational unit 4005 Could not find information in Active Directory to allow cross-forest requests 4011 Cross-forestRequestFailed

Test-OutlookWebServices Cmdlet
Test-OutlookWebServices -id:user1@contoso.com -TargetAddress: user2@contoso.com

48

Microsoft Confidential

Troubleshooting Availability Service, cont.


OWA vs Outlook 2007
OWA runs against the Availability Service API(s) on the CAS server OUTLOOK 2007 runs against the Availability Service Web Service and relies on the Autodiscover service to find the Availability URL Mostly, free/busy problems in Outlook 2007 might be related with configuration of Autodiscover rather than Availability Service

Following commands can be used to get more information:


Get-webservicesvirtualdirectory Get-WebServicesVirtualDirectory -Identity EWS(default web site) Get-WebServicesVirtualDirectory -Identity CAS01

49

Microsoft Confidential

Test-OutlookWebServices
Test-OutlookWebService is a diagnostic task to verify AutoDiscover , Availability Service, RPC/HTTP and OAB distribution configuration for connectivity only
Test-OutlookWebServices -Identity <Alias, Domain\User or SMTP address> -ClientAccessServer <FQDN or NetBIOS name> -TargetAddress <Alias, Domain\User or SMTP address>

Scope can be set for:


For an Individual User: -Identity <Alias, Domain\User or SMTP address> For a specific CAS Server: -ClientAccessServer <FQDN or NetBIOS name> Free/Busy queries: -TargetAddress < Alias, Domain\User or SMTP address>

Returns information about SSL Certificate problems Determines the validity of the returned service URLs The request is made for one day of free busy data and the data is not returned in the task output.
50 Microsoft Confidential

Test-OutlookWebServices Basic Functionality


Step 1: Get a user context Step 2: Determine the Autodiscover URL Step 3: Submit an Autodiscover request Step 4: Validate that services exist Step 5: Return results of all tests to the console in the form of events.

52

Microsoft Confidential

Troubleshooting Free/Busy using Outlook 2007 Logging


Outlook 2007 can be used to troubleshoot problems with the Autodiscover service The Availability service log files are located in the \Documents and Settings\ <username> \Local Settings\Temp folder Three Log types:
OOF (Out of Office) MS (Meetings Suggestions) FB (Free/Busy)

Example:
20070305-110303994-fb.log

54

Microsoft Confidential

Free/Busy Log Files


Generated each time a user is added to the meeting request from Scheduling Assistant tab or for each request sent from the client Includes GetUserAvailabilityRequest XML message There are only three blocks of interest that contain the detail information needed for diagnostics:

MessageText Contains information about the failure. ExceptionCode Contains the exception that caused the failure. ResponseCode Contains the web response code for the failure.

56

Microsoft Confidential

Troubleshooting Free/Busy specific failures


Mailbox Logon Failure : Check the status of the target user's mailbox to see if it is available
<MessageText>Mailbox logon failed., inner exception: Cannot open mailbox /o=Fourthcoffee/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=mod4user11.</MessageText> <ResponseCode>ErrorMailboxLogonFailed</ResponseCode>

Permissions Error : Engage the target mailbox owner to confirm calendar permissions
<MessageText>Caller does not have access to free busy data. </MessageText> <ResponseCode>ErrorNoFreeBusyAccess</ResponseCode>

57

Microsoft Confidential

Troubleshooting Free/Busy specific failures cont.


Proxy Failures : The configuration need to enable cross-forest sharing of free/busy information is incomplete or missconfigured
<MessageText>The proxy request failed because the remote server returned an error <ResponseCode>ErrorProxyRequestProcessingFailed</ResponseCode>

Legacy Free/Busy Failures : Public folder store on Exchange 2003 is not mounted or inaccessible
<MessageText>The remote server returned an error: (503) Server Unavailable.. <ResponseCode>ErrorPublicFolderRequestProcessingFailed</ResponseCode >

58

Microsoft Confidential

Availability Service Error Codes


Description of Exchange 2007 Availability Service Error Codes
RequestStreamTooBig = 5000 IdentityArrayEmpty = 5001 IdentityArrayTooBig = 5002 TimeIntervalTooBig = 5003 InvalidMergedFreeBusyInterval = 5004 ResultSetTooBig = 5006 InvalidClientSecurityContext = 5007 MailboxLogonFailed = 5008 MailRecipientNotFound = 5009 InvalidTimeInterval = 5010 PublicFolderServerNotFound = 5011 InvalidAccessLevel = 5012 And more
60 Microsoft Confidential

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

61

Microsoft Confidential

Offline Address Book (OAB)


Used for offline GAL view and cached mode GAL lookups New OAB uses HTTPs & BITS (Background Intelligent Transfer Service)
Legacy support available by using public folders distibution

Relies on:
OABGen Exchange File Distribution OAB Virtual Directory Autodiscover

BITS Client does not support self-signed certificates, so by default OAB Distribution Points use HTTP SSL can be enabled with a fully trusted certificate in IIS

62

Microsoft Confidential

Offline Address Book - Process

64

Microsoft Confidential

Troubleshooting Offline Address Book Legacy clients


Need to publish OAB to public folders Need OAB Public folder replication

Outlook 2007
Make sure Autodiscover works and the URLs are correct Check the OAB distribution on nearest CAS server Check IE proxy settings (KB939765)

Non client specific issues


GAL deleted No permissions
66 Microsoft Confidential

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

68

Microsoft Confidential

SSL handshake
SYNC (TCP_Port = 443)
Outlook clicks on URL to access Secure Webmail Server (https://mail.msft.com)

SYNC + ACK ACK


SSL Handshake on the new TCP connection

The browser establishes a TCP connection on the HTTPS TCP Port 443

CLIENT_HELLO SERVER_HELLO CERTIFICATE


Outlook Web Access

SERVER_DONE

Client Access Server

To continue with the authentication process, client should verify the servers certificate

1 2 3 4

Is today's date within the validity period? Is the issuing Certificate Authority (CA) a trusted CA? Does the issuing CA's public key validate the issuer's digital signature? Does the domain name in the server's certificate match the domain name of the server itself? 5 The server is authenticated.
Microsoft Confidential

69

Certificates and Subject Alternate Name


By default during CAS installation a new self signed certificate will be generated and assigned to the Default Web Site for encrypted HTTP communication The authenticity of self signed certificate cannot be verified during Internet access For Internet access to CAS Servers, it is recommended to use a 3rd party CA for authenticity to work However the certificate which was assigned to company name may not match the CAS Web Address Subject Alternate Names can be used as an alternative web address for an existing certificate to override that problem

71

Microsoft Confidential

Certificate Request

72

Microsoft Confidential

Self-signed Certificates & Outlook Anywhere


If you use self-signed certificates, OL will not successfully connect using HTTPS by using default settings pushed by Autodiscover
By default, the CertPrincipalName parameter for OutlookProvider is not configured OL uses the ExternalHostname parameter for OutlookAnywhere to populate the server name listed after MSSTD: By default, the ExternalHostname parameter will not match the default Issued To value on the selfsigned certificate

Client Access Server

Outlook Anywhere

Outlook Provider CertPrincipalname = Null

74

Microsoft Confidential

Troubleshooting Self-signed Certificates


There are two methods to solve self-signed certificate problems with Outlook Anywhere:
Get a new certificate where the Issued To property matches the Certificate Principal Name -ORChange the CertPrincipalName Value on OutlookProvider: set-outlookprovider -identity EXPR -server 'owatest.mail.msft' CertPrincipalName 'msstd:owatest.mail.msft'

This allows OL2007 to get complete the Autodiscover phase of Outlook Anywhere profile creation Domain-joined clients do not display Invalid CA certificate warnings
Note: Self signed certs would generate warnings for end user and we recommend our customers to buy the required certificates before deploying CAS for the end-users.
75 Microsoft Confidential

Certificate Issues for Combined CAS&HUB Role


Events 1037 and 2019 if third party certificate
Is not enabled for SMTP Service on CAS/Hub Server andIncludes Netbios name in the Certificate Request

To fix this:
If the domain name parameter includes Netbios or Server FQDN in the certificate request, the certificate should be enabled for SMTP service. Run Enable-ExchangeCertificate command to enable for SMTP Alternatively, do not use Netbios or Server FQDN in the certificate request, use only Public FQDN

76

Microsoft Confidential

Common Certificate Issues

Using the self signed certificate


The certificate common name is the server NetBIOS name There is no automatic way to make the self signed certificate trusted

Using the old exchange 2003 certificate


Does not have Autodiscover URL

Using new certificate without considering Autodiscover ISA server is not publishing Autodiscover URL

78

Microsoft Confidential

Troubleshooting CAS Security & Certificate Issues


Tools Components Get-ExchangeCertificate Command used to view certificates from the
local certificate store

EXTRA Tracing

Enable the following components/tags: Common\Certificate Validation Networking Layer\Certificate Transport\Certificate Tool to troubleshoot transport security problems Used to troubleshoot authentication errors while accessing web service or any other web page Unofficial utility to dump the msExchServerInternalTLSCert value in AD into a readable format

Protocol Logging IIS Log Files Access Certlib

80

Microsoft Confidential

Outlook Anywhere Authentication methods after SP1


Exchange 2007 RTM
Mandatory Parameter: ExternalAuthenticationMethod Used to update OL2007 clients using Autodiscover service Basic and NTLM Authentication methods were always reenabled on /rpc virtual directory regardless of this parameter

Exchange 2007 SP1


Ability to choose the authentication methods New parameters:

ClientAuthenticationMethod IISAuthenticationMethods DefaultAuthenticationMethod

set-OutlookAnywhere -IISAuthenticationMethods <Basic or NTLM>

82

Microsoft Confidential

LAB 1 Troubleshooting Certificates


Exercise 1: Introduction to SSL Certificates Exercise 2: Subject Alternative Names (SAN)

85

Microsoft Confidential

LAB 2 Troubleshooting Autodiscover


Exercise 1: Understanding the AutoDiscover Service Exercise 2: Configuring the AutoDiscover Service for use by external Outlook clients

86

Microsoft Confidential

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

87

Microsoft Confidential

Outlook Web Access Overview


Client Access Server
Doc. HTML transcoder SpellCheckers Active Directory Driver CAS business logic
Exchange 2007 Mailbox Active Directory

/owa

HTTP to Exchange 2007 OWA

IIS SSL
/exchange /exchweb /public

OWA Auth ISAPI

OWA 2007 rendering

SharePoint & File Shares

HTTP to Exchange 2003 OWA & WebDAV

Outlook Web Access Proxy

Exchange 2003 Mailbox OWA 2003 rendering

88

Microsoft Confidential

Authentication Methods
Authentication Method Basic authentication Digest authentication Security Level Low (unless Secure Sockets Layer [SSL] is enabled) Medium How Passwords Are Sent Base 64-encoded clear text Client Requirements All browsers support Basic authentication Microsoft Internet Explorer 5 or later versions Internet Explorer 2.0 or later versions for Integrated Windows authentication; Microsoft Windows 2000 Server or later versions with Internet Explorer 5 or later versions for Kerberos

Hashed Hashed when Integrated Windows authentication is used Kerberos ticket when Kerberos is used Integrated Windows authentication includes the Kerberos and NTLM authentication methods Uses cookies to help secure a user's name and password

Integrated Windows authentication

High

Forms-based authentication

High

Internet Explorer

90

Microsoft Confidential

Troubleshooting OWA FBA Login


OWA Client CAS Server

Anonymous GET /owa

Intercept
Owaauth .dll

Redirect to owa/auth/logon.asp
Anonymous GET /owa/auth/logon.asp Return FBA logon page POST including username + password Redirect to /owa + set Auth Cookie
Auth Cookie

Owaauth .dll Set Cookie Owaauth .dll


Authenticated

Authenticated Request GET /owa


Auth Cookie

91

Microsoft Confidential

Troubleshooting OWA: Tools


Exchange Management Shell (EMS)
Get-OwaVirtualDirectory (Set/Remove/New) Test-OwaConnectivity Cmdlet Test-OwaConnectivity ClientAccessServer:ServerName Test-OwaConnectivity -URL:https://mail.domain.com/owa MailboxCredential:(get-credential DomainName\AccountName) Get-CASMailbox (Set) Get-casmailbox UserName | fl owa*

Exchange Managment Console (EMC)

Note: Looking for a specific command? Use Get-Help with correct wildcards Example: get-help *OWA*
93 Microsoft Confidential

Troubleshooting OWA: Tools cont.


Internet Information manager
MSExchangeOWAAppPool must be started using Local System identity Web Services Extensions must be enabled for: ASP.NET (), Microsoft Exchange Client Access Server (owaauth.dll)
CAS Only: Microsoft Exchange Server (exprox.dll) CAS + MBX: Microsoft Exchange Server (davex.dll)

Check Mapped application for Legacy Virtual Directories (/Exchange, /Public and /EXCHWEB)
CAS Only: exprox.dll CAS + MBX: davex.dll

Check Authentication settings (change using EMC or EMS)


Anonymous for owa/auth folder

95

Microsoft Confidential

Troubleshooting OWA: Configuration


Check Configuration files
Forms Registry file: Registry.xml Web.Config

Registry Keys
Disable LDAP Encryption (Troubleshooting ONLY)
Key: HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeADAccess DWORD: Disable LDAP Encryption Value: 1 = LDAP Encryption disabled

96

Microsoft Confidential

Troubleshooting OWA: Microsoft Fiddler HTTP debugger


Fiddler is a HTTP Debugging Proxy which logs all HTTP traffic
Internet Explorer Plug-In View HTTP(S) traffic (real-time) HTTP(S) Statistics Capturing web traffic logs Session Inspector RAW view of HTTP traffic And many more Download available at http://www.fiddlertool.com/fiddler/

IMPORTANT: Fiddler is a Client Side debugging Tool and should never be installed onto a production Exchange server. Serious problems have been reported with Active Sync when Fiddler is installed on an Exchange 200x server!
98 Microsoft Confidential

Understanding Proxying and Redirection


OWA Internet
Note: If the mailbox is on E2K3, the CAS will proxy directly to the back-end. Integrated Windows authentication for /Exchange and /Microsoft-ServerActiveSync virtual directories must be enabled

EAS

Execute request on this CAS

Yes

CAS in Users mailbox AD Site

No
OWA/EAS not available

No

Find a CAS In Mailbox AD Site

Yes
Best CAS has ExternalURL Set?

Note: CAS to CAS proxying is not supported between virtual directories that use Basic authentication, the virtual directories must use Integrated Windows authentication.

Yes
Note: Redirection is only supported for OWA
99

No

REDIRECT Client using ExternalURL


Microsoft Confidential

PROXY request to CAS using InternalURL

Understanding Proxying and Redirection - cont.


CAS->MBX comm. Between AD sites

Redirection CAS->CAS Yes


Unnecessary: Autodiscover

CAS-> proxying between AD sites Yes Yes

Comments/ Consequences

OWA EAS Web Services used by 3rd party LOB applications Availability Service used by Outlook 2007 Outlook Anywhere WebDAV and OWA 2000/2003

No No

No

No

Yes

Must have a CAS server in each Exchange AD site to use OWA/EAS/Web Services

No

Unnecessary: Autodiscover
Unnecessary: Autodiscover

Yes

Yes, RPC

Not applicable Proxying to legacy E2003 server IMAP/POP clients must access a CAS in the mailbox AD Site directly

Yes, HTTP

No

Not applicable

IMAP4/POP3

No

No

No

101

Microsoft Confidential

CAS Proxy Scenarios


Between Exchange 2007 Client Access Servers
Internet facing CAS proxy requests to other CAS with no Internet presence Known as CAS-CAS proxying

Between an Exchange 2007 Client Access Server and an Exchange Server 2003 Back-end server when:
OWA Clients connect to /Exchange virtual directory EAS Clients connect to /Microsoft-Server-ActiveSync virtual directory

103

Microsoft Confidential

Issues with Coexistence and DAVEX


When CAS and Mailbox roles are combined together, OWA clients are prompted for credentials two times or redirected to a different server to access an Exchange 2003 mailbox For script mapping, /Exchange virtual directory will use Davex.dll instead of Exprox.dll Davex.dll cannot act as a proxy for mailbox requests Instead it redirects the requests to the Exchange 2003 Mailbox Server based on the internal (intranet) name of the server External users will get DNS errors if internal name is not exposed to Internet

104

Microsoft Confidential

Troubleshooting Proxying - Redirection


Verify Configuration using Exchange Management Shell
InternalURL and ExternalURL values RedirectToOptimalOWAServer=$True Certificates SelfSigned/Public/Private vs

Set Diagnostic Logging for MSExchangeOWA\Proxy


set-EventLogLevel -Identity "ServerName\MSExchange OWA\Proxy" -Level "Expert"

106

Microsoft Confidential

Proxying Performance and Scalability & Debugging


ASP.NET Proxying performance and scalability (KB821268) Debug Registry Keys:
Allow Proxying without SSL
Registry Key: AllowProxyingWithoutSSL = 1

Allow Proxying only using Trusted Certificate


Registry Key: RequireTrustedCertForProxying = 1

108

Microsoft Confidential

LAB 3 Troubleshooting Proxying


Exercise 1: CAS to CAS proxying Exercise 2: Configuring OWA Redirection

110

Microsoft Confidential

Web Ready Document Registry Keys


Set in the registry of the Client Access server under:
HKLM\SYSTEM\CurrentControlSet\Services\MSExchange OWA

Value Name RecycleByConversions ExcelRowsPerPage MaxDocumentInputSize MaxDocumentOutputSize TempFolderLocation CacheDiskQuota ConversionTimeout

Value Type DWORD DWORD DWORD DWORD String DWORD DWORD

Value #>0 #>0 1000000 > # > 0 1000000 > # > 0 Valid Path #>0 #>0

Default Value 1000 200 5000 (in KB) 5000 (in KB) %SYSTEMROOT%\Temp 1000 (in MB) 20 (seconds)

111

Microsoft Confidential

Troubleshooting WebReady Document Viewing


Exchange Management Shell (EMS)
Get-OwaVirtualDirectory (Set/Remove/New) Get-CASMailbox (Set)

Exchange Managment Console (EMC) ADSIEdit


CN=owa (Default Web Site),CN=HTTP,CN=Protocols, CN=<ServerName>, CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=<Exchange Organization>,CN=Microsoft Exchange,CN=Services, CN=Configuration,DC=<Domain>,DC=<Domain>

113

Microsoft Confidential

LAB 4 Troubleshooting OWA


Exercise 1: Examining the OWA login process (Forms based Authentication) Exercise 2: WebReady Document viewing

114

Microsoft Confidential

Agenda
Introduction Overview Locating CAS Configuration and Topology Data Troubleshooting:
Autodiscover Availability Service Offline Address Book Client Access Security Outlook Web Access Exchange ActiveSync

115

Microsoft Confidential

Exchange ActiveSync Overview


Synchronization protocol optimized to work with high-latency and low-bandwidth networks Based on HTTP and XML Enables mobile device users to access their e-mail, calendar, contacts and tasks Enhanced in Exchange Server 2007:
Support for HTML messages Support for follow-up flags Support for fast message retrieval Meeting attendee information Enhanced device security through password policies

Enhanced Exchange Search

Windows SPS and LinkAccess document access

PIN reset

AutoDiscover for over the air provisioning


116

Support for OOF configuration


Microsoft Confidential

Support for tasks synchronization

Troubleshooting Exchange ActiveSync


Coexistence Exchange 2003 Back end authentication (KB937031) Read IIS logs
Use logparser

Certificates Use device emulator EXTRA Logging

117

Microsoft Confidential

All Troubleshooting Tools


IIS Logs LDAP tools (LDP, ldifde, adsiedit, etc ) Event Logging Performance Monitoring Exchange Troubleshooting Assistant (Supervised Trace Logs) Browser Outlook logging

118

Microsoft Confidential

Additional resources
Support WebCast: Introduction to AutoDiscover in Microsoft Exchange Server 2007 http://support.microsoft.com/kb/935438 White Paper: Exchange 2007 Autodiscover Service http://technet.microsoft.com/en-us/library /59adba4e-44e1-4aa2-b09d-06988cbeab2d.aspx Autodiscover and Exchange 2007 http://technet.microsoft.com/en-us/library /7c44814d-bb46-4fb8-9b6b-a082be35afdc.aspx Managing the Autodiscover Service http://technet.microsoft.com/en-us/library/aa995956.aspx Exchange 2007 Autodiscover and certificates http://msexchangeteam.com/archive/2007/04/30/438249.aspx More on Exchange 2007 and certificates - with real world scenario http://msexchangeteam.com/archive/2007/07/02/445698.aspx Exchange 2007 Offline Address Book Web Distribution http://msexchangeteam.com/archive/2006/11/15/431502.aspx
120 Microsoft Confidential

Troubleshooting Hub Transport Server (HT)

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
122

Local Delivery & Mail Submission: Architecture


Local Delivery
Process by which messages are delivered from a Hub Transport server to mailbox on a local Mailbox server

Mail Submission
Process of picking up a message from a users mailbox and getting it into the Submission queue on a local Hub Transport server

Multiple components involved


Exchange System Objects (XSO) Microsoft Exchange Mail Submission Service Store Driver

123

Local Delivery & Mail Submission: Architecture


Store Driver
Runs on Hub Transport Server Submits and retrieves mail to and from a local Mailbox server Performs MAPI to MIME and MIME to MAPI conversion Two primary components Mail Submission RPC Server Used to accept New Mail Notifications from the Mail Submission Service on Mailbox Server XSO Used to submit and retrieve mail to and from a Mailbox Server using MAPI.Net over RPC Performs the MAPI to MIME and MIME to MAPI conversion

124

Local Delivery & Mail Submission: Architecture


Mail Submission Service
Runs on Mailbox server Notifies Hub Transport server of new messages for delivery Two Primary components Assistant Infrastructure Processes and manages events that occur within the Information Store Receives Event notifications about new messages for delivery Triggers Mail Submission Service to generate new message notifications for Store Driver Mail Submission RPC Client Connects to Mail Submission RPC Server on Store Driver Submits New Mail Notifications generated by Mail Submission Service

125

Local Delivery & Mail Submission: Local Delivery


Local delivery process
One Queue per Mailbox Server Does not require Mailbox Store to run on local Mailbox is in local AD Site server Does not use hidden SMTP Mailbox for conversion No Store Driver on Mailbox Server
Hub Transport Role EdgeTransport.exe Local Delivery Queue CAT

Dumpster

Submission Queue

MailSubmission RPC Server

Message submission is completed


XSO

StoreDriver

Convert to MAPI

RPC

RPC

Mailbox Role MailSubmissionSvc.exe Store.exe

MailSubmission RPC Client Assistant Infrastructure (AI)

MAPI. Net MAPI

JET

Outlook Client

STORE

STORE

STORE

126

Local Delivery & Mail Submission: Mail Submission


Mail Submission process
Hub Transport Role EdgeTransport.exe Local Delivery Queue CAT

Dumpster

StoreDriver Submission Queue

Move message from Outbox to Sent Items


XSO

MailSubmission RPC Server

Convert to MIME

RPC

RPC

Mailbox Role MailSubmissionSvc.exe Locate Hub Store.exe

Server
Mailbox Server DN, Senders MB MailSubmission GUID, Messages RPC Client EntryID, etc

MAPI. Net MAPI

New Assistant Message Infrastructure (AI) Notification


JET

Outlook Client

STORE

STORE

STORE

127

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
128

Local Delivery & Mail Submission: Troubleshooting


Run the Exchange Mail Flow Troubleshooter Check Queues on the Hub server Run Test-ServiceHealth on Mailbox and Hub server Check event logs on Mailbox and Hub Server Test basic RPC connectivity between Hub and Mailbox server Enable connectivity logging Verify AD site configuration Verify DNS configuration Check permissions for the Exchange Servers group on the Mailbox server object

129

Local Delivery & Mail Submission: Troubleshooting


Run the Exchange Mail Flow Troubleshooter
Available in the EMC Toolbox Analysis is symptom specific, so choice of correct symptom is critical to success
NDR Inbound Outbound Queue Submission EdgeSync

130

Local Delivery & Mail Submission: Troubleshooting


Check Queues on the Hub server
Use Queue Viewer or the Get-Queue cmdlet to check the Status and LastError fields of the Mailbox Delivery queue If Status is Retry then LastError should be an error message that can be used to help identify the problem

Run Test-ServiceHealth cmdlet on Mailbox and Hub server


Ensures all required services configured to start automatically have started Returns an error for any service that is required by a configured role and is set to start automatically but is not currently running

131

Local Delivery & Mail Submission: Troubleshooting


Check event logs on Mailbox and Hub Server
Check both Mailbox and Hub server for errors or warnings If necessary increase diagnostics logging using the registry or SetEventLogLevel Mailbox server: MSExchangeMailSubmission\General Hub Transport server: MSExchange Store Driver\General The possible logging levels that you can set are: 0 (Lowest), 1 (Low), 3 (Medium), 5 (High), 7 (Expert) Always return the logging level to the default setting after completing your troubleshooting activities

132

Local Delivery & Mail Submission: Troubleshooting


Test basic RPC connectivity between Hub and Mailbox server
Using RPCPing Two components Rpings.exe (server-side RPC ping utility) Rpingc.exe (client-side RPC ping utility) Verify RPC connectivity in both directions Using Test-MAPIConnectivity cmdlet Logs onto the system mailbox or a specified mailbox using the -Identity parameter Verifies that the MAPI server, Exchange store, and Directory Service Access (DSAccess) are working

133

Local Delivery & Mail Submission: Troubleshooting


Test-Mailflow
Tests mail submission, transport and delivery Tests services by verifying that each mailbox server can successfully send itself a message. Remote functionality to test between remote Mailbox Servers

134

Local Delivery & Mail Submission: Troubleshooting


Enable connectivity logging
When the Hub server is having problems sending to the Mailbox server Not enabled by default get-transportserver|set-transportserver -ConnectivityLogEnabled:$TRUE Provides summary information on outbound connections via SMTP or StoreDriver Much less verbose than Protocol logs Protocol logs more useful for SMTP conversation Useful when wanting to see connectivity issues with the sending StoreDriver as protocol logs do not capture this information Multiple events/connection, each using same connection ID for correlation

135

Local Delivery & Mail Submission: Troubleshooting


Verify AD site configuration
Ensure AD sites and subnet configuration is correct Run Nltest.exe /dsgetsite on Hub and Mailbox servers to verify they are in the same AD site Beware: Changing IP address of Hub server may cause it to fall under another AD site and therefore leave Mailbox server in an AD site with no other Hub servers

136

Local Delivery & Mail Submission: Troubleshooting


Verify DNS configuration
Correct AD site determination is heavily dependant on correct DNS configuration Verify Hub and Mailbox servers are configured with the correct internal DNS server Ensure correct AD information such as domain controller SRV and A records as well as correct AD site information is stored in DNS Use DCDIAG /TEST:DNS on domain controllers (use /X and /XSL to log to XML for easier reading) Ensure correct A records are stored in DNS for Hub and Mailbox servers Nslookup can be used to help verify the above

137

Local Delivery & Mail Submission: Troubleshooting


Check permissions for the Exchange Servers group on the Mailbox server object
Using ADSIEdit.msc verify that the Exchange Servers group has an explicit allow set on the Mailbox server object for Store Constrained Delegation Store Read and Write Access Store Read only Access Store Transport Access Ensure there are no explicit denies set as this will override the allow Hub server requires these permissions in order to retrieve messages from and submit messages to mailboxes

138

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
139

Remote Delivery: Architecture


Remote Delivery refers to messages delivered using SMTP to
Another Hub server in the organization An Edge server Another remote SMTP mail system

Can also be to a foreign mail system using the Drop directory

140

Remote Delivery: Architecture


DNS
Exchange 2007 uses an enhanced DNS client to resolve the next hop selection to a list of target server names Standard DNS client used to resolve list of server names to IP addresses Enhanced DNS also provides load-balancing for Hub servers by using round robin

SMTP
Used for communication when messages are relayed between SMTP servers

141

Remote Delivery: Architecture


Routing Tables
Holds information that the routing component uses to make routing decisions Composed of a map of topology components and their relationship to one another Linked connectors map Server map Legacy server map MDB map Active Directory site map Routing groups map Send connectors map

142

Remote Delivery: Routing Architecture


Routing Tables
Built every time that a transport server is started Recalculated when configuration changes are received Configuration changes can be detected in the following ways Active Directory change notifications Configuration reloading caused by service control commands Periodic reload to track changes that are not supported by Active Directory notifications Information in the routing tables is logged to routing logs C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\_ Routing New log generated every time routing tables are calculated If the Hub server is unable to contact AD routing will use currently cached routing data

144

Remote Delivery: Architecture


Viewing the Routing Table Log
Routing Log Viewer used to view the Routing Table Log New tool with Microsoft Exchange Server 2007 SP1 Can be used to Find the Lowest Cost Path to a Site Find the Preferred Connector for a Given Address Can open a second routing log and determine changes that have occurred within the routing topology between two time periods

145

Remote Delivery: Demo


In this demo your instructor will show you
How to use the Routing Log Viewer tool to Find the lowest cost path to a site Find the preferred connector for a given address space Compare two routing logs to identify changes

147

Remote Delivery: Architecture


Determining Site Membership
NetLogon service determines site membership for the computer upon startup Uses DNS queries to compare the local IP address to defined subnets Exchange AD Topology service retrieves the site membership value from the NetLogon service msExchServerSite attribute Value of this attribute is the DN of the AD site of an Exchange server Reduces overhead associated with DNS queries Also used to associate a non-domain computer, such as a subscribed Edge server to an AD site Populated and kept up to date by the Exchange AD Topology service

148

Remote Delivery: Architecture


Detecting Site Membership Changes
May occur due to IP address change or AD subnet association change Exchange 2007 must update its configuration data so that the change is considered when making routing decisions The NetLogon service polls frequently for changes in AD site membership The Exchange AD Topology service queries NetLogon regularly to determine the AD site membership of the local Exchange server and updates the MsExchServerSite attribute if necessary The Exchange servers in the organization update the routing tables with the new AD site membership attribute value

150

Remote Delivery: Architecture


Controlling IP Site Link Costs
By default, Exchange Server 2007 uses the cost that is assigned to the AD IP site links You can use the Set-AdSiteLink cmdlet to assign an Exchange-specific cost to an IP site link connector Exchange-specific cost is a separate attribute and overrides the AD assigned cost for the purpose of determining the Exchange routing path Allows the Exchange administrator to override existing links however it does not allow the ability to create links where none exist Useful when the AD IP site link costs do not result in an optimal Exchange message routing topology Using it for permanently overriding costs simply adds complexity Better to work with AD and network administrators

152

Remote Delivery: Architecture


Message size limit for IP site links
New to Exchange 2007 SP1 By default Exchange does not impose a maximum message size limit on messages that are relayed between Hub servers in different AD sites Useful when low-bandwidth connections to remote sites exist Use the Set-AdSiteLink cmdlet to configure Set-AdSiteLink -Identity <IP Site link name> -MaxMessageSize 10MB

153

Remote Delivery: Troubleshooting


Back Pressure
Back pressure is a system resource monitoring feature of the Exchange Transport service If utilization of a system resource exceeds the specified limit the server stops accepting new connections and messages Prevents the system resources from being completely overwhelmed and enables the server to deliver the existing messages When utilization returns to a normal level the server accepts new connections and messages For each monitored system resource the following three levels of resource utilization are applied: Normal Medium High

154

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
155

Troubleshooting Mail Flow


Back Pressure (Cont...)
System resources monitored Free space on the drive that stores the message queue database Free space on the drive that stores the message queue database transaction logs The number of uncommitted message queue database transactions that exist in memory The memory that is used by the EdgeTransport.exe process The memory that is used by all processes

156

Troubleshooting Mail Flow


Understanding Non-Delivery Reports
NDRs redesigned in 2007 to make them easier to read and understand Separated into user information and diagnostic information for admins

157

Troubleshooting Mail Flow


Understanding Non-Delivery Reports
The following fields are present in most NDRs Generating server Rejected recipient Remote server Enhanced status code SMTP response Original message headers Enhanced status codes Success is indicated by a 2.x.x status code Persistent Transient Failure is indicated by a 4.x.x status code Permanent Failure is indicated by a 5.x.x status code

158

Troubleshooting Mail Flow


Message Tracking Logs
Detailed log of all message activity as messages are transferred Enabled by default EventID describes tracking event action Source describes component involved SMTP, STOREDRIVER, ROUTING, AGENT Use Set-TransportServer or Set-MailboxServer cmdlets to configure

161

Troubleshooting Mail Flow


Message tracking log event types
BADMAIL DELIVER DEFER DSN EXPAND FAIL POISONMESSAGE

RECEIVE REDIRECT RESOLVE SEND SUBMIT TRANSFER

162

Troubleshooting Mail Flow


Filtering message tracking logs
Search depends on the Microsoft Exchange Transport Log Search service Get-MessageTrackingLog cmdlet or Message Tracking tool in EMC to filter and display results Get-MessageTrackingLog -sender user@domain.com -start 07/01/2007 09:00 AM -end 07/01/2007 09:30 AM In SP1 use GetMessageTrackingLogE2EwithTime.ps1 to search for specific entries in all message tracking logs on all Hub and Mailbox servers

163

Troubleshooting Mail Flow


Protocol Logs
SMTP protocol conversation without data Useful for diagnosing SMTP mail flow problems Use Set-ReceiveConnector or Set-SendConnector cmdlets or the EMC to enable

164

Troubleshooting Mail Flow


Troubleshooting mail flow issues between Hub servers
Use EXMFA with symptom matching the case Determine where message delivery failed or where a non-delivery report (NDR) is being generated by Using the Mail Flow Troubleshooter tool Using the Queue Viewer on a Hub server to determine where message delivery failed Checking the NDR to verify which server and component are generating the NDR Use message tracking in order to determine the path messages are taking and at what point delivery is failing Check the delivery status notification (DSN) error codes contained in the NDR and search the Microsoft support site for the error code Run the Test-ServiceHealth cmdlet on the Hub servers where message delivery failed or on the transport server that generated the NDR

165

Troubleshooting Mail Flow


Troubleshooting mail flow issues between Hub servers (Cont...)
Check the application event log on the Hub Transport servers that are involved in the delivery of the message Increase diagnostic logging levels on the Exchange processes that are generating errors if necessary (SMTP Send and Receive, RemoteDelivery, Routing etc) Verify that back pressure is not occurring Event ID 15001 and 15002 for RTM and 15004 and 15005 for SP1 Check Physical connectivity From the server where message delivery is failing ping the next hop servers by IP and FQDN and ensure a reply is received At the same time ensure that the correct IP is resolve Run netstat anb on the receiving Hub server and verify that MSExchangeTransport.exe is listening on port 25

166

Troubleshooting Mail Flow


Troubleshooting mail flow issues between Hub servers (Cont...)
From the server where message delivery is failing verify you can connect to SMTP port 25 of the next hop servers by using telnet Verify the necessary connectors are enabled and configured appropriately on the Hub servers involved Default receive connector configuration, Remote IP range, message size restrictions, authentication methods and types Verify AD site configuration Ensure AD sites and subnet configuration is correct Run Nltest.exe /dsgetsite on next hop Hub servers to verify they fall in the correct AD site Check msExchServerSite attribute using ADSIEdit.msc for next hop Hub server objects to ensure they are stamped with the correct AD site

167

Troubleshooting Mail Flow


Troubleshooting mail flow issues between Hub servers (Cont...)
Verify DNS configuration Verify Hub servers are configured with the correct internal DNS server Ensure correct information such as A records for next hop servers are stored in DNS Ensure correct AD information such as domain controller SRV and A records as well as correct AD site information is stored in DNS Use DCDiag /TEST:DNS on domain controllers Ensure the correct A records are stored in DNS for the Hub servers Nslookup can be used to verify the above Enable protocol logging for the Send connector on the sending Hub server and Receive connector on the receiving Hub server

168

Troubleshooting Mail Flow


Troubleshooting mail flow issues between Hub servers (Cont...)
Look for possible certificate issues If custom certificates are installed ensure they are enabled for the SMTP service using Get-ExchangeCertificate Use network monitor to capture network traffic between the sending and receiving server Test-Mailflow Tests mail submission, transport and delivery Tests services by verifying that each mailbox server can successfully send itself a message. Remote functionality to test between remote Mailbox Servers

169

Troubleshooting Mail Flow


Troubleshooting outbound internet mail flow
Follow the same steps as when troubleshooting mail flow issues between hub servers Ensure that an appropriate Internet send connector has been configured with a * address space When the send connector is configured to use DNS From the sending server use Nslookup to determine external email domains MX records Set type=MX From the sending server telnet to port 25 of the identified remote host and send a test message When the send connector is configured to use a Smart Host From the sending server telnet to port 25 of the Smart Host and send a test message

170

Troubleshooting Mail Flow


Troubleshooting inbound internet mail flow
Ensure that an authoritative accepted domain has been configured for the email domain and has public MX records registered Ensure that an email address policy has been enabled for the accepted domain and users have the email address applied Run netstat anb on the Hub or Edge servers responsible for receiving inbound internet mail and verify that MSExchangeTransport.exe is listening on port 25 Verify the public MX records registered for the receiving email domain using nslookup Set type=MX May need to point nslookup to an external DNS server server <External DNS Server IP> Using a machine on the internet telnet to port 25 of the server identified in the MX record and send a test message

171

Troubleshooting Mail Flow


Troubleshooting inbound internet mail flow (Cont...)
Ensure that any firewalls are configured to forward inbound internet messages onto the Exchange servers responsible for inbound internet mail Verify the necessary receive connectors are enabled and configured appropriately on the Hub or Edge servers involved
Default receive connector configuration, Remote IP range, message size restrictions, authentication methods and types (specifically anonymous connections)

For Hub servers default Receive connector will need to be modified to allow anonymous connections

172

LAB 5 Mailflow
173

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
174

Mail Queues: Architecture


Queues are temporary holding locations for messages that are waiting to enter the next stage of processing Each queue represents a logical set of messages that a transport server processes in a specific order Exist only on Hub Transport or Edge Transport server roles Exchange 2007 all messages stored in the same location All queues are stored in a single ESE database

175

Mail Queues: Architecture


ESE Database Queue files
ESE Database files centrally located on the server By default located at ...\Exchange Server\TransportRoles\data\Queue Circular logging is used Configuration options stored in EdgeTransport.exe.config
File
Mail.que Tmp.edb Trn*.log Trntmp.log Trn.chk Trnres00001.jrs & Trnres00002.jrs

Description
ESE Database file that stores all the queued messages Temp database file used to verify the queue database schema on startup Transaction logs that record all changes to the queue database Temporary transaction log created in advance Tracks the log entries that have been committed to the database Transaction reserve log files. Used when the hard disk drive that contains the transaction log runs out of space to stop the queue database cleanly

176

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
177

Mail Queues: Queues


Several queues in Exchange Server 2007
Submission Remote Delivery Mailbox Delivery Poison Message Unreachable

178

Mail Queues: Queues


Submission Queue
Used by the categorizer to gather all messages that have to be resolved, routed, and processed by Transport agents Only one Submission queue per transport server Messages remain here until categorization is complete Messages enter Submission Queue from various sources

179

Mail Queues: Queues


Messages may queue up in the Submission Queue due to
Issues with AD such as slow DCs, AD permissions, unable to communicate with a DC Problems with Transport Routing Agents Performance issues on the server itself such as a disk or processor bottleneck Large nested or query-based distribution groups

180

Mail Queues: Queues


Remote Delivery Queues
Handles the queuing of a message to a specific SMTP based target destination Remote SMTP Host Drop Directory Queue for every remote domain Created and deleted dynamically Each remote delivery queue contains messages being routed to the same delivery destination Edge Servers always places messages into Remote Delivery Queues unless they are poison or unreachable

181

Mail Queues: Queues


Messages may queue up in the Remote Delivery queue due to
Connectivity issues such as network links being down or ports being blocked DNS name resolution issues Mis-configurations of receive connectors such as incorrect Authentication settings or Remote IP Ranges Certificate issues Performance issues on either the sending or receiving server

182

Mail Queues: Queues


Local Delivery Queue
Holds messages being delivered to a Mailbox server in the local AD site using encrypted Exchange RPC Only available on Hub Servers Processed by Store Driver More than one mailbox delivery queue can exist

183

Mail Queues: Queues


Messages may queue up in the Local Delivery queue due to
Connectivity issues to the Mailbox server such as network links being down or ports being blocked Services or stores being offline on the Mailbox server DNS name resolution issues Performance issues on the local transport or receiving Mailbox server Unable to log onto mailbox due to incorrect permissions for the Mailbox server

184

Mail Queues: Queues


Poison Message Queue
Used to isolate messages that are detected to be potentially harmful Contains messages that caused the Transport Worker Process to crash Exchange 2003 messages could repeatedly crash SMTP Service Required manual extraction Exchange 2007 can detect if a message crashes Transport Message is removed from processing and placed in the Poison Message Queue Maintains Poison Message Count on each message If count exceeds threshold, moved to Poison Message Queue Threshold controlled by PoisonThreshold value on TransportServer settings Default 2 Messages can be deleted by admin, exported for debug, resubmitted or expire

185

Mail Queues: Queues


Unreachable Queue
Contains messages that cannot be routed to their destinations Messages that have no route due to configuration errors Problems with connectors Missing attributes on mail enabled objects (HomeMDB) Messages without routes dont NDR immediately they are placed in the Unreachable queue Each transport server can have only one Unreachable queue Messages are automatically resubmitted for categorization if routing topology changes are detected

186

Mail Queues: Queues


Scenarios where messages are placed into the Unreachable queue
The recipient is a valid Active Directory recipient object. However a routing path cannot be calculated for that recipient The recipient is an external SMTP address and a matching connector cannot be found for the address space The recipient is a distribution group and the expansion server for the distribution group is invalid or does not have the Hub Transport server role installed The recipient is an SMTP address recipient of a message that was received on a Receive connector that is linked to a Send connector that is ignored by the routing component of the categorizer because it is disabled or misconfigured in some way

187

Mail Queues: Queues


Scenarios where messages are NOT placed into the Unreachable queue and an NDR is generated instead
The routing path cannot be calculated for a recipient because constraints, such as message size restrictions, prevent delivery of the message using the single, deterministic route calculated by the categorizer The recipient is a non-SMTP address and a matching connector cannot be found. Or the matching connector is disabled or misconfigured The recipient is a non-SMTP address recipient that was received on a Receive connector that is linked to a Send connector that is ignored by the routing component of the categorizer because the Send connector is disabled or misconfigured

188

Mail Queues: Queues


Queue message processing in RTM
All queues except for Submission queue process messages using First In/First Out (FIFO) Submission Queue uses Round Robin For every X number of higher priority messages processed process a lower priority message Keeps higher priority messages from stopping delivery of low priority Ratio cannot be modified

189

Mail Queues: Queues


Priority Queuing in Exchange 2007 SP1
Priority queuing affects the transmission of messages from a delivery queue to the destination messaging server When enabled higher priority messages are transmitted before lower priority messages to their destinations Helps admins define specific SLA requirements for message delivery times Enabled or disabled in EdgeTransport.exe.config PriorityQueuingEnable parameter When enabled the priority message queue limits (such as expiration of messages) in EdgeTransport.exe.config override the message queue limits set by Set-TransportServer

190

Mail Queues: Queues


INBOUND SMTPIn CATEGORIZER

PICKUP/REPLAY DIRECTORY

QUEUING

SUBMISSION

AGENTS

RESUBMIT

STOREDRIVER

UNREACHABLE

MAPI

SMTPOut REMOTE POISON

OUTBOUND

191

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
192

Mail Queues: Troubleshooting


Troubleshooting mail queues involves looking at queue and message properties to help isolate the issue, specifically
Status Either Active, Ready, Retry or Suspended Next Hop Domain Available for queues only Shows the destination of the queue such as a Smart Host or another SMTP or mailbox server Last Error Lists the reason for message delivery failure

If a queue is or messages are in a retry state look at the Last Error value to help isolate the reason for failure Use the Next Hop Domain value to help determine where the message is being delivered
193

LAB 6 Disaster Recovery


194

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
195

Certificate and Transport: Overview


SMTP STARTTLS
Mutual TLS Opportunistic TLS Force Required TLS
By Connector Domain Security

POP IMAP Edge Synchronization

Certificate and Transport: Overview (cont.)


Direct Trust Certificate
Retrieved from AD Must be available also in local Certificate Store

SMTP X-AnonymousTLS
Hub to Hub Hub to Edge/Vice Versa
Direct Trust Authentication

198

Microsoft Confidential

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
199

Certificate Selection - STARTTLS


Process that components go through to determine which certificate should be used for an incoming connection FQDN
Always based upon the connector FQDN Get-<SEND | RECEIVE>Connector | FL FQDN

Certificate Selection - STARTTLS (cont.)

List of valid Certificates are found based on the connector FQDN


Has either no extended key usage extension or has the extended key usage extension containing the Server Authentication Object Identifier Has either no key usage extension or it has the key usage extension with digital signature bit asserted It has a RSA public key > 1024 bits in size Has a valid certificate chain up to a trusted root (or self signed) Revocation checking passes on the certificate chain The private key is present and accessible (Network Service) The private key is not stored in a removable device The private key is not UI protected

Certificate Selection - STARTTLS

From the remaining list, pick the best certificate (in order of preference)
Trusted CA Issued Certificate preferred over Self-Signed Newest installed certificate over oldest
An older 3rd Party CA issued certificate would be used over a newer self-signed certificate

Certificate Selection Process Inbound STARTTLS

203

Microsoft Confidential

POP3 and IMAP4

Selection process is similar to SMTP STARTTLS Three Exceptions:


Instead of FQDN they use X509CertificateName
Get-POPSettings Get-IMAPSettings

Newest valid certificate wins changed in SP1 Does not support Wildcards such as *.fourthcoffee.com changed in SP1

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
205

Troubleshooting - STARTTLS

Tools Used
Exchange Management Shell TELNET.EXE (is STARTTLS advertised?) TCP.EXE (What certificate is being served?) CertUtil to verify certificates Application Logs

Troubleshooting - STARTTLS

What is the FQDN of the Receive Connector Get-ExchangeCertificate DomainName <FQDN> Get-ExchangeCertificate DomainName <FQDN> | FL * Is there more than one certificate?
Are any certificates issued from a trusted CA? Are the certificates valid? What is the newest certificate? Can we access the CRL Distribution Point? Proxy?

Troubleshooting
[PS] C:\>Get-ExchangeCertificate 91BFBDD870D8928018E22B736922411645218B85 | fl *

CertificateDomains
CertificateRequest IisServices

: {clt-e2k7.fourthcoffee.com, clt-e2k7}
: : {IIS://clt-e2k7/W3SVC/1}

IsSelfSigned RootCAType Services Status


PrivateKeyExportable Archived FriendlyName IssuerName

: : : :

False Enterprise IIS, SMTP Valid

: True : False : Microsoft Exchange : System.Security.Cryptography.X509Certificates.X500Distin guishedName

NotAfter NotBefore HasPrivateKey


SerialNumber SubjectName Thumbprint Version Handle Issuer Subject

: 9/17/2009 10:11:42 AM : 9/18/2007 10:11:42 AM : True


: 610FCD9F000000000011 : System.Security.Cryptography.X509Certificates.X500Distin guishedName : 91BFBDD870D8928018E22B736922411645218B85 : 3 : 133224224 : CN=LON-E2K7, DC=fourthcoffee, DC=com : CN=clt-e2k7.fourthcoffee.com

Troubleshooting
[PS] C:\Get-ExchangeCertificate 6CC3257C2236DFC88BA40CD9A374C9E53CC18E2B | fl *

CertificateDomains
CertificateRequest IisServices

: {clt-e2k7, clt-e2k7.fourthcoffee.com}
: : {}

IsSelfSigned RootCAType Services Status


PrivateKeyExportable Archived FriendlyName IssuerName

: : : :

True Registry SMTP Valid

: False : False : Microsoft Exchange : System.Security.Cryptography.X509Certificates.X500Distin guishedName

NotAfter NotBefore HasPrivateKey


SerialNumber SubjectName
Thumbprint Version Handle Issuer Subject

: 9/19/2008 9:44:29 AM : 9/19/2007 9:44:29 AM : True


: 9816558F2C99EF924E7A5AA1730498DA : System.Security.Cryptography.X509Certificates.X500Distin guishedName : 6CC3257C2236DFC88BA40CD9A374C9E53CC18E2B : 3 : 68983992 : CN=clt-e2k7 : CN=clt-e2k7

Troubleshooting
TCP.EXE

Troubleshooting
Certutil Sample
[PS] C:\>certutil -verify certnew.cer Issuer: CN=LON-E2K7 DC=fourthcoffee DC=com Subject: CN=clt-e2k7 Cert Serial Number: 610cbc3c000000000010
dwFlags = CA_VERIFY_FLAGS_DUMP_CHAIN (0x40000000) ChainFlags = CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT (0x40000000) HCCE_LOCAL_MACHINE CERT_CHAIN_POLICY_BASE -------- CERT_CHAIN_CONTEXT -------ChainContext.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)

ChainContext.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40) ChainContext.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)


ChainContext.dwRevocationFreshnessTime: 4 Days, 22 Hours, 34 Minutes, 49 Seconds

SimpleChain.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)

SimpleChain.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40) SimpleChain.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)


SimpleChain.dwRevocationFreshnessTime: 4 Days, 22 Hours, 34 Minutes, 49 Seconds

CertContext[0][0]: dwInfoStatus=102 dwErrorStatus=1000040


Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com Subject: CN=clt-e2k7 Serial: 610cbc3c000000000010 Template: WebServer 06 99 41 18 54 db 2d 8b 2c ae 0a 5d d7 b5 27 54 42 d8 20 0b Element.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2) Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)

Element.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40)

Troubleshooting
Certutil Sample
CRL 15: Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com 93 a6 c0 1b ad cf 8f 9a 91 3b 6e b5 7e bc 93 ed 53 89 89 5c Delta CRL 16: Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com 35 9e 39 96 9d e8 08 ce 3c 16 a5 99 d5 aa 28 89 d1 54 db 3e Application[0] = 1.3.6.1.5.5.7.3.1 Server Authentication

cont

::

Element.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)

CertContext[0][1]: dwInfoStatus=10c dwErrorStatus=0 Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com Subject: CN=LON-E2K7, DC=fourthcoffee, DC=com Serial: 115643e01d0eab874e228cc4545d7e6c d6 80 20 2f 11 ad f2 39 53 b5 92 df c1 5a 26 28 c4 5c e5 90 Element.dwInfoStatus = CERT_TRUST_HAS_NAME_MATCH_ISSUER (0x4) Element.dwInfoStatus = CERT_TRUST_IS_SELF_SIGNED (0x8) Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100) Exclude leaf cert: ef af 86 b5 a7 9e 25 51 44 18 98 b6 69 f9 df 62 c5 65 31 66 Full chain: a8 51 bc 17 d8 b4 94 5a 6a 3d b9 01 89 bc c6 63 37 8e 0b ea Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com Subject: CN=clt-e2k7 Serial: 610cbc3c000000000010 Template: WebServer 06 99 41 18 54 db 2d 8b 2c ae 0a 5d d7 b5 27 54 42 d8 20 0b

The revocation function was unable to check revocation because the revocation se rver was offline. 0x80092013 (-2146885613) Revocation check skipped -- server offline ERROR: Verifying leaf certificate revocation status returned The revocation func tion was unable to check revocation because the revocation server was offline. 0 x80092013 (-2146885613) CertUtil: The revocation function was unable to check revocation because the rev ocation server was offline.
CertUtil: -verify command completed successfully.

212

Microsoft Confidential

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
213

Internal Transport - Certificate

The Direct Trust Certificate: Whenever X-AnonymousTLS is used


Hub to Hub Hub to Edge Edge to Hub

Used to establish Direct Trust authentication after XAnonymousTLS negotiation between a Hub and Edge server. Used to establish secure LDAP connections from Hub for Edge Synchronization Used to encrypt and decrypt EdgeSynchronization credentials which are stored in the directory. Direct Trust Certificates = Default Certificate

Internal Transport - Certificate selection


The Internal Transport Certificate selection process simply loads from msExchServerInternalTLSCert property on the Exchange Server object Must also be found in the local Computer Store Expired Direct Trust Certificates do not affect mail flow
Warning:
Event Type: Warning Event Source: MSExchangeTransport Event Category: TransportService Event ID: 12017 Description: A direct trust certificate will expire soon. Thumbprint: 2135A85FC400DF078D56A7A1EBB1E4330DD68596, hours remaining: 720 Event Type: Warning Event Source: MSExchangeTransport Event Category: TransportService Event ID: 12015 Description: A direct trust certificate expired. Thumbprint: 2135A85FC400DF078D56A7A1EBB1E4330DD68596

Error:

Internal Transport - Determining Certificate

RTM use Certlib.ps1

SP1 use Get-TransportServer

Internal Transport
Error when Certificate Store cannot be found
Occurs when the Direct Trust certificate has been forcibly removed from the system (e.g MMC)

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
218

Troubleshooting Direct Trust

Tools Used:
Exchange Management Shell TELNET Protocol Logs CertLib.ps1

1st Determine this is a direct trust issue

Troubleshooting Direct Trust - Direct Trust Issue?


[PS] C:\>Get-Queue 5 | fl

Identity DeliveryType NextHopDomain NextHopConnector Status MessageCount LastError

: : : : : : :

Either there are LastRetryTime NextRetryTime IsValid ObjectState : : : :

clt-e2k7\5 SmtpRelayWithinAdSiteToEdge edgesync - northamerica to internet 4039e148-2af7-4889-9de8-6cf6f1b2716e Retry 1 451 4.4.0 Primary target IP address responded with: 454 4.7.0 Temporary authentication failure. Attempted failover to alternate host, but that did not succeed. no alternate hosts, or delivery failed to all alternate hosts . 9/19/2007 8:45:33 AM 9/19/2007 8:50:33 AM True Unchanged

Troubleshooting Direct Trust


Confirm X-AnonymousTLS

Is ExchangeServer authentication enabled on the receive connector? Make sure both Hub and Edge show X-ANONYMOUSTLS available

Troubleshooting Direct Trust - Protocol Logging


2007-09-19T12:27:38.994Z,0,,10.0.200.200:25,*,,attempting to connect 2007-09-19T12:27:39.004Z,1,+,, ... 2007-09-19T12:27:39.014Z,17,>,X-ANONYMOUSTLS, 2007-09-19T12:27:39.024Z,18,<,220 2.0.0 SMTP server ready, 2007-09-19T12:27:39.024Z,19,*,,Sending certificate 2007-09-19T12:27:39.024Z,20,*,CN=clt-e2k7.fourthcoffee.com,Certificate subject 2007-09-19T12:27:39.024Z,21,*,"CN=LON-E2K7, DC=fourthcoffee, DC=com",Certificate issuer name 2007-09-19T12:27:39.024Z,22,*,610FCD9F000000000011,Certificate serial number 2007-09-19T12:27:39.024Z,23,*,91BFBDD870D8928018E22B736922411645218B85,Certificate thumbprint 2007-09-19T12:27:39.024Z,24,*,clt-e2k7.fourthcoffee.com;clt-e2k7,Certificate alternate names 2007-09-19T12:27:51.292Z,25,*,,Received certificate 2007-09-19T12:27:51.292Z,26,*,3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D,Certificate thumbprint 2007-09-19T12:27:51.292Z,27,*,SMTPSendEXCH50 SendRoutingHeaders SendForestHeaders SendOrganizationHeaders,Set Session Permissions 2007-09-19T12:27:51.292Z,28,*,,DirectTrust certificate 2007-09-19T12:27:51.292Z,29,*,CN=E2K7-EDGE1,Certificate subject 2007-09-19T12:27:51.292Z,30,*,CN=E2K7-EDGE1,Certificate issuer name 2007-09-19T12:27:51.292Z,31,*,126DA18D9F7FE69B48044E9D4985B894,Certificate serial number 2007-09-19T12:27:51.292Z,32,*,3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D,Certificate thumbprint 2007-09-19T12:27:51.292Z,33,*,E2K7-EDGE1;E2K7-EDGE1.fourthcoffee.com,Certificate alternate names 2007-09-19T12:27:51.292Z,34,>,EHLO clt-e2k7.fourthcoffee.com, 2007-09-19T12:27:51.302Z,35,<,454 4.7.0 Temporary authentication failure, 2007-09-19T12:27:51.302Z,36,>,QUIT, 2007-09-19T12:27:51.302Z,37,-,,Remote

Troubleshooting Direct Trust


Confirm Certificate Thumbprints:
[PS] C:\>gettlscertfromad clt-e2k7 Running on an Edge Server - pulling cert details from Adam System.DirectoryServices.DirectoryEntry Running on an Edge Server - pulling cert details from Adam (&(objectclass=msExchExchangeServer)cn=clt-e2k7) Getting Prop

Thumbprint Subject ---------------715F6840E3B4CC241CFDC9D912A5E8491B2BDE74 CN=clt-e2k7

[PS] C:\>gettlscertfromad e2k7-edge1 Running on an Edge Server - pulling cert details from Adam System.DirectoryServices.DirectoryEntry Running on an Edge Server - pulling cert details from Adam (&(objectclass=msExchExchangeServer)cn=e2k7-edge1) Getting Prop Thumbprint Subject ---------------3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D CN=E2K7-EDGE1

Agenda
Troubleshooting Mail flow
Local Delivery & Mail Submission: Architecture Local Delivery & Mail Submission: Troubleshooting Remote Delivery: Architecture Troubleshooting Mail Flow

Troubleshooting Mail Queues


Mail Queues: Architecture Mail Queues: Queues Mail Queues: Troubleshooting

Troubleshooting Transport Certificates


Certificates and Transport: Overview Certificate Selection Troubleshooting STARTTLS Internal Transport Troubleshooting Direct Trust SP1 Changes
224

SP1 Changes
Certificate Logging Get-ExchangeCertificate Domain FOO.COM Opportunistic TLS fallback Direct Trust Certificate
Changing Terminology -Internal Transport Certificate Updated anytime SMTP provided as service Warnings upon update Dont have to re-subscribe w/update on Hub No more 1037/2019 events Attempts to fallback to alternate certificates if cant be loaded

POP/IMAP
Selection now prefers PKI Supports Wildcards

SP1 Changes
Logging Sample
EBDDE4C98F71840199B4256B9368F265A92DDB0C: Rejected. Unable to access the associated private key for the certificate.
Searching for a certificate that has one of the following FQDNs :

mail.fourthcoffee.com 60CF94F4522A25DFEBB734F4636FB1D1F77D819A: Is not valid for signing, dropping from consideration.


Considering certificate A7C7D7B88B767BAA6003A1F0DBE0DF0937210C68

A7C7D7B88B767BAA6003A1F0DBE0DF0937210C68: Rejected. Has a key size less than 1024 bits, dropping from consideration.
Considering certificate CA979162AC2854BBE389153D965358379F8CD43E CA979162AC2854BBE389153D965358379F8CD43E: Selected. PKI issued certificate.

Troubleshooting Mailbox Server (MBX)

Agenda

Databases
Public Folders Recipient Management Distribution Lists & Address Lists Offline Address Book Exchange Search ExMON

ESE System
Transaction Log File
7 15 5 25 4 15 8 4 1 10

Database Storage Files


.EDB Storage File 8 KB

Memory
4 8 12 7 17 10 15 8 25 3 4 1

1 5 9

2 6 10

3 7 11

Making Changes to the Database

Transaction Database Storage Files Log File


.EDB Storage File 1001

Memory
1001 980

7 15 5 25 4 15 8 4 1 10

1 5 9

2 6 10

3
1001

4 8 12

7 11

Important JET Database Files


Three important file extensions to remember:
EDB (Exchange database files) LOG (Exchange Transaction logs) CHK (Exchange transaction checkpoint files)

Checkpoint File

Transaction Log Entries Written to the Database

Exx.chk

Transaction Log Entries Not Yet Written to the Database

Exxnnnnn.log

Storage Groups
Set of all databases that share common log files Separate instance of Jet Up to 50 Storage groups per Server (Enterprise) 5 Databases per Storage Group but 50 maximum databases (Enterprise) Recommendation is to have one database for one storage group

Utilities in 2007
No change compared to 2003!
Eseutil command and options are the same but now can be also used on Hub and Edge transport databases Isinteg command and options are the same

Troubleshooting Exchange Databases


For database problems all the troubleshooting techniques used in 2003 are still valid

236

Microsoft Confidential

Continuous Replication flavors


LCR: Local Continuous Replication CCR: Cluster Continuous Replication SCR: Standby Continuous Replication

237

Microsoft Confidential

Troubleshooting Continuous Replication


Exchange Management Console
Configuration Management Monitoring

Exchange Management Shell


Get-StorageGroupCopyStatus Test-ReplicationHealth

Performance counters
Many counters available to monitor Continuous Replication

238

Microsoft Confidential

Exchange 2007 SP1 Changes

Online Database Checksum (DBScan)


Addresses CCR scenario where backup occurs off passive (Active will never be completely scanned) Opt in feature via Registry key
Set on a per server basis Throttle parameter (unlimited by default)

Uses half of the Online Defrag Maintenance Window time Will notify corruption via eventlog (-1018, -1022 etc)

Exchange 2007 SP1 Changes


Online Defrag Monitoring:
New Perfmon Counter:
MSExchange Database->Online Defrag Pages Freed/sec

Extended Event information:


Event Type: Information Event Source: ESE Event Category: Online Defragmentation Event ID: 703 Date: 6/20/2007 Time: 6:34:26 AM User: N/A Computer: DF-MBX-30 Description: MSExchangeIS (19052) SG06: Online defragmentation has completed the resumed pass on database 'e:\MDB06\priv06.edb', freeing 42794 pages. This pass started on 6/16/2007 and ran for a total of 124919 seconds, requiring 7 invocations over 4 days. Since the database was created it has been fully defragmented 14 times over 73 days.

Exchange 2007 SP1 Changes


How to determine if OLD is running often enough?
2 week rule of thumb or analyze server Analyze Perfmon log taken during OLD window

121:1

If Read:Freed ratio is greater than 100:1 then the OLD window can be reduced If Read:Freed ratio is less than 50:1 then the OLD window should be increased

Why reduce?
Increase backup window Reduce snapshot/block level differential sizes (DPM v2) Validate that Online Checksum/Page Zeroing can be introduced with current OLM window

Agenda
Databases

Public Folders
Recipient Management Distribution Lists & Address Lists Offline Address Book Exchange Search ExMON

Public Folder Replication


Mail based as in previous versions of Exchange
Ensure the public store has an email address Check message tracking Enable diagnostic logging

CCR cannot have a public store if there is more than one PF store

Force Public Folder replication


In the management console
Update Hierarchy Update content

Using the management shell


Update-PublicFolderHierarchy Update-PublicFolder

Public Folder Replication Storm


Public folder replication storm can still occur in Exchange 2007 Suspend-PublicFolderReplication Stops all replication except hierarchy Resume-PublicFolderReplication

Public Folder Referrals


Defines Client Public folder access Routing Group based in Exchange 2000/2003 Active Directory Site based in Exchange 2007

Troubleshooting Referrals
Connection status in Outlook Get-PublicFolderDatabase UseCustomReferralServerList CustomReferralServerList Link cost (AD and RGC)

Agenda
Databases Public Folders

Recipient Management
Distribution Lists & Address Lists Offline Address Book Exchange Search ExMON

Recipient Management
Simplified Recipient Provisioning for the Exchange Administrator Support for Split Permissions within a single forest Ability to delegate Recipient management to a lower level administrator Ability to create Active Directory object and mail- or mailbox-enable it Instant-on recipients no need to wait or kick the RUS to stamp objects Rich filtering support includes domain- and forest-wide scoping Allows administrators to see only the objects relevant to them New recipient types plus clear distinction of all recipient types Conference Room and Equipment Mailbox (Resource Mailbox) Policy support for select mailbox settings Ability to apply the same settings to all recipients associated with a policy Unified Messaging, Messaging Records Management, and ActiveSync Recipient Policies still exist but are now called E-Mail Address Policies

Page 253

Working with Recipients


Recipients are primarily mailbox-enabled Active Directory users Recipients are managed through the Exchange Management Console or the Exchange Management Shell Active Directory Users and Computers (ADUC) is no longer extended for management of Exchange recipients User AD properties relevant for the Global Address List can be managed through the Recipient Configuration container in Exchange Management Console Active Directory User accounts can be created from within the Exchange Management Console when they are mailbox-enabled

Page 255

Working with Recipients and ADUC Active Directory Users and Computers (ADUC) is no longer extended to manage Exchange recipients

It is not supported to mailbox-enable user accounts using ADUC when the mailboxes will be housed on Exchange 2007 servers. If there is an Exchange Server 2003 RUS server operational, the ADUC mailbox operation will succeed, so the mailbox will be able to send and receive messages Mailbox is considered legacy and certain features or actions, or properties will be blocked Set-Mailbox -ApplyMandatoryProperties
Page 256

Scoping
Recipient Configuration Center supports domain- and forest-wide scoping Ability to specify which DC Console should connect to Scope is configurable, even down to OU $AdminSessionADSettings session variable (in shell) Domain Scope is default behavior Determined by domain of which the Server is a member: Only recipients (e.g., redmond\evand) in selected domain can be found Referenced recipients (e.g., Membership, Delegate, Owner, etc.) are exempt Reduces issues related to replication Forest Scope can display and find all recipients within the forest Provides a complete view of the GAL
Page 259

Enable/Disable vs. New/Remove

Enable/Disable Adds or removes Exchange attributes from existing Active Directory objects Enable adds attributes to an existing Active Directory object mail-enabled or mailbox-enabled Disable removes attributes returning Active Directory object to non-Exchange state StoreMailbox in MDB will fall under mailbox retention and will eventually be purged

New/Remove* Creates or deletes Active Directory objects plus adds and removes Exchange attributes in one step New creates Active Directory object and mail-enables or mailbox-enables the object Default Remove removes Active Directory object. StoreMailbox in MDB will fall under mailbox retention and will eventually be purged -Permanent: removes Active Directory object and StoreMailbox in MDB will be purged immediately (shell only) * Must have Account Operator privileges

Page 262

Email Address Policies


Created pre-canned filters to simplify definition and usage for common cases All Recipient Types, Users with Mailboxes, Resource Mailboxes, Mail-Enabled Contacts, and Mail-Enabled Groups Conditions Supported: State or Province, Department, Company, and Custom Attributes Ability to schedule the creation and application of Email Address Policies for off-hour execution when using EMC RUS as a service no longer needed, resulting in reduced system processing demand Mailbox Manager functionality separated from EAPs Replaced by Messaging Records Management functionality Advanced or Non-Mainline (Shell Only) Custom Filters - will be visible, but not editable, in GUI
Page 263

Managing Mailboxes

Well-known functionality are still there New mailbox Move mailbox Delete mailbox Change Mailbox properties

Page 265

New Mailbox Management tasks

Statistics Get-LogonStatistics Get-MailboxStatistics Get-MailboxFolderStatistics

Troubleshooting Mailbox access


Test-MapiConnectivity Outlook logging 831053 How to turn on the Enable Mail Logging option for troubleshooting in Outlook 2003 and Outlook 2007 Network Take network trace Reproduce the problem locally

Moving Mailboxes
You can use the Exchange Management Console or the Exchange Management Shell to move mailboxes You can move mailboxes across mailbox databases, across servers, across domains, across Active Directory sites and across forests You can also move mailboxes among different versions of Microsoft Exchange Server (2000/2003/2007 only) Move mailbox is more resilient (Pre-Validation) Exchange Management Shell Command: move-mailbox More options available Note: You cannot use the Exchange Management Console to move mailboxes across forests. You must use the Exchange Management Shell instead.

Page 269

Troubleshooting Mailbox Move

Email Address Enforcement IgnoreRuleLimitErrors cmdlet option Damaged or corrupted messages BadItemLimit cmdlet option Skip errors validation from EMC Move-Mailbox Wizard MfcMapi Isinteg

Exmerge Replacement Need


Exmerge is not shipped with Exchange Server 2007 The Move-Mailbox, Export-Mailbox, and Restore-Mailbox tasks are implemented to cover many of the scenarios where ExMerge is used with Exchange Server 2003 Export to PST is possible in SP1

Page 274

ExMerge Replacement cmdlets


Export-Mailbox
Export mailbox content to another mailbox or PST Must run 32-bits if exporting to PST Must have Outlook installed Can filter content Can delete source message Will export dumpster

Import-Mailbox
Imports from PST Must run 32-bits console

Restore-Mailbox

LAB 7 Troubleshooting MAPI access


LAB 8 Using MFCMAPI

278

Microsoft Confidential

Agenda
Databases Public Folders Recipient Management

Distribution Lists & Address Lists


Offline Address Book Exchange Search ExMON

Distribution List Types


Mail-enabled Universal Distribution Group Mail-enabled Universal Security Group Mail-enabled Non-Universal Group Dynamic Distribution Group

Automatic Group conversion


Users can select Universal Distribution Group to set permissions on folders Exchange will automatically convert the group to security Can potentially growth user security token Can be disabled
Set msExchDisableUDGConversion through ADSIEdit

Interaction with Exchange 2003


Dynamic Distribution groups created in Exchange 2003 must be upgraded to be modified in Exchange 2007
Get-DynamicDistributionGroup | Format-List Name,*RecipientFilter*,ExchangeVersion If RecipientFilterType is "Legacy and ExchangeVersion is "0.0 (6.5.6500.0) Set-DynamicDistributionGroup recipientfilter {... } forceupgrade $true

Exchange 2007 Distribution List can only use Universal group scope

Common Issues
Unable to send to the Distribution Group from users external to the Organization
When Require that all senders are auhenticated flag is set on DL Properties To solve the issue run: Set-DistributionList RequireSenderAuthenticationEnabled $true

Unable to view the Distribution Group in EMC


When the group scope is Global or Domain Local To solve this issue change the group scope to Universal scope

Default Address Lists


Default Global Address List All Contacts All Groups All Rooms Public Folders All Users

Populating the Address Lists


In Exchange 2007 there is no Recipient Update Service (RUS) PreCanned Filters In order to update an Address List you have to run the cmdlet Update-AddressList Update-AddressList can be scheduled to run using Exchange Management Shell

Issues
Unable to Edit the Address List Properties (Address List must be upgraded)
If ALs created by using Exchange 2003 Upgrade them to Exchange 2007 to use OPATH filters
Set-GlobalAddressList "Default Global Address List" -RecipientFilter {(Alias ne $null -and (ObjectClass -eq 'user' -or ObjectClass -eq 'contact' -or ObjectClass -eq 'msExchSystemMailbox' -or ObjectClass -eq 'msExchDynamicDistributionList' -or ObjectClass -eq 'group' -or ObjectClass -eq 'publicFolder'))}

Address List are not updated after modifying them


Exchange 2007 has no RUS to update them Address Lists must be updated using EMC or EMS (Update-AddressList) RUS API can be troubleshooted via : Get-EventLogLevel MSExchangeAL | Set-EventLogLevel -Expert

Agenda
Databases Public Folders Recipient Management Distribution Lists & Address Lists

Offline Address Book


Exchange Search ExMON

What is the Offline Address Book?

An Offline copy of the Global Address List Used by Outlook clients in Offline mode or Cache mode Several versions appeared over time Version 2 appeared in Exchange 5.5
For clients Outlook 98 and later

Version 3 Appeared in Exchange 2003


For clients Outlook 2003 and later

Version 4 Appeared in Exchange 2003 SP2


For clients Outlook 2003 SP2 and later

292

Microsoft Confidential

Offline Address Book Overview


Exchange 2007 introduces a new mechanism for distributing the OAB that does not require Public Folders HTTP(S) and Background Intelligent Transfer Service (BITS) can be used by Outlook 2007 clients Advantages of this new method are: Support for more concurrent clients Reduction in bandwidth usage More control over the distribution points The web distribution is only available for OABv4 Legacy Clients will still need to utilize public folders for OAB obtainment Outlook 2007 clients can utilize the web distribution for obtaining the OAB
Page 293

Offline Address Book generation


The Offline Address Book (OAB) is generated as usual by the OABgen component on a Mailbox Server
Files are published to \\mbxserver\ExchangeOAB share and to the public store if available

On all Client Access servers, an OAB virtual directory is created to serve the OAB The Exchange File Distribution Service that runs on the CAS servers is responsible to getting the OAB content from the OABGen server The virtual directory points to the directory %programfiles%\microsoft\exchange server\ClientAccess\OAB In that directory, the different OABs are stored per <guid>
The .lzx files contains the OAB data in V4 format The oab.xml contains metadata for Outlook 2007

Outlook 2007 is configured to retrieve the OAB via the OAB URL that is obtained through AutoDiscover. Otherwise it will download OAB from public folders like all other legacy clients
3/24/2009 | Page 294

Offline Address Book size and network bandwidth usage

Started to become an issue with Outlook cache mode deployments No limit for Public Folder connections OAB throttling to control network bandwidth usage Outlook Random Full OAB Request Timer
Key: HKCU\Software\Microsoft\Exchange\Exchange Provider DWORD: Max Full OAB Download Wait Value: Integer >=1

295

Microsoft Confidential

OAB V4 Improvements

OAB V4 is more compressed


Binpatch technology and LZX compression method used

Rebuild needs are reduced


Indexes generated by Outlook Client

Limited property sizes Web distribution optimizes network bandwidth usage

297

Microsoft Confidential

Offline Address Book Web Distribution Scenario


Scenario

User B

Remote Office (Sao Paulo)

Corporate Headquarters (London)

Legend
CAS Server

User A

Outlook Fast Connection

Mailbox Server Slow Link

The Internet

User C

Page 299

Web distribution self healing

OAB generated files are kept within the System Attendant mailbox Deleted files from the mailbox role OAB share will be copied back Deleted files from the CAS web virtual directory will be copied back from the Mailbox OAB share

302

Microsoft Confidential

Offline Address Book Public folder distribution

Outlook clients will connect through RPC to the public folder server holding a replica of the OAB To reduce bandwidth usage you should: Make sure to use OAB V4 Replicate the OAB on a public folder in every Active Directory site holding a Mailbox role Or create an OAB per site and assign the mailbox stores to the local OAB Dont forget OAB Threshold registry setting

303

Microsoft Confidential

OAB Version used


Outlook 2003 SP2 and Outlook 2007 can use V4 Will failback on previous version if not available Ensure that Version4 of the OAB is enabled Get-OfflineAddressBook | fl Name,Server,Versions Set-OfflineAddressBook Versions Version4 If the profile is ANSI, OAB V2 will be used Mainly for profiles linked to mailboxes moved from Exchange 5.5 Deploy GPO to force Profile conversion Set registry key to force OAB V4 use
Key: HKCU \Software\Microsoft\Exchange\Exchange Provider DWORD: OAB v4 Only Value: 1
305 Microsoft Confidential

Troubleshooting OAB using diagnostic logging

Set Diagnostic level Set-EventLogLevel -Identity MSExchangeSA\OAL Generator -Level Expert Read event logs Using the event viewer Using Powershell Get-EventLog Application | Where {$_.Category -eq "OAL Generator"}

309

Microsoft Confidential

Offline Address Book Integrity Checker (OabInteg)

Tool to simulate
Client connection to download OAB files from public folder store
Does not yet test web distribution (should be available soon)

OAB generation process

Downloadable from Internet


You can download OABInteg from here: http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleG uid=A2338E73-F521-4071-9B1D-AAF49C346ACD

If run from the server install CDO 1.2.1 to test MAPI access. Downloadable from:
http://www.microsoft.com/downloads/details.aspx?FamilyID=e17e7f31079a-43a9-bff2-0a110307611e&DisplayLang=en

312

Microsoft Confidential

OAB Generation Errors Exchange server configured to generate the OAB By default it is first Exchange Server in org which is Exchange 2003 in mixed modes In mixed mode:
Move OAB from Exchange 2003 server to Exchange 2007 server Local replicas of OAB on Exchange 2007 server should be successfully replicated All mailbox stores on Exchange 2007 server under Client Settings tab should have Default Offline Address Book associated

314

Microsoft Confidential

OAB Generation on CCR Clusters

CCR cluster Only one node is generating OAB When the node becomes passive OAB is not updated Logs error event 9395 How to Fix:
HKLM\System\CurrentControlSet\Services\MSExchangeSA\Parameters\Se rver-Name\EnableOabGenOnThisNode ="ThisNodeName"

316

Microsoft Confidential

OAB Download troubleshooting


Outlook send receive errors are saved in the sync issues folder Use Err.exe to interpret the error Enable Outlook logging 831053 How to turn on the Enable Mail Logging option for troubleshooting in Outlook 2003 and Outlook 2007 http://support.microsoft.com/default.aspx?scid=kb;ENUS;831053 Check which OAB to download Configured either on the mailbox store or on the recipient itself Check Application log Take a network trace

318

Microsoft Confidential

OAB Download from public folder troubleshooting


Has the OAB been generated on public folder?
Is the OAB public folder enabled? Is the version we want to download available? Verify Application log on the OAB generator server Look in the OAB public folder Use OABInteg

Is the OAB public folder reachable by Outlook?


Verify where are the replicas of the OAB we should download Make sure the public store holding the nearest replica is reachable Check public folder replication or referrals Look in the Outlook connection status if Outlook did effectively connect to the server holding the replica Throttling might be preventing download

320

Microsoft Confidential

OAB Download from Web distribution troubleshooting


Make sure OAB is configured to generate V4
Only V4 are web distribution enabled

Make sure the Outlook profile is Unicode


Ansi profile wouldnt be able to download V4

Check if OAB has been generated on the Mailbox server


Web distribution wont be generated if system attendant cant logon to its mailbox

Check if it has been replicated to the OAB virtual directories


Exchange file distribution service will copy it from the mailbox server to the CAS

Verify Outlook auto configuration (See CAS module)

321

Microsoft Confidential

Troubleshooting out of date OAB


Verify by connecting Online that inconsistency is only on OAB Entries not stamped by Recipient Update API
Update-Recipient

Force an update and check event log


Update-OfflineAddressBook

Out of date only for old Outlook clients


Those clients will use old OAB version. Might be an issue when Exchange 2007 is introduced in an existing organization

Active Directory replication


The Global Catalog used to generate the OAB may have directory replication issues. Use tools like replmon or repamin to verify

324

Microsoft Confidential

LAB 9 Troubleshooting OAB

327

Microsoft Confidential

Agenda
Databases Public Folders Recipient Management Distribution Lists & Address Lists Offline Address Book

Exchange Search
ExMON

Exchange Search

Understanding Exchange Search Difference between Exchange Search and store search Unexpected results scenarios Troubleshooting Exchange Search

Understanding Exchange Search

Microsoft Exchange Server 2007 Search is a feature that allows you to quickly search text in messages through the use of pre-built indexes Indexes occupy approximately 5 percent of the total mailbox database size Kept separately in same location as database files

Performance Enhancements
Used by OWA and Outlook online mode Outlook cached mode uses new client-side search Windows Desktop Search Instant Search goes through attachments in Outlook Can be extended to use any filter in Windows

Page 331

Performance Improvements
Outlook in online mode Exchange Server 2007 Search Indexer and advanced find in Outlook 2007 Faster indexing than Exchange Server 2003 and Exchange Server 2000 New messages indexed in under a minute Small storage tax (~5%) for indexes Indexes/searches message bodies and attachments Uses any filter installed in Windows Can install new filters later Outlook in Cached Exchange Mode On Windows XP, Outlook uses Windows Desktop Search On Windows Vista, Outlook uses Vistas built-in search engine

Page 332

Understanding Exchange Search

Exchange Search Service


Update

Mailbox

Index

Notificatio n MS Information Store Service

Difference between Exchange Search and store search


Exchange Search Faster Based on words Search attachments* Uses index to search Not case sensitive Doesnt support MAPI Exchange Store Search Slower Based on bytes stream Cannot Search attachments Uses serial scans Case Sensitive Support MAPI restrictions

restrictions
* Attachments types that are supported by the installed filters

Unexpected results scenarios


Documents that are encrypted with the Digital Rights Management feature will not be indexed.
For attachments that do not have associated filters, the attachment will not be indexed, but the e-mail message will be indexed. Advanced search grammar (for instance, typing "From:xyz" in the basic search bar searches the from: property for the string "xyz) is supported only when Instant Search is enabled. Instant Search requires that Windows Desktop Search 3.0 is installed.

Troubleshooting Exchange Search


Is the MSExchangeSearch service started?

Step 1

Is the IndexEnabled parameter set to true Step 2 Get-MailboxDatabase |ft Name,IndexEnabled Has the Exchange database been crawled? Step 3 MSExchange Search Indices performance object=0 Run the Test-ExchangeSearch

Step 4

Check Event Viewer Step 5 Source: MSExchangeSearch Indexer restart the Microsoft Search

Step 6

Test-ExchangeSearch
The Test-ExchangeSearch cmdlet creates a message and attachment that only the Microsoft Exchange search can find. Unless a mailbox is specified in the Identity parameter, the message is stored in the System Attendant mailbox. The command waits for the message to be indexed and then searches for the content. The command reports success if the message content is found. The command reports failure if the content is not found after the interval set in the IndexingTimeout parameter has elapsed. To run the Test-ExchangeSearch cmdlet, the account you use must be delegated the following: 1. Exchange Recipient Administrator role -and2. Exchange Server Administrator role and local Administrators group for the target server

How to rebuild the search index

Programmatically: use the ResetSearchIndex.ps1 Manually stop the service and deleting the file GetDatabaseForSearchIndex.ps1 When the index directory files are provided, this script returns the associated mailbox database names. GetSearchIndexForDatabase.ps1 This script returns index directories for the specified mailbox database names.

LAB 10 Troubleshooting Search

339

Microsoft Confidential

Agenda
Databases Public Folders Recipient Management Distribution Lists & Address Lists Offline Address Book Exchange Search

ExMON

What is Exmon?

Originally developed by Microsoft to understand user load on servers Shows per user activity in details Allows to track down high users

Introducing ExMon
Administrators can view the following using ExMon: IP addresses used by clients Microsoft Office Outlook versions and mode (Cached Exchange Mode versus classic online) Outlook client-side monitoring data CPU usage Server-side processor latency Total latency (network and processing) Network bytes In Exchange 2007 works for all mailbox access

342

Microsoft Confidential

When to Use ExMon

Some Outlook users are complaining about latencies regarding mailbox access RPC Average Latency is high Want to know what Outlook versions are really in use Want to find high RPC activity users Are they in cache mode? Want to know who is working among connected users Determine usage pattern on healthy systems

343

Microsoft Confidential

Collecting ExMon Data


ExMon data can be collected in 3 ways Live from the ExMon User Interface Command Line Perfmon collection Command line or perfmon are recommended The 'Live' mode constantly rolls, does not save data Can use server CPU to process data Command Line and Perfmon accommodates large files and ability to script and control Larger files give much more insight including Better aggregate statistics Some data (like process name) is only traced on MAPI logon ExMon data should be collected to local disks only Consumes roughly 1MB/hour for every RPC Operation/sec

Viewing ExMon Data Must be viewed on same OS or higher as collected on Windows 2003 Server required to view data from Windows 2003 Server Large files can take a long time to open, use CPU Saving your work Command Line can save any 'By ' without displaying UI File->Save will save all 'By ' views in one .csv By Event (for a given user) can be saved only in UI The Save Icon on the toolbar instructs ExMon to save ETL files captured during Live Capture

Analyzing Exmon Data

Know your environment Establish baseline to compare Detect RPC Average Latency peak using the performance wizard

Main Exmon points

CPU Time Server Latency Client Latency Foreground Client Latency Network Bytes

Basic Principles
Focus on the most expensive Users or Operations (unless you are troubleshooting a particular user) Statistics are best for expensive operations or LOTS of inexpensive one Look for problem to repeat in ExMon, then tackle Longer captures are better than short ones Expect some expensive operations to happen Full sync of an OST Occasional searches and sorts Trick is to find the ones that happen frequently or really hurt When looking at an individual user Look for patterns of repetition Compare to 'normal behavior'

Terms of Use

2008 Microsoft Corporation. All rights reserved. Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. For more information see Microsoft Copyright Permissions at http://www.microsoft.com/permission Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. The Microsoft company name and Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. THIS DOCUMENT IS FOR INFORMATIONAL AND TRAINING PURPOSES ONLY AND IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT.

You might also like