Troubleshooting internal/external mail flow issues

The NSLookup tool is used to check the configuration of the mail exchange records.

Performing the NSLookup

• Go to Command Prompt
• Key in NSLOOKUP and Press Enter
• Enter server {required IP of DNS server}
• Press Enter
• Enter the following
  • Set q=MX
• Now if you enter the domain name required and press enter, its particular MX record will be shown. This ensures that MX record has been properly configured.

Example:

C:\> NSLOOKUP

Default Server: ns1.domain.com

Address: 10.0.0.1

> set q=mx

> domain.com

Server: ns1.domain.com

Address: 10.0.0.1

mailhost.domain.com MX preference = 0, mail exchanger = mailhost.domain.com

Blacklist check

If the MX Record is properly configured, we check for any blacklisting issues. Usually blacklisting is checked for when there are multiple domains from which mail is not incoming. To run a blacklist check, we need a black list tool for the outbound IP address and the domain name of the remote server, which can be obtained from http://www.mxtoolbox.com

Once we obtain the tool, we check for any blacklists marked against the different domains IP addresses which are having inbound mailflow issues. If there is a blacklisting, we need to contact the providers of Blacklist so that we can resolve the issue by removing the domain from the blacklist. This will obviously require some time as per the blacklist server. If the issue needs to be resolved urgently, we can always try to change the outbound IP address and take a look into the reason behind blacklisting.

Submit email using Telnet

Next test is to use Telnet to submit an email. For this, we first go for a network which is also an exchange/HUB server that can have proper inbound mail reception and use telnet to resubmit the email. By doing this, we can ensure that the port 25 traffic is properly received by the server. After this test is passed, we check for the MX record of port 25 to obtain its IP address.

Check the port 25 listening accessibilities and find out if exchange is listening on port 25 or is it some other hosts. If an exchange verb is obtained as the response to the EHLO command in case of the MX showing an exchange/HUB server.

If the host is not exchange, then the issue might be in the configurations of the smart host. If there are no issues in the configuration of the smart host, the SMTP logs may be checked for errors against submission of inbound mail to exchange.

EXRCA:

Next, we check the SMTP verbs are properly returned by the tool https://www.testexchangeconnectivity.com/

Receive Connector (E2k7 & E2k10)

Next, if the exchange server is 2007 or 2010, we check the receive connector. This is done if a particular error is shown by the sender’s server namely, “530 5.7.1 Client was not authenticated and generate NDR”. Under such circumstances, we do the following

• Check the properties of the receive connector.
• Check if NIC specified for the receive connector is correct.
• Check if the IP address specified in the receive connector is configured properly
• Check if anonymous permissions are available for the receive connector configurations. This should be made available.
• The receive connector should have SMTP protocol logging enabled to Verbose and check if the SMTP traffic is actually hitting the server.
• The receive protocol logs may be verified so that there are no errors in them.

Default SMTP Virtual Server Properties (E2k3):

If the exchange version is 2003, we follow the following steps to check the SMTP virtual server properties:

• Ensure that the IP address port binding is properly configure.
• Verify the Default Virtual server IP address.
• Check for the port configuration from General-> Advanced. The port should be configured as 25.
• Navigate to Authentication from Access on the Default Virtual server properties and make sure that the Anonymous permission is enabled.
• Check the connection control configurations from Properties à Access
• Make sure that NCSA logging is enabled and find errors in them.

Backpressure:

Backpressure issues account to most of the email queue building issues. In simple words, backpressure means exchange is unable to process emails due to resource bottleneck issues.

Backpressure events look like this:

Log Name: Application Source: MSExchangeTransport Date: 4/17/2013 7:00:34 AM Event ID: 15004 Task Category: ResourceManager Level: Warning Keywords: Classic User: N/A Computer: HUBServerName Description: Resource pressure increased from Normal to Medium.

Resource utilization of the following resources exceed the normal level: Version buckets = 123 [Medium] [Normal=80 Medium=120 High=200]

Back pressure caused the following components to be disabled: Inbound mail submission from the Internet Mail submission from the Pickup directory Mail submission from the Replay directory Mail delivery to remote domains

The following resources are in the normal state: Queue database and disk space (“Q:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\mail.que”) = 3% [Normal] [Normal=95% Medium=97% High=99%] Queue database logging disk space (“Q:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\”) = 4% [Normal] [Normal=95% Medium=97% High=99%] Private bytes = 16% [Normal] [Normal=71% Medium=73% High=75%] Physical memory load = 40% [limit is 94% before message dehydration occurs.]

So, make sure that there are no 1500x events listed in the application logs of the event logs.

The following links are good resource for troubleshooting-

http://technet.microsoft.com/en-us/library/bb201658(v=exchg.141).aspx

Known issues

These are some known issues in improper inbound mail flow.

• NDR of Sender shows the error “550-5.1.1 User unknown”

Resolution

This may be an AD Replication issue or an Edge Sync issue

• Firewall issues
  • http://technet.microsoft.com/en-us/library/dd277550(v=exchg.80).aspx
  • http://support.microsoft.com/kb/320027

Quite often, clients may face issues were multiple users within the exchange are unable to make email communications within the network or to other networks or sometimes both. Here, we shall look into the troubleshooting and resolution of such issues.

Under such circumstances, we take into account, some key information we collect from the client. They are

• Availability of Non Delivery Report for senders. If there is an NDR available, we obtain the NDR and look up the error generated in the NDR and the generating server.
• In the absence of an NDR, we need to know if the failed email has been housed in the users Outbox or in the Sent Items folder.
• Next, we need to know if the user is able to get Incoming emails.
• We also check if there is an issue with outbound email. In case of an issue, we need to know if the user is facing issues for all internet domains or just some specific domains.
• Next, we need to gather some information regarding the environment. Is the environment a Mixed Environment or not, the number of Active directory sites in the environment etc.
• Next, we need information regarding the configuration of the mailflow on which the user is facing issues. If it is inbound mailflow issues, ask for the MXrecord pointer.
• Finally, check the type of connecter used for the internet- DNS or Smart Host. If the user is using a smart-host, the type and details of the smart-host is also required.

Now, we begin troubleshooting based on the type of mail flow that has been impacted.  

Mail being blocked at Exchange 2003 server: 

In case of mail being stuck at the exchange server, we follow these steps:

• Use the exchange system manager to check the queue.
• Check the additional information for the queue where the mail is being blocked. Additional Information can be found on the bottom left corner of the exchange system manager.
• Check the anti-virus the server houses. If the antivirus has made any exclusions, disable them.
• Check for the type of queue where the mail is being blocked. Use the following cases to solve for the appropriate queue viewer error.

Error- Unable to bind destination server in DNS.

In such cases, we utilize the NSLOOKUP tool to obtain the name resolution of the remote domain or remote bridgehead server.

Error – Connection was dropped off due to SMTP Protocol Event Sink

If the error returned is that the connection was dropped due to SMTP protocol event sink, we utilize Telnet to check the SMTP verbs of the remote server. Here we first check if anonymous submission of messages is pliable. If so then we have to verify the logs of NCSA protocols to check for the command at which we face the error.

If there are no information on the NCS log files, we verify that Netmon captures are available on the sending and recipient servers, preferably simultaneously.

For such an error, we begin by verifying the queue for errors or warnings. If there are no events in the queue, we retry the queue after changing the transport component’s diagnostics logging to expert level.

Error in queue – The remote server did not respond to connection attempt.

Here, we once again utilize the NSLOOKUP tool to ensure that the resolution is towards the proper server. We also utilize Telnet to check the SMTP connectivity of the remote server and anonymous submission capabilities.

Error in Queue – An SMTP protocol error occurred.

Verify SMTP verbs and anonymous message submission capabilities using Telnet. If anonymous submission is available, check for the logs of NCSA protocol and verify the command at which error occurred.

If there are no information on the NCS log files, we verify that Netmon captures are available on the sending and recipient servers, preferably simultaneously.

Mail being blocked at Exchange 2007/2010 server:

• Use the exchange system manager to check the queue. Find the queue in which the failed message is there and check for its status-Retry or Active.
• Check the additional information for the queue where the mail is being blocked. Repeat the above section’s procedures for different types of errors.
• In the SMTP virtual server, try enabling NCSA Logs.
• Check the anti-virus the server houses. If the antivirus has made any exclusion, disable them.
• Verify that the properties of the Routing Group Connector are properly configured. The cmdlet for this is:

Get-RoutingGroupConnector “Name of routing group connector” | fl

• Enable the logging for Receive Connectors protocol at the recipient server to check the receive connector that has been chosen for Exchange 2003 server and find any errors in the SMTP communication.

  The cmdlet for checking the location of the Receive connector protocol log file is,

Get-TransportServer “Name of Hub transport server” | fl

• Verify errors on the Application and system logs of the Exchange 2007/2010 target transport servers at the destination server.
• Go to the receive connector logs location to obtain logs by going using the cmdlet,

  Get-transportserver | fl for both the receiving server.

Mails stuck on E2k7/2010 for Exchange 2003.

• Use the queue viewer     to check the queue.
• Check the Queue type, Last error and Next Hop Domain for the queue where the mail is being blocked. Use appropriate tool like nslookup, Intra-orgs send connector protocol logging, Telnet, netmon, Eventviewer, diagnostics logging based on the Last Error type to resolve the issue.
• At the send connector, intra-org logging for can be increased using the command:

  Set-TransportServer “Hub server where message are queued up” –IntraOrgConnectorProtocolLoggingLevel

• Verify the source and the destination servers using the Get-RoutingGroupConnector | fl command.
• Verify the port 25 using telnet and verify the verbs.

Mails stuck on E2k7/10 for Exchange 2007/2010.

• Use the queue viewer     to check the queue.
• Check the Queue type, Last error and Next Hop Domain for the queue where the mail is being blocked. Use appropriate tool like nslookup, Intra-orgs send connector protocol logging, Telnet, netmon, Eventviewer, diagnostics logging based on the Last Error type to resolve the issue.
• Check the status of the queue using get-queue | fl command.
• At the send connector, intra-org logging for can be increased using the command:
• Set-TransportServer “Hub server where message are queued up” –IntraOrgConnectorProtocolLoggingLevel
• At the destination end (exchange 2007/10) server, verbose logging must be enabled.
• Check for the properties of the Receive Connector on the target server i.e. Exchange 2007/10. The properties to be verified are:
  • Navigate to the RemoteIPRanges list. Find the various IPs listed there. Check if the Sending server’s IP is properly configured on the list.
  • Ensure that the exchange server authentication is enabled at the Authentication tab.
  • Navigate to Permission Groups and make sure that the option selected is Exchange server 

• Use the following powershell cmdlet to facilitate Diagnostic Logging for MSExchangeTransport components at the target server:

  Get-EventLogLevelMSExchangeTransport\* | Set-EventLogLevel –Level Expert

• Verify that errors like Back pressure (15004, 15005), temporary authentication failures (Kerberos), ADTopolgy events (2080) are absent in the Application and System Logs for events related to transport. If you see these, resolve these issues first.
• Checking the Last error in queue can also be helpful in resolving the issue.

Sender receive NDR:

If the sender is receiving a non delivery report, obtain the following information from the NDR:

• Whether more than one user is getting an NDR.
• Is the NDR limited to users on a particular mailbox server or does this happen to multiple mailbox server.
• Does mails sent to a particular recipient/DL/mailbox server/site/domain result in the NDR
• Does the NDR appear for all instances of communications or do they happen at random points
• In case of random NDRs find the frequency of occurrence of NDR
• Was mail communication to the affected recipient in a working condition before the occurrence of NDR? Check for the duration from which the NDR was first seen and also check for changes made in the environment that resulted in the NDR.

  Obtain an NDR copy that can be saved. Verify the recipient information, generating server and the diagnostic information from the NDR

After obtaining the NDR, check the delivery status notification and the server generating NDR to resolve the issue. Also we can utilise the diagnostic logging at the application event logs to resolve the issue.

For the codes in the NDR, we can refer the article – http://technet.microsoft.com/en-us/library/bb124840(EXCHG.65).aspx

Mails getting stuck in Outbox and Drafts.

In the absence of NDRs and email tracking, check where the email is stuck. It should be in the Outbox in case of outlook and in the drafts folder for Outlook web app.

Further, verify the following details

• Check if the issue persists for every mailbox in the server. Under such circumstances, follow:
  • Check if the submission service is working fine and properly configured for the sender’s mailbox server
  • Make sure that the transport service is working fine.
  • Restart the Microsoft exchange mail submission service.
  • Mail Submissions service running on the MBX server is the one responsible for alerting the HUB server to pick up the email
  • In the MSExchangeMailSubmission service check for any occurrences of 1009

If the issue is occurring for isolated mailboxes, then the issue may be because of different add-ins in the outlook or because of connectivity problems at the client side. Try sending emails from OWA under such circumstances.

Message Tracking:

During the absence of NDRs tools like Message tracking comes in very handy.

Tracking should be begun from the mailbox server of the sender. The server name under the tracking tool should be updated to the mailbox server of the sender. Any HUB transport server in sender’s site can be used to run the tracking.

We can also use power shell to perform message tracking.

http://blogs.technet.com/b/messaging_with_communications/archive/2011/04/22/how-to-track-message-in-exchange-2003-2007-2010.aspx

Troubleshooting Outbound Mail flow issues

Up until now, we were discussing issues with inbound mail messages. Now we shall look into troubleshooting of outbound mail issues.

There are three main characteristic natures by which we can distinguish an outbound email issue

In the first scenario, the sender is unable to send emails and the sender receives an NDR regarding the issue.

In the next scenario, sender is unable to send the email and he can see that the message is blocked in the queue.

And finally, the sender is able to send email but the delivery of the email message is uncharacteristically delayed.

Before we begin troubleshooting, we need to gather some basic info regarding the issue from the client.

• Check if the internal mail flow was facing any issues. Also check if inbound mails are properly receiving by the user.
• Find out the time from which the user started facing the issue and if the user was able to send emails prior to that. Also find out if any changes were made to the exchange environment which may have triggered the issue.
• Find out whether the user has configured any smarthost or antispam apps for the outbound emails.
• Also check if the user received any NDR or delayed deliver report regarding the failed email. If yes, collect the report.
• Also find out if the issue is reproducible or not.
• Check where the sent email (which failed) has been housed by outlook (sent Items/Outbox)
• Ask if the issue persists for a single domain or multiple domains.
• In case the user did not receive any NDR, ask if the email is blocked in the queue of the exchange server. If yes,
  • Track the email using message tracking tools
  • Collect the queue information like the queue name etc
  • Collect the error message written against the email in the queue
  • Ask for the result of the cmdlet execution Get-Queue|fl.
• Ask for the send connector configurations (it can be obtained by executing Get-sendconnector | fl Cmdlet)
• Find out if the user has installed antivirus or antispam software on the server. If there is one, try disabling it

Exchange 2003: E-mails stuck in queue

• Use the exchange system manager to check the queue.
• Check the additional information for the queue where the mail is being blocked. Additional Information can be found on the bottom left corner of the exchange system manager.
• Find out the following details from the properties of SMTP Connector
  • The connector’s Address Space and Delivery Restrictions
  • Any smart host that the user is using to route the emails
    • If so, to the IP or FQDN of the smart host, perform Telnet and find the output
    • If connection is not going through, check whether the port 25 of the smart host is enabled to recieve emails from the server
    • If you are unable to complete the telnet to remote bridgehead, perform the telnet to some other target routing group server
  • If the user is using DNS to route the emails
    • Check if the issue is faced for all domains or just specific domains
    • Check if the MX record specify the correct IP address of the recipient using the NSLOOKUP tool.

      Performing the NSLookup

• Go to Command Prompt
• Key in NSLOOKUP and Press Enter
• Enter server {required IP of DNS server}
• Press Enter
• Enter the following
• Set q=MX

Now if you enter the domain name required and press enter, its particular MX record will be shown. This ensures that MX record has been properly configured

• Make sure that NCSA logging is enabled and find errors in them.
• From the Exchange Transport regedit, activate the diagnostic logging feature

Exchange 2007/2010: E-mails stuck in queue

• Check the information for the queue and error message details using the cmdlet Get-Queue |fl
• Find out if the send connector is based on DNS or Smart Host
• Find out the configurational details of the send connector using Get-sendconnector | fl
• If the client is running a smart host,
  • If so, to the IP or FQDN of the smart host, perform Telnet and find the output in the source server
  • If connection is not going through, check whether the port 25 of the smart host is enabled to recieve emails from the server
  • If you are unable to complete the telnet to remote bridgehead, perform the telnet to some other target routing group server
• If the user is using DNS to route the emails
  • Check if the issue is faced for all domains or just specific domains
  • Check if the MX record specify the correct IP address of the recipient using the NSLOOKUP tool.
    • Performing the NSLookup
    • Go to Command Prompt
    • Key in NSLOOKUP and Press Enter
    • Enter server {required IP of DNS server}
    • Press Enter
    • Enter the following
    • Set q=MX
    • Now if you enter the domain name required and press enter, its particular MX record will be shown. This ensures that MX record has been properly configured
• Resubmit the email using telnet from the server with HUB Transport. Check for the SMTP verbs and the IP address of the Outbound Service.
• Also enable the Verbose Logging on the Sending Exchange 2007/2010 server
• Make use of the message tracking tools to track the message
• In case of remote/local server resetting the connection before completing the message transmission, using Netmon verify if any data packets were lost or retransmitted

NDR is received by the sender of the e-mail

If the sender is receiving a non delivery report, obtain the following information from the NDR:

• The type of client from which the message was send-OWA/Outlook version etc.
• Whether more than one user is getting an NDR.
• Is the NDR limited to users on a particular mailbox server or does this happen to multiple mailbox server.
• Does mails sent to a particular recipient/DL/mailbox server/site/domain result in the NDR
• Does the NDR appear for all instances of communications or do they happen at random points
• In case of random NDRs find the frequency of occurrence of NDR
• The manner in which the recipient was chosen for the message( like from global address book, manual typing etc)

Once you get these basic information, fetch a copy of the NDR. Based on the NDR, check the User and Diagnostic Information.

Also make sure to note the delivery status notification code in the NDR and from the application event log keep the diagnostic logging of the message categorizer at level 7 / Expert level and obtain the application logs.

For the codes in the NDR, we can refer following table to troubleshoot accordingly and to find the cause.

Take a look at these Additional Information:

Using NSlookup:

http://support.microsoft.com/kb/200525

Using Telnet:

http://support.microsoft.com/kb/153119

EHLO Verbs between two Exchange servers:

http://support.microsoft.com/kb/812455

List of SMTP Verbs:

http://smtpfilter.sourceforge.net/esmtp.html

Enabling Protocol Logging on the Receive/Send Connector

http://technet.microsoft.com/en-us/library/bb124531.aspx

Definition of Queue : Queue Viewer

http://technet.microsoft.com/en-us/library/bb125105(v=exchg.65).aspx

Error Codes for NDR: Please refer to the KB 2297581

–